Is Process Killing the Software Industry?

blackbearnh writes "We all know by now that Test Driven Development is a best practice. And so is having 100% of your code reviewed. And 70% unit test coverage. And keeping your CCN complexity numbers below 20. And doing pre-sprint grooming of stories. And a hundred other industry 'best practices' that in isolation seem like a great idea. But at the end of the day, how much time does it leave for developers to be innovative and creative? A piece on O'Reilly Radar argues that excessive process in software development is sucking the life out of passionate developers, all in the name of making sure that 'good code' gets written. 'The underlying feedback loop making this progressively worse is that passionate programmers write great code, but process kills passion. Disaffected programmers write poor code, and poor code makes management add more process in an attempt to "make" their programmers write good code. That just makes morale worse, and so on.'"

Comments

"Creative"

[Anonymous Coward]

If you want to go off and do your own thing, fine. Have at it.

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

Re:"Creative"

[Entrope]

This is dead on. Every software-developing business needs to decide its own process needs. Even if they are developing safety-critical code that has to pass rigorous certifications (FDA, DO-178B, etc), sane people do not order everything on the process menu. However, an organization does have to look at its market, its code base, its structure/culture, and its history to figure out what kinds of process it should use. Sure, having a defined process means developers spend less time throwing code at the wall, but the code they do throw usually sticks to the wall better and longer, and they usually feel good about that.

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization. As an software developer and occasional manager and/or process guy, I have seen both cases. I have also seen cases where having a defined process helps channel creativity. Good process tools help you focus on the right parts of the problem; for example, having a template for a design description that identifies particular subjects to focus on, and may suggest areas that have been rabbit-holes in the past.

Re:"Creative"

[realxmp]

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

Keeping any aircraft/metro/car safely in the air/on the rails/road does require a process yes (specification in Z etc), but it needs to be balanced with a certain degree of interest in your subjects that only comes from enthusiasm. In the real world no specification is 100% complete and there are often times when a coder has to make a judgement call and send a memo asking for clarification saying "This was unclear I did this, confirm it is correct" (blocking until you get an answer isn't usually an option owing to deadlines). Quite often these aren't answered and it's only the programmers own interest in what they're writing and passion that ensures the correct decision is made. Keeping that passion and enthusiasm is nigh on impossible if EVERYTHING is process.

Re:"Creative"

[Anrego]

Obviously the amount of process is going to be proportional to the number of metric ass-tonnes of shit that hit the fan if something goes wrong, but I still think there is room to at least choose processes that don't overlap too much or interfere with each other. In the aerospace, defense and medical industries, it's all about more and more process to make everyone feel better (and produce better software and such..). Bug gets caught in one phase of testing.. add more process so it would have been caught earlier!

I would add however that excessive process, or more destructively, processes that fight each other, can not only hamper creative solutions, but can prevent people from making enhancements due to the excessive amount of "other stuff" that has to change in addition to the code. I would certainly say that a lot of process gets tacked onto stuff that's non critical, mostly for buzzword purposes, which is I think mainly what this article is speaking to. Having a process in place so management can say "we do agile cloudsource coffeecup modeling!" or whatever is always bad for the guys who have to work in that environment.

Re:"Creative"

[donscarletti]

If you want to go off and do your own thing, fine. Have at it.

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

You need creativity to write something accurate, elegant and easily reviewed, tested and verified. You need discipline to actually do the review, testing and verification steps. Both are equally important if you want something solid and delivered on time.

Re:"Creative"

[heathen_01]

This is dead on. Every software-developing business needs to decide its own process needs.

This is just another cargo cult problem. Managers, seeing that another company/team use process X on a successful project, decide to implement the same process for their team. However no process will make a difference if the developers have been directed to build the wrong solution in the first place, even if the code is 100% bug free.

Re:"Creative"

[vlm]

That is the type of scenario that we need discipline, not creativity.

Its a mistake to think discipline and creativity must be bipolar exclusive opposites. Anecdotes from all parts of that venn diagram claimed as "universals" are not really insightful, either. The best solution will maximize both within constraints and specs, assuming you have the discipline to provide any, and enforce them.

Re:"Creative"

[pedantic bore]

Please accept this comment in lieu of mod points, which I do not have at the moment.

Re:"Creative"

[HungryHobo]

I've heard some horror stories of companies which, as you say, "order everything on the process menu".
Everything has to be signed off on by 20 people who have no relationship to the project, integration networks that a guarded more closely than production networks and even trivial projects take years to production etc etc.

I know it can be painful writing documents which you know nobody will ever read because the one thing the org I last worked at lacked it was a good documentation tracking system so people spent huge chunk of their time documenting their actions but when things went wrong their either wasn't enough time to wade through all the poorly labeled crap or the documentation had been lost.

Quick general tip for anyone working small app development in a large org: if you have documents, even if there's some place they're supposed to go as well just throw an extra copy into a "doc" file with the application if you can.
It's for the sake of the guy 10 years down the line when nobody remembers where the original docs went.

Re:"Creative"

[RyuuzakiTetsuya]

My thoughts exactly.

I worked basically in the skunkworks for an ISP's call center. We developed the CMS that handled all of our policy and procedure documentation, we developed neat tools that interacted with our IVR for outages, displayed coverage maps with said outage data, and various little tools here and there.

When we started, we weren't bound by our IT department's change management process. As our tools grew and as their usefulness permeated our business, we started to get noticed and before we knew it, we were being held to the same standards for process as teams who developed tools like our CRM and our internal ticketing system.

The whole point of our skunkworks was to be nimble, efficient, and effective on smaller projects those teams didn't have the ability to take on due to commitment to other projects. Yet, as IT's demands for compliance grew, all of our advantages went away. Soon, i found myself having to answer, "Why is late?" and invariably the answer would be, "Oh, it's held up by IT's Process. It'll be three weeks." Managers weren't happy. I REALLY wasn't happy and stressed and my user base wasn't happy either(Although my user base had a certain degree more sympathy than management did).

It sucked the life and effectiveness out of our team.

When layoff time came a month ago, on paper, we looked like were banging rocks together, and bunch of us were laid off, with those teams that didn't have the time or resources to build these tools in the first place having to absorb yet more work.

(note: I didn't mind the particular IT policies, they made sense, but, I felt like a sense of scale was lost when it came to our projects. Oh well.)

Re:"Creative"

[HungryHobo]

there not their*2

god I need to sleep more.

Re:"Creative"

[Barbara, not Barbie]

The usual problem is bosses who either don't want you to follow a process, or they don't want to have to bother with the work that they would have to do.

How many times have you been told to forget updating the documentation or specs (or worse, "what documentation?"), or you're told to "just make it work"?

Or you've sat in a meeting during a review of the current project, and it's just a waste of time because it's like management is just filling up their buzzword bingo card by spouting various processes that could be used to improve things - and you should implement them and then tell them how (or if) they work. (as long as THEY don't need to be part of it beyond reading a half-page summary every once in a while)

Often, these are the same guys who will demand a 120-page report on the current state of the project "and it needs to be in my inbox in 48 hours" - and then "tl;dr" it.

In many cases, their reluctance to follow process is because it would show that they don't have the understanding to make a meaningful contribution to the process. If they don't tell you what to do, it's not their fault when the horse pucky gets into the HVAC. They'll blame you for not following their ad-hoc "management process."

Like the boss who dumps 3 different emergencies on you at the same time, and then gets all snarky because you didn't fill in the detailed spreadsheet of what you did every 10 minutes ... because now he can't use it to figure out how to fill in his time sheet of what he did all day, because he never keeps track of his time - just works it out backwards from yours.

Most developers would be more appreciative of processes if they weren't used as a weapon by the same management who refuses to adhere to sane processes themselves. For them, it's just CYA all the way down ...

Re:"Creative"

[joebagodonuts]

...and you've exposed the secret - developers can create and deliver good code WITHOUT being audited to death. The whole "lean six-sigma/ITIL/insert-the-name-of-your-favorite-process-methodology-here" is just another way for bean-counters and MBAs to pretend they can do something to add value to an endeavor.

Re:"Creative"

[ZeroExistenZ]

But don't expect to write code that keeps a 777 safely in the air.

He writes about creativity. You write about engineering.

There are now a multi-tude of disciplines in software development, yet this isn't very well understood.

I agree that the "management squeezing the life out of their dev-teams to make deadlines" with their charts, while devs are filling out sheets like monkeys to justify the space they take up, then it is absolutely a process that is killing.

I remember a time filling out 3 timesheets for the same work, filling out a scrumlog, doing a scrum-board, sitting in meetings, linking my code-check-ins with my timesheet. Management could make pretty graphs, but I wrote 20% of the code I did before. And it didn't work. All this overhead work is also not "estimated into the planning", so pressure mounts, creativity and production lowers.

Maybe the ad-hoc wont keep the 777 in the air, your papertrail and logs wont even get the software to be installed into your 777 and it'll never lift off. But it justified the bill presented for all the hypothetical work done (spent on filling in "justification files")

Re:"Creative"

[speculatrix]

+1 sadly true

Re:"Creative"

[Trailer Trash]

Right. And maybe 1% of developers out there write code that is that critical. *Maybe* 1%. Most are writing web sites or internal applications that have nowhere near that kind of criticality.

Re:"Creative"

[Omnifarious]

I rated you up, but I notice that most people seem to be missing the most important part of your post.

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization. As an software developer and occasional manager and/or process guy, I have seen both cases. I have also seen cases where having a defined process helps channel creativity. Good process tools help you focus on the right parts of the problem; for example, having a template for a design description that identifies particular subjects to focus on, and may suggest areas that have been rabbit-holes in the past.

I agree that software development can be 'over-processed'. And I think that management frequently decides to 'apply more process!' in lieu of actually carefully thinking through what would help. But usually, when developers complain about process and delivering things on time, I think "Oh, so you just want to slap that code in there without even really knowing if it works. Write code, compile, problem solved!". Good process is what keeps you on-track and disciplined to write quality software. Too much software in our industry is of extremely poor quality, and that's still the case. Developers should feel proud of writing software that works, not how much software they write.

Most of the responses I see to your post seem to be of the "You're darn right, all that process just kills development!" variety. No talking about how to balance things for quality results or anything of that nature. I'm kind of disappointed.

Re:"Creative"

[Calsar]

You cannot manage your way to good code. Good developers write good code and bad developers write bad code. At best these processes help you mitigate some of the risk and ensure your requirements are being met. If management spent as much time working on the hiring processes as they do on managing the development process they’d see a much higher return on their investments.

Re:"Creative"

[easyTree]

It's always a hard-sell to tell management that "they" are the problem and should be removed ASAP.

Re:"Creative"

[easyTree]

+1 Feeling your pain.

Re:"Creative"

[RyuuzakiTetsuya]

I didn't mind having to abide by a process, but my point was, I felt like scale was being lost. Mine wasn't the only team that had this same complaint.

For instance our database access policies when we were integrated into the larger IT infrastructure was the same as the guys who were dealing with PCIDSS data, and we were held to the same standards, so I couldn't just log in and just run a few simple reports. In our working environment management constantly demanded new and ever changing reporting. Usually a one off data pull to prove some point in a powerpoint slide. Having to say, "this is going to take three weeks because the guy who IT operations trusts with DB access is currently so swamped he's not going to see the light of day until well after the sun has burnt out and turned into a lump of coal roughly the size of a golf ball" to a manger who needed that data 2 hours from now wasn't acceptable.

Re:"Creative"

[OptimusPaul]

I think you've hinted to the point that the article failed to make.

Processes are great, and absolutely required. But if you spend more time on the process than on the work then you have a problem. All the pre-work and planning might save you from a costly refactor once you find out you went down the wrong path, but honestly I've seen projects that had all the tests written up front and months of architecture planning and sprints planned only to jump into development and find that the whole approach wasn't feasible. Had they jumped into coding sooner they could have saved themselves close to a million dollars. I've also seen the opposite, the planning paid off and they had a great product, but they were late to market and it never sold well. Their competitors made it to market faster, their product wasn't as good but it was there and it improved over time.

Also, processes aren't going to protect you from bad developers. It just allows the bad developers to burn through budgets longer before they prove that they are not up to the task.

Re:"Creative"

[jellomizer]

Over the past 10 years I realized something about software. While it hasn't been too much innovative it has gotten a lot more stable and secure. I think most of us has forgotten the hazards of computing during the 80's and 90's. Perhaps we need to enter an era that we need to innovate more again, but it will come at a cost of stability.

Re:"Creative"

[Entrope]

One thing that I have noticed -- which my boss (the director of engineering here) also harps on -- is that most companies cannot afford to create processes that assume good developers. They can assume that bad developers are moved out of the organization, but many developers are in the middle. At least in this area of the country, we simply cannot find enough developers who know the basic principles of the kind of systems we work on. (I am not talking about the particular use of our software, but things like hard real time guarantees, designing usable user interfaces or analyzing control loops.) We can find plenty of people who are .NET or Java enterprise-y experts or who can write DSP code, but they would not walk in the door as "good" developers for our projects.

Besides, even good developers make mistakes. They make more mistakes when they are under serious time pressure. That is human nature. Development processes -- at least when they are a good fit for the project -- help ensure that good developers deliver good code on a predictable schedule.

Re:"Creative"

[cpu6502]

Bad developers OR bad managers.

In my experience Process Assurance means little. The board or software fails certification testing, and the managers over-rule the failure, and mark it "passed" anyway. PA is just a smokescreen and managers act like politicians.

Re:"Creative"

[AshtangiMan]

In my (somewhat limited) experience as a developer I would say there is a failure in general to have the requirements defined up front, all at once prior to beginning the design process. More often than not what happens is the initial requirements scope creeps along as the development progresses. This can make it difficult to have a design that is cohesive across the full set of requirements. The clients don't usually know enough about their own requirements to have them fully flushed out at the beginning, and as they see the incremental development versions are reminded of other things they would like to do. This leads to bad code almost necessarily as new requirements are added on top of the existing code base. This seems to happen in every single career field I have had, not just development. Management's solution in general is the "cargo cult" problem discussed by parent.

Re:"Creative"

[Entrope]

I am disappointed too. I have lost count of the times that my coworkers (or direct reports) decided to skip documenting a design because of pressure to get the code done, only to have that come back and bite them (and the project manager who was applying the schedule pressure) later. Then there is one coworker who made commits to reindent and reformat every source file in a project -- mixing that with functional changes. Another coworker decided to write his own not-quite-XML parser.

I probably annoy my coworkers occasionally with my insistence that we define and follow certain (usually minimal) processes. I know there are one or two people who hate our code review system. However, I have never advocated for a new process unless it was meant to fix a recurring problem and I thought it was close to the most efficient way we could mitigate the problem.

Re:"Creative"

[Entrope]

Requirements management is an inherently hard problem, for the reasons you list. Unfortunately, for almost all projects, it is impractical for all of the requirements to be laid out in advance. Fortunately, though, a valiant initial effort to document and analyze the requirements can save a lot of rework and confusion down the road.

Re:"Creative"

[radtea]

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization

This.

People who blame process are like people who blame XML, or C++, or any other tool.

The problem is not the tool. The problem is the stupidity of the people who insist on using it in inappropriate ways.

Process is never the problem. Stupidity is the problem, which leads to inappropriate processes, poor metrics and bad management.

People who complain that "process kills creativity" are just as stupid as people who implement inappropriate processes. Ironically, they lack the creativity to see how appropriate processes would enhance their productivity and freedom. As engineers say, "Form is liberating."

Re:"Creative"

[d'fim]

Reminds me of The Gervais Principal.

Re:"Creative"

[nschubach]

I'm sure I'm not alone in this, but I find if there's a "defined process" the only "creativity" I have is in creating a tool to automatically fill in that process.

You want me to enter my time card on that webpage every day with the same numbers? Fine! Automated.
You want me to generate comments for every function I write even though the function name explicitly states in it's name what it does? Fine! Automated.
You want me to get at least 6 hours worth of work on my Scrum each day and you berate me in front of the other developers if I don't? (yes... that happens) Fine! Automated.

All that most of these "busy work" processes that get in the way of me making a product have only made me want to remove the hindrance of the processes. And you know what? Management doesn't care because the numbers work. It's like a few years back when they were required by their management to have 100% participation in United Way donations (yep, that happened too) and my manager would basically punish me for not donating to charity until I started donating a penny to make it more expensive to process than it was worth... but it got their numbers on par.

Re:"Creative"

[swanzilla]

sudo chmod -R 650 *

Re:"Creative"

[angel'o'sphere]

"This was unclear I did this, confirm it is correct" (blocking until you get an answer isn't usually an option owing to deadlines).

That depends. The better approach likely is to block this task and to work on something else.
The question is easy answered with a risk analyis. What risk is higher: a) developping somthing that is not confoirmed right and has to be reworked
b) not meeting the deadline by blocking and having idle time instead
If you assume you have idel time, you ofc can develop a "educated guess" until the requirements are clarified.

However if you would follow an agil approach such a requirement would not be in the sprint. The developers had rejected it during sprint planning.

angel'o'sphere

Re:"Creative"

[angel'o'sphere]

You need creativity to write something accurate, elegant and easily reviewed, tested and verified.

I doubt that.
I only need creativity when I program in "unknown" languages and terrain. E.g. trying to write a DSL in groovy to describe UML models, or playing with Scala etc.
In real live I likely did not need creativity in "coding" for the last 15 years. What creativity is needed to code a crystal clear requirement or specification? Sorry I don't get it.
angel'o'sphere

Re:"Creative"

[pixelpusher220]

In my experience Process Assurance means little.

At my company we call it 'Quality Assurance' which is right up there with Military Intelligence. These QA people are not on the projects, they are almost exclusively at the main office corporate level and just take the projects word for it that everything is working. There are document milestones (req's, design, etc), but no even cursory investigation as to how the documents are created or how it relates to the actual process going on.

I understand that politics and budgetary concerns can trump 'best practice', that's just normal real world stuff. But pretending it doesn't affect how the 'process' is what gets people into trouble. We're currently running req's, design and development in parallel, yet still pretending it's a waterfall schedule. sigh.

Re:"Creative"

[Archangel Michael]

Was gonna mod you up, but instead, it hit me. The problem isn't the process, it is where the process is engaged. And I think the Article and subsequent posts clearly indicate there is a time and place for process, but not everything needs to be stuck in a process.

In your case, you and your team were building tools for your own use and were doing it the quick and agile way. Dirty code that got stuff done quickly. You bypassed the systemic processes that were in place because they just got in the way.

The successful code base often starts out as UGLY spaghetti code, but eventually needs to be cleaned up.

What needs happen is your skunkworks program should be autonomous from the normal process. Only when something you've created gets noticed does it move from the skunkworks over to the normal development channel. To accomplish this, you have to be willing to hand off your code to someone who is, more than likely, going to "ruin" it by running it through the process of getting it clean and neat.

This way, you can still build fast efficient code that is messy and ugly, and the managment can get code that is functional, and everyone can work towards getting it all nice and pretty by process. Everyone wins.

Just my thought

Re:"Creative"

[Americano]

Actually, "blocking until you get an answer" is a very effective method for getting a critical answer or clarification.

An email to the person you need an answer from, their boss, and your boss, clearly stating:

"I do not know how to proceed on this issue, because the spec is unclear. I require clarification and answers to these questions before I'm able to proceed:
(list of questions)
I cannot proceed on this effort until I receive this clarification; this is a blocking issue and runs the risk of disrupting our schedule if clarification is not received soon. In the meantime, I will be moving on to work on my second-most-important task until I receive this clarification.
(optional: provide a recommendation or possible alternatives if appropriate, and ask them to make a choice or to provide their own preferred alternative.)"

Slamming on the brakes on a critical project, or threatening to do so, will get you a very fast response, and it's not an inappropriate technique for mission critical projects. It's better to stop & get clarification than to proceed and say "don't say anything if you agree with what I did," because that guarantees your message will be skimmed-at-best by the people you're asking for a clarification from.

Re:"Creative"

[majestic_twelve]

Bad programmers write bad code, period. Process can't get better code out of a bad programmer, sorry.

Re:"Creative"

[TheLink]

For most projects you will never have all the requirements defined up front.

Writing detailed complete requirements takes about as long as writing code, and as you've mentioned clients usually don't know what they need.

So the developers/analysts will have to write the requirements for them and guess what they want and how they might change their mind in the future. How much they guess wrong depends on luck and experience ;).

I've put in stuff before which people say they don't want, but later they want/need it because of some new external demand.

If you get the foundation/schema/design right, you don't have to rewrite stuff to add features they want later. You don't have to write all the code in advance, just prepare the way "just in case" :).

One of the big pains I find is fixing/overhauling other people's code. I'm far from a great coder, but often it seems the reason other developers seem so fast is because they're skipping a lot of stuff, or even doing things plain wrong. The other big pain is dealing with crap like "legacy PHP" and similar crap that make it so hard to do stuff the right way.

FWIW, some people have had decent results writing the "user manuals" first! I see some merit with that idea, but I'm not sure what projects will be good with that approach.

Re:"Creative"

[Archangel Michael]

No answer is an answer. Depending on who you're talking to, it could mean "yeah that is fine" or "stop, wait, don't do it", or it simply could mean "if the shit hits the fan, it is on you, but all success is on me".

Know your boss.

Re:"Creative"

[Skapare]

And the certifications don't actually check any code, they just check that the processes chosen are the right ones, and can even miss if the processes don't get adhered to. Perfectly developed and certified software can still have a ton of bugs that will crash that 777 or shut off the heart monitor in a hospital.

Less on process and more on coder competency would help. Pick the best coders and double their pay.

Re:"Creative"

[Bengie]

There are three things you can pick: Fast, Cheap, Tested

Seems the whole point of what he did was to be done NOW. So, Fast. Seems they weren't willing to throw more resources into the mix to speed up testing, so Cheap.

They wanted Fast and Cheap. You can't hold a person fully to "standards" when his job demands Fast and Cheap.

Re:"Creative"

[mcvos]

You think that code was written with less process? Less testing? Less code review? I think the process for 777 software is quite a bit heavier than that for your average webshop.

The real point though is this: wasn't Agile supposed to reduce the reliance on process, and increase the focus on people? TFA sounds like that didn't quite work out.

Re:"Creative"

[DiegoBravo]

Maybe you need a process in order to detect and fire those bad programmers.

Re:"Creative"

[sconeu]

I didn't mind the particular IT policies, they made sense, but, I felt like a sense of scale was lost when it came to our projects

This.

For large scale projects, process is absolutely necessary, if only to manage the issues that Brooks pointed out back in the Dark Ages (The Mythical Man-Month). And they are definitely needed when DO-178B or other field-specific qualification is required.

I was at a major defense contractor when a CMMI level 3 process was imposed from above. There was a lot of rebellion, but we gave it a go.

We had a skunk-works type project that was 4 developers. Upper Management insisted we follow the process to the letter. This included Fagan-style formal inspections of all documents. Of course, they didn't give us enough time to do it right. We were doing 2-a-days, for 10 days straight, without enough prep time, or enough people to do it right. It wound up being useless nitpicking, with no useful results. On top of that, we lost 2 weeks to this exercise, and the developers were totally drained for about 2 days after it.

A more traditional-style document review would have worked much better, but The Holy Process had to be followed.

Process is great, but it needs to be fine-tuned to the particuar project.

Re:"Creative"

[DrgnDancer]

Interestingly I've worked in Boeing facilities, and I can comment a bit on their CM and process management (I'm not and never have been a Boeing employee, I was a sub on one of their government projects). You are 100% right about the way Boeing generates, well, everything. Code, systems, engineering, from rockets to airplanes, from missiles to widgets. It all gets vigorous change management, quality control, and review processes. Specs *are* complete, becasue review of requirements is complete, becasue requirements generation is complete, and if any of it isn't complete it'll get pushed back up the chain until it is. Period. Because being a few mils off on the kind of stuff Boeing builds probably means someone is going to have a really bad day. Possibly a fatal day.

I understand completely why they work the way they do, and every time I fly to Boston on a 737 I thank them for it. That said it's an incredibly painful way to work, and it generated a phenomenal amount of inefficiency on systems and processes that really don't need to be perfect. The corporate culture is utterly incapable of turning off the "we build airplanes, everything must be within a 1 mil tolerance" frame of mind. The pain and suffering required to make the most minute changes to our lab environment made my head spin. I was really glad to get out of that place.

I think there are basically three things to consider when it comes to process and change management:

1) The same level of care is not required in a simulations lab as is required to build 777's. Though the level of care should definitely be more than the level used in a college research project. Pick a a level of process and control that is appropriate to your environment and goals.

2) Know your team. Different people flourish under different environments. In some ways this is related to (1). A team of game designers is never going to be happy working under the constraints that a team of aeronautical engineers would consider fairly loose.

3) Accept that some level of process and control is probably inevitable. One of the many memes that Slashdot loves is that software engineers are not real engineers. There is a level of validity to this statement, but it is in fact changing. We are becoming, more and more, the engineers that we've been claiming to be for a while. Software engineers at a company like Boeing *are* engineers in every way except maybe the PE cert. The rest of the industry is following a bit more slowly. There are both pluses and minuses to this formalization and professionalization, but it's probably going to happen regardless.

Re:"Creative"

[richlv]

If someone seriously and repeatedly complains that following the process kills their passion, it is due either to a failure of that analysis or them being in the wrong organization.

sometimes it even sounds like "stop finding bugs in my code, that just slows me down !" ;)

Re:"Creative"

[CorSci81]

I work at a place where CMMI is being imposed as well. The problem I've seen is a lack of understanding what CMMI actually means and requires. Instead of defining existing processes that are already in place (and mostly work) in a way that met the CMMI requirements, a boatload of new processes were written blindly to the "requirements" with no regard for how software development actually works. Needless to say that failed miserably and we're just now documenting processes that are in place so new projects can pick from a menu of processes that have already been proven to work that also meet CMMI reqs.

Re:"Creative"

[Creepy]

For most projects you will never have all the requirements defined up front.

That depends on the development methodology. For Waterfall all requirements are supposed to be defined up front. For Agile they are pulled off a stack and added to while in process based on need. I remember from school the three traditional methodologies are Prototyping, Waterfall, and Spiral, but I've really only used Waterfall until recently.

  I'm actually working in a traditionally Waterfall methodology company on a small but growing Agile team. Audits have been a pain for the Agile team because of required pre-coding documentation that doesn't really apply to Agile (like a functional spec). The European audits have been particularly bad (worse than the ISO 9001 audits) because they seem to require Waterfall. We are required to be certified by all of these audit companies by contract and by the US Government, so it isn't by choice (even though they are pretty much worthless, IMO).

I know a few people that can spit out good code fast (one guy even documents well, but most don't), but they are few and far between. I am fairly slow, but I usually get it right the first time.

Re:"Creative"

[Jane Q. Public]

This illustrates what a lot of people here have been dancing around but nobody yet has said explicitly: it isn't just process, but bureaucracy itself that kills passion and creativity.

If you want Agile programmers, then let them be Agile. Don't stultify them with too many meetings, re-working of requirements, and constantly waiting for features to be signed off on by the corporate bigwigs. They don't know how to write software.

Granted: some mission-critical software has to meet certain standards. But there is no rule or law saying that rigid processes need to be followed to get there. That's just wrong. Thorough testing is necessary in those cases, certainly. But that's what TDD is for. Let it work.

All in all, I think this is just another example of the old "Waterfall" method of software development creeping back in, due to pressure from management who want to feel more in control. Which does, in fact, kill creativity. And appreciation for the job.

Re:"Creative"

[TapeCutter]

People who complain that "process kills creativity" are just as stupid as people who implement inappropriate processes. Ironically, they lack the creativity to see how appropriate processes would enhance their productivity and freedom. As engineers say, "Form is liberating."

Excellent post, I had almost given up hope of finding someone in this thread worthy of the title of software engineer!.

Re:"Creative"

[jafac]

Exactly right.
Coders' "passion" is fine and dandy and all. But process recognizes that we don't put passion above sound engineering principles. And that individual coders are not infallible. Without process, you'll write very cool flash games, to be sure.

Re:"Creative"

[TheLink]

AFAIK the waterfall model was first used as an example of a flawed broken process in 1970.

http://en.wikipedia.org/wiki/Waterfall_model
http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf

Re:"Creative"

[Jane Q. Public]

"In your case, you and your team were building tools for your own use and were doing it the quick and agile way. Dirty code that got stuff done quickly."

This kind of thinking is exactly the problem. There is no reason "agile" code has to be "dirty" code. Studies have shown that it can be of as good or better quality than code done "the other way". But management resists that, because... they think like you: if they are not dictating the process, it isn't going to get done "right".

Re:"Creative"

[lonelytrail]

I feel sorry for all you guys. I've worked in "no process" companies before and it sucks. No requirements, no process, no cohesion. Politics were rampant. It was just plain elementary.

These days I work in a company that's been CMMI level 3 certified for years and we have a process in place for each level of necessity; light, medium, heavy, manned-flight. All of our requirements are part of the contract, before any design even begins, and if they change the customer must pay for the change. It's a joy and my creativity is unleashed, not restrained.

Process is not your enemy. It's the lack of buyin. Everyone must participate and ensure the process is held. Requirements definition is part of process and if you don't have stable requirements, you aint got shit.

Again, I love working within our process, whether it's the light one or the manned-flight one. It's well defined and provides a schedule, but doesn't tell me how to write my design/code/test, just that I get it reviewed at all of the proper phases.

Re:"Creative"

[lonelytrail]

Continued from above.
One thing I forgot: Our process provides templates for all the BS documents we all hate to write. Half the effort is done by the boiler-plate and the only thing I have to write is my creative part about how I'm getting the job done.

Re:"Creative"

[Intron]

For most projects you will never have all the requirements defined up front.

Writing detailed complete requirements takes about as long as writing code, and as you've mentioned clients usually don't know what they need.

So the developers/analysts will have to write the requirements for them and guess what they want and how they might change their mind in the future. How much they guess wrong depends on luck and experience ;).

I've put in stuff before which people say they don't want, but later they want/need it because of some new external demand.

If you get the foundation/schema/design right, you don't have to rewrite stuff to add features they want later. You don't have to write all the code in advance, just prepare the way "just in case" :).

One of the big pains I find is fixing/overhauling other people's code. I'm far from a great coder, but often it seems the reason other developers seem so fast is because they're skipping a lot of stuff, or even doing things plain wrong. The other big pain is dealing with crap like "legacy PHP" and similar crap that make it so hard to do stuff the right way.

FWIW, some people have had decent results writing the "user manuals" first! I see some merit with that idea, but I'm not sure what projects will be good with that approach.

The thing is, the developers have to write the requirements anyway. You can either make it explicit and write a document up front, or you can do it piecemeal as you go along, which will lead to constant errors and backtracking. So you might as well do it up front.

I'm writing a document right now (which wasn't requested by managment, BTW) just so I could nail down all the points so they can't come back later and say "that's not what I expected".

Re:"Creative"

[Jane Q. Public]

"There are three things you can pick: Fast, Cheap, Tested"

If done right, TDD can be faster than other development methods. Why? Because, among other things, it eliminates having to go back and fix regression bugs and other such issues, which standard "after the fact" testing does not address.

And at the same time, it actually reduces the time, size, and budget needed for traditional "after the fact" testing and QA.

Re:"Creative"

[Intron]

OK. And when was Agile first used as an example of a flawed broken process?

Re:"Creative"

[Jane Q. Public]

That is kind of a running joke in the Agile development world: how "waterfall" management fails, and they add more process in an attempt to "fix" it.

Kind of like borrowing more money to "fix" the economy, when cutting spending is the only thing that really will.

Re:"Creative"

[Jane Q. Public]

It probably would not earn you any Brownie points at work to bring this up, but in most states I believe requiring employees to donate to somebody's favorite charity is illegal as hell.

Re:"Creative"

[Omnifarious]

And none of the processes you outline do anything useful. Aside, possibly, from filling out a time sheet. And if you get away with filling one out with the same hours every day, then something is broken.

On the other hand, test-driven design and code reviews are immensely helpful. And if the process is working you don't get to skip either of those things without the rest of the team knowing you've done it and approving your behavior, in which case the team and environment is the problem, and not the process.

Re:"Creative"

[Jane Q. Public]

"People who blame process are like people who blame XML, or C++, or any other tool."

If that were really true, then Agile programming practices would never have gotten anywhere. But in reality, it has been one of the fastest-growing industry methodologies that ever existed. Why? Because it gets the job done BETTER than more "process". (Of course a lot of companies do not know how to implement it properly, but that isn't an argument against its effectiveness when it is done properly.)

Sure, there are cases in which it is not appropriate. But I would say roughly 90% of the time, it is.

Re:"Creative"

[__aaklbk2114]

I wish you weren't AC so I could mod you up. Your insightful explanation is exactly how software "engineering" works at any large corporation and it's a total joke. The only way to fix it is to add more "Enterprise Architects" or "SOA Integration Experts". :)

Re:"Creative"

[Jonner]

If it really didn't require any creativity, you wouldn't need a human to write it.

Re:"Creative"

[Omnifarious]

I came to the conclusion awhile ago that making money is the wrong goal. The money should be treated as a reward, a signal that people think you're doing a good job, not as the goal. If you're treating it as the goal, you will do all kinds of things to get there that actually make things worse for everybody rather than better.

Re:"Creative"

[Jane Q. Public]

"But if you spend more time on the process than on the work then you have a problem."

AND people should keep in mind that a lot of that time devoted to process is often hidden in their everyday work: wasted hours in meetings every week, having to wait to get things signed off by management, etc. The lost time to process isn't all stuff that happens in the offices of upper management. It's how they manage the work that goes on every day.

Re:"Creative"

[Omnifarious]

That's not actually true. Sometimes bad programmers just have bad habits. And if they get a constant stream of negative feedback about their bad habits, maybe they'll change them.

I, for example, always have extensively tested everything I've ever written. But those tests were usually not done in a way that was repeatable by anybody but me. Recently, the drive for test-driven development has made me rethink design so my programs were easy to put into a test harness with tests anybody could run with repeatable results. This has made me a better programmer. My programs are easier for others to work with, and communicate better to the rest of the world how they work, which makes them easier to modify by people who aren't me.

Though I'm still rather proud that my programs have always had DJB levels of defect freedom. :-)

I agree that some people simply can't improve. But I think those kinds of people are rarer than you think.

Re:"Creative"

[MDillenbeck]

If you want to go off and do your own thing, fine. Have at it.

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

I just finished an intensive course on Human Computer Interface (HCI) design. True, poor code can cause these things to happen - but so can poor design. In your example, we read a research paper where they discovered pilots use spacial location and position of analog instruments and components to gain information "at a glance", but with an all digital design they received information in purely numerical means and thus were unable to quickly process the information (in other words, the cognitive load of the analog systems was lower by 'chunking' the information via visuospatial encoding). I would suggest to the coder who wants to expand into a "creative" realm, they get involved in HCI and design teams or help form one in their company.

Re:"Creative"

[unrtst]

So many posts, like this one, make me thing "gee, if my coworkers are reading this, they're going to think my /. nick is Entrope"

Been there on the code-reformat as part of a large functional change within one patchset. Make me feel old when that happens, cause no one that has ever had to troubleshoot/bugfix someone else's patch like that would ever again do it themselves.

There is a point where there is too much process, but that's often process that's mandated for management reporting purposes and not something that makes development smoother. On the dev side, we mandate a Changelog entry in a specific format, but since we did it, we can also automate that - a script grabs the version control logs on that branch and formats the changelog entry for the developer, and they just have to paste it in and maybe clean up any krufty log messages they had put on their changes.... that's easier and more accurate. On the other hand, requiring project management entries, and change control tickets, and manager signoff, and hours accurals, etc etc etc for a bugfix that simply has to go to production ASAP is just silly, and doesn't help matters at all.

Re:"Creative"

[Lonewolf666]

We're currently running req's, design and development in parallel, yet still pretending it's a waterfall schedule. sigh.

Sounds familiar to me. I can even tell you how and why this happened on the current project:
Step 1) Project started with ambitious (but still doable) time plan.
Step 2) Design input is late, because the marketing and customer support guys who should provide it cannot or won't develop detailed specifications without a prototype. At least, they can agree on some basic requirements.
Step 3) Developers building a prototype without knowing the exact wishes of marketing and customer support.
Step 4) Prototype shown to marketing and customer support. NOW those guys wake up and provide detailed input. Of course, this leads to a bunch of changes.
Step 5) Since there is already a prototype, the waterfall schedule devolves into "oh, let's try this...". Sort of agile development, only without the discipline. The waterfall schedule is history ;-)

Re:"Creative"

[sjames]

That would be why TFA said:

Now, I'm certainly not advocating some kind of Wild-West approach where nothing is tested, developers code what they want regardless of schedule, etc.

We actually need both creativity AND discipline to keep the 777 safely in the air. That does mean some process. TFA doesn't dispute that at all.

What it's talking about are those dreadful shops where there is so much focus on 'the process' it actually becomes the objective in the minds of everyone there and producing code that keeps the 777 up (hopefully) becomes a mere side effect. In the worst cases, eventually the side effects are eliminated in order to better perform the objective of following "THE PROCESS".

What is really needed is good programmers focused on the objective doing the right thing. "The Right Thing" will include some process as well to make sure the code can be maintained and that it actually keeps the 777 in the air.

Re:"Creative"

[hey!]

However no process will make a difference if the developers have been directed to build the wrong solution in the first place, even if the code is 100% bug free.

This observation is wise, but I'd go further.

I was thinking recently about all the methodologies that have come down the pike over the years, and reflecting that I have found most of them to be good, but none of them to be sufficient. Furthermore, a lot of us had pretty consistent success before these methodologies came along. In some cases those methods crystallized and focused things we had been doing, but in just as many cases they corrected mistakes we had habitually made. Yet despite making those mistakes we had successful projects, so none of those best practice processes really encapsulate the essential factor in project success.

So what is that essential success factor? I think it's how much you care about the outcome.

The project that enjoys the highest commitment to success by all the parties involved has the greatest chance of succeeding. There are some ironic twists of course, such as projects that receive so much funding that they lose focus, but losing focus is in itself a kind of indifference to the mission of a project, a kind of going through the motions for their own sake. Imagine a software development team that had unlimited funding. Imagine everyone on that team had a child with an incurable disease, and the project when finished would enable a cure for that disease to be found. It wouldn't matter how much money you threw at them, they wouldn't lose focus.

So many failures I've seen have come from project participants who had agendas that took precedence over success. The most common agenda is career ambition: we're going to use XML, or Struts, or REST, or cloud because it'll look good on my resume, not because that technology is what is needed here. Methodologies are somewhat different because they tend to have a broader scope of applicability than a specific technology, but they also provide scope for resume boosting but time wasting activities (e.g., producing design artifacts that aren't strictly necessary).

It's not that resume boosting, or trying out technologies and processes are bad things; they're good things. Trying out some of your pet ideas in the next project is a good thing too. But they have to be overruled by the drive to make *this* project a success.

Re:"Creative"

[nschubach]

It wasn't required, per say, but highly encouraged. You weren't going to get fired, but there was enough of an abstraction from pay raises that it could not be tied directly, but it was a factor. Whoever worked that one out was criminal in their own right. The practice has since stopped.

Actually it's been changed from [Donate > 0 || Donate == 0] -to- [Donate N || Not]. There's a minor distinction there, but the management is no longer penalized as long as everyone at least fills out the card saying "I choose to (or not to) donate." Previously, choosing to donate 0 was considered not responding and since they wanted 100% response they changed it to Donate or Not.) Come to think of it, I haven't seen one of those slips in over a year. Hopefully they stopped the guilt trip of saying "No" to some fabricated emotional picture of a starving kid sitting next to the "No" box.

Re:"Creative"

[bstender]

"There are three things you can pick: Fast, Cheap, Tested"

i think you meant to say "there are two things you can pick, out of these three"

a variation on the old; "we can produce quality, fast and inexpensive...you can pick two"

Re:"Creative"

[hey!]

Anybody who sits down and tries to do something "creatively" is a fool, unless he's doing some kind of art for art's sake. And if he calls himself an engineer, he's a fraud. Creativity isn't bad, it's just not an end in itself for an engineer. Creative is what you're forced to be when you've run out of proven ways of doing things.

What I think people may be reacting to is that you can't consistently do your best work if you're churning out one routine project after another. I think a project benefits from having a little bit of ambition built into it. Every project should have an element where you try something you haven't tried before, or do something better than you've ever done before. Each project should teach you something new, something that makes you better on the next one.

I am all too familiar with creativity as a desperate measure to rescue a poorly conceived project. While there is some satisfaction in pulling an astonishing design rabbit out of the hat, ultimately I find such solutions disappointing. There's always some compromise I wish I needn't have made. It's better to do the very best you can across the board and then stretch a bit beyond that, than to hammer together something you thought was impossible and shove it out the door.

Another way of saying this is you should attempt something on each project that might fail, but you should *choose* what this element will be rather than letting chance select your project's Waterloo.

Re:"Creative"

[wolfemi1]

NO! If there's one thing management absolutely CANNOT do, it's select the "best coders". Seriously, one person in particular I can think of keeps getting to be a higher-up in the software design process, then leaves me huge messes to clean up.

Re:"Creative"

[kurokaze]

Sounds like you've got a great job... my company is also CMMI Level 3 and all of tehse boilerplate templates are a huge freakin waste of time - especially when half of it repeats itself over and over.

On top of that, we have developers filling out documents that a BA or PM are supposed to fill out - leaving even less time to site down and figure out the problem. Yes, PMs seem to love handing off the docs to the developers - claiming that since we are building the system, we are the ones that should be filling out project documentation as well! Might as well give me their salary too!

Re:"Creative"

[Have Brain Will Rent]

The real problem is this:

1. An ever increasing demand for software
2. A larger and larger pool of developers whose talent follows a bell curve

These lead to a need to make a system where mediocre. and worse, developers can still generate useful product.

In other words you have to dumb things down so they average guy can handle it. That inevitably leads to a process where the people further up the curve are limited by the process itself. The further up the curve they are the more the process will limit them.

You can try and design processes that ameliorate this,e.g. giving more interesting bits to the more talented, but that only goes so far. The goal is homogeneity. Industry wants software to be like an assembly line and we all know how much fun they are.

Re:"Creative"

[obarel]

Ah, the magical "specification to code" machine. Don't you have one?

In my company we're one step beyond that - we believe that writing specifications should not involve any creativity (after all, that could be just as risky as using creativity when coding), so we have a "requirements to tested code" machine. During 2012 this will be upgraded to "rough idea to tested, deployed code". As the resulting code can be nothing but perfect (no creativity involved in any part of the process), you will soon find it in every safety-critical environment.

Now we only have to automate the idea-generation part and we're done.

Re:"Creative"

[RyuuzakiTetsuya]

well, my code worked, it wasn't the most elegant or efficient code, but we needed it now, now, now and now.

I can do fast(produced quickly that is; I'm pretty sure that while my code wasn't completely inelegant, I know of lots of cases where I took short cuts just to save my own ass and make deadlines), tested and cheap, what made me feel like some sense of scale was lost was when we were held to the same standards our internal ticketing system that tracked everything from cancellations to field maintenance work was held to. While I don't mind having my shoulder checked over, After IT's structural changes occurred, I'd be spending twice the time i spent coding wrangling our change management process. I'd write a module, be done an hour after lunch and spend the next day and a half tracking down approvals so I could deploy it live.

I was lucky in that the area i worked in was pretty much creatively free form and I had a lot of control over how projects went. I feel that there are different levels of impact work can have in software. If you're writing the back end handler for say, the online ordering management system that ties into the CRM and spits tickets out all over, sure, let's make sure that there's a lot of red tape there to make sure nothing breaks. This breaks, entire business work flow goes POOF.

But if you were doing the junior level shit i was doing, like writing or enhancing a CMS built for a handful of users where, the worst case scenario is that you're spending the next hour chain smoking while waiting for a tar ball to unroll while restoring a DB dump taken prior to roll out late at night when the user load is down to zilch.

Re:"Creative"

[hitmark]

Or one can use something like ADA, that will scream bloody murder when doing things that C would not even raise an eyebrow over. The right tools for the right jobs basically.

Re:"Creative"

[SixDimensionalArray]

This is why things like "Google Labs" exist, in my opinion. Just imagine if every company had a formal "Labs" entity within IS/IT. The only problem is that being able to have a free-flowing anything goes R&D-type department (which is separate from your ordinary every-day "structured" or process-based type department) usually requires corporate support, a company of a certain size, enough people, or enough discipline to separate R&D time from planned/organized/structured time.

I believe the core purpose behind Agile methods was originally to support what comes naturally to most developers, which is light-process and getting things done incrementally (which as I recall, is even a practice used in total quality management/TQM). So, really, many of us do "agile" intrinsically in some form, unless you work in a monolithic situation, a large company, one that makes complex or very expensive products that require lots of up-front planning/documentation, or one that requires traditional SDLC/waterfall/project-planning type processes.

If you built an aircraft carrier using agile methods, vs traditional project planning, I wonder, what would the difference be?

SixD

Re:"Creative"

[drb226]

Where are my mod points when I need them? Very, very true.

Re:"Creative"

[dave87656]

Bad developers OR bad managers.

In my experience Process Assurance means little. The board or software fails certification testing, and the managers over-rule the failure, and mark it "passed" anyway. PA is just a smokescreen and managers act like politicians.

The last big company I worked for had all kinds of process and the end result was that the code was written to meet the test and not the requirements. It was crap, usually, with no inspiration, frustrated developers which took forever to complete. I'm not saying you don't need quality control but each situation is different. If you're writing software for the Space Shuttle the most important criterium is quality (reliability), if you are writing the ticketing system for the Olympics, the most important criterium is the deadline, if you are writing for a fixed price contract, the most important criterium is the price.

Re:"Creative"

[Jane Q. Public]

Genuine Agile methodology includes testing. The problems you cite could not arise if adequate testing had been in use.

Any system can be used the wrong way, or misused. That is not a failure of the system.

But you basic point is no doubt correct. I did not claim that it always led to an improvement.

Re:"Creative"

[Ol Olsoc]

Process kills everything. My experience is that people end up serving a process, that the process somehow mutates into the goal. Eventually, actually doing something becomes a distant second to serving the process.

Re:"Creative"

[brrgo]

But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.

Is just an ignorant statement; because the two are not exclusive of each other. The creative mind may come up with a piece of code or test scenario solves many problems elegantly; and avoid a disaster where a disciplined mind would have just written to some poorly written specs (all specs are poorly written).

Re:"Creative"

[Entrope]

So, were you working for incompetent managers who had incompetent process people who were all working for incompetent executives, or were you working for some contractor who was more interested in billing hours than getting the job done? The behavior you describe tends to happen when "actually doing something" is not seen as an important business objective -- this can be due to regulatory strangulation, but more often it is due to poor or unethical leadership.

Re:"Creative"

[freudigst]

Real-time software for airplanes isn't particularly creative as opposed to robust. What validity does your comment have, then?

Re:"Creative"

[rjstanford]

Right. And maybe 1% of developers out there write code that is that critical. *Maybe* 1%. Most are writing web sites or internal applications that have nowhere near that kind of criticality.

Agreed. But fewer than 1% are writing solutions that require code-creativity. Most problems are well-understood, and even in systems that are creatively solving new problems, most code follows established patterns - or should, especially if you want it to still be working next year, or five years from now.

Re:"Creative"

[joebagodonuts]

Nope. My point is there is a difference between doing something, and having an over-abundance of documentation and detail describing HOW something was done.

Re:"Creative"

[mcvos]

I agree that's an important difference. When you're doing something, it is useful to verify you're doing it right, and when something went wrong, it's valuable to analyse it afterward so you can fix it. But some big companies seem to think that producing piles of documentation about that is a worthy goal in itself, but I doubt anyone ever really reads that stuff.

Re:"Creative"

[turbidostato]

"On the other hand, requiring project management entries, and change control tickets, and manager signoff, and hours accurals, etc etc etc for a bugfix that simply has to go to production ASAP is just silly, and doesn't help matters at all."

But then, when you are overwhelmed you'll damn your manager because he is unable to bring some more hands where they are so needed. It is your tickets and your hour sheets what will allow him to show higher managment that the case is that your work load has increased 50% last year so some more hands are needed instead of you being a lazy one hidden on dark corners to avoid working as needed.

The more ASAP and firefighting work you find yourself doing, the more all that managerial stuff is needed to bring a solution to the table.

Does anyone know the Happy Medium?

[AdmiralXyz]

I'm all-too-aware of this issue and how quickly it sucks the life out of you and prevents anything from getting done, but at the same time obviously having no process doesn't lead to stellar code either. My question is, do any of you work in a place where you think you've struck the right balance? What are you doing?

Re:Does anyone know the Happy Medium?

[CynicTheHedgehog]

Good issue tracking, short iterations, and daily stand-ups to discuss issues. Assign mentors to junior developers that you know need help, and have the mentors spot check the code. Let the more experienced developers choose the stack, define the interfaces, an create prototypes, and then hand those off as a template for other developers to follow.

Re:Does anyone know the Happy Medium?

[tripleevenfall]

Where I used to work, (I posted elsewhere here why I left development), part of the problem is that the attitude was always "another small task won't hurt engineering". "Another step in this process is not a big deal"... until there are so many of these tips and checks that you aren't doing anything else but these microtasks.

I think that most places have big problems going cheap on staff. They cheap out on testing staff. They cheap out on training for the support people. They cheap out on resources for environments. All of these things cause more weight to be placed on the developers.

And it's all by design... develppers are replaceable, in many companies' view - more are churned out of college annually and they only have a 2-5 year lifespan on average. Rather than expand budgets to reflect what development really should cost, they simply treat developers as disposable resources on a burnout/replace cycle.

Re:Does anyone know the Happy Medium?

[vlm]

My question is, do any of you work in a place where you think you've struck the right balance? What are you doing?

Think of the brilliance of the parental strategy of having once kid cut the cake and the other kid select the slice? The way that cuts down on squabbling about who got the bigger piece of the pie? Application of this principle to coding/coders vs testing/testers seems obvious... Doesn't it?

An inherently self stabilizing and self balancing system always works better than one managers best (possibly misguided) attempt at neo-authoritarianism.

Also for gods sakes make everyone in management read Fred Brooks book and fire anyone who disagrees. Absolutely timeless meta-advice to frame the problem.

Re:Does anyone know the Happy Medium?

[Eivind]

Brooks - and also The Pragmatic Programmer.

Agreed. Read both. Fire people who do not or can not understand it.

Re:Does anyone know the Happy Medium?

[Eponymous Coward]

I think you nailed it. The only thing I would add to your list is a dedicated QA team. And perhaps the ability for senior developers to veto management decisions on features and functionality.

Re:Does anyone know the Happy Medium?

[mooingyak]

I'm also a huge fan of checklists. Have a checklist for each phase... make it a requirement for everything to be checked off (or an appropriate excuse filled in).

Make sure you also take the time to remove things from the checklist. If you keep things around that were relevant to be checked a few years ago, but aren't anymore, then the checklist starts to become less helpful. When outdated items outnumber necessary ones, it becomes ignored.

Re:Does anyone know the Happy Medium?

[smelch]

What do you mean developers only have a 2 - 5 year lifespan? Do you mean with one company or until they burn out? I'm only 24 and I've been a developer in the corporate realm for the past 6 years with no intention of stopping, but I am averaging about 3 years per company with some overlap.

Re:Does anyone know the Happy Medium?

[tripleevenfall]

With that company, which is all they really care about.

Re:Does anyone know the Happy Medium?

[divisionbyzero]

And perhaps the ability for senior developers to veto management decisions on features and functionality.

Right, because senior software engineers know exactly what customers want, right? Those senior software engineers are out there on the front lines with customers every day and know exactly how customers use the software. Oh wait... no, they are holed up in their cubes (or at home) and don't want to be bothered by the external world while writing "creative" code. I get that product people should not be stipulating implementation details or the finer details of designs. Instead they should have a clear cut understanding of the business problem the feature is supposed to solve and have the capacity to recognize when a software engineer's proposed solution is or is not going to solve the problem adequately. I realize a lot of engineering effort gets wasted on useless features but the solution isn't to give engineers the ability to veto features. The solution is to get good product people, preferably ones with a background in software engineering.

Re:Does anyone know the Happy Medium?

[CynicTheHedgehog]

And perhaps the ability for senior developers to veto management decisions on features and functionality.

The thing is that as an employee you serve at the pleasure of your employer. If you are in a designer/analyst role, one of your functions is to identify and effectively communicate issues, options for resolving them, and the pros and cons of each, keeping in mind that there is no perfect solution and different parties have different goals and priorities. If the problem is, for example, a conflict between elegant code or elegant UI, and user experience is a top priority, then that's management's call. It helps to have a prioritized list of objectives from the outset that you can reference in these discussions, but usually that's not the case.

If, after you have explained the consequences of each option and given justification for your preference, management decides to stay the course, then all you can do is document the discussion in the issue log and do the best with what you have.

Now, if we're talking scope and release schedules, then that is a slightly different scenario. In that case the best way to make everyone happy is to have management (or the customer) prioritize their wish list, and then either specify a release date or a feature set, but not both. From there you can return a list of guaranteed deliverables in whatever timeframe is agreed upon based upon your professional estimate. But that involves negotiation and compromise, not vetoes or overrides.

Of course, all this is assuming that both sides are rational and reasonable. If that isn't the case then no amount of process or project management is going to make things go smoothly.

Re:Does anyone know the Happy Medium?

[nschubach]

And perhaps the ability for senior developers to veto management decisions on features and functionality.

Did you stay at a Holiday Inn Express last night?

I can maybe see Senior Developers being able to veto process changes... but features/functionality? Maybe my workplace is different, but usually when someone from management asks for a list of features, they expect it to be complete. It usually happens at the expense of the timeline, but they want all the features no matter how long it takes.

Re:Does anyone know the Happy Medium?

[JaredOfEuropa]

Success doesn't come from finding a happy medium, or from having more process or less process. It comes from understanding what you are doing and trying to accomplish, so that you can have the right amount of process that is lean and fit for purpose. The problem is that this understanding rarely exists. Managers often do not have it, and in larger organisations where you have mutliple departments working together you rarely find processes that have been designed with the full organisation in mind. Once you understand this, you can design processes that fit your organisation, and they will be different in every case.

It's rarely a question of too little or too much: people will ignore pointless processes and bypass hopeless ones even if there are only a few of them. And the same people will cheerfully stick to heavyweight, time consuming controls, if it makes sense for their work (aircraft control systems for example).

Re:Does anyone know the Happy Medium?

[Eponymous Coward]

First of all, veto privileges are fairly common in software companies. The best known example of this is at Microsoft.

Secondly, a veto needs to be backed with a well reasoned argument or you aren't likely to be a senior developer at the company for long. Prima donna opinions and attitude aren't going to be tolerated by management or fellow developers.

I agree with your last point though. Product people with a software background aren't likely to request ridiculous features. These typically come through sales who promise customers mind-reading software delivered on a pony.

I have only vetoed once and the feature came back in a later iteration in a much better form and was accepted by the entire team.

If management takes on a more dictatorial stance, they are going to lose their developers. Good developers are like good salespeople in that there are always opportunities out there for them.

Re:Does anyone know the Happy Medium?

[Kjella]

Funny, because I've worked another place where the attitude was always "another small task won't hurt (non-IT) operations". Instead of fixing and automating routines, the quickfix was always to add a manual check or correction or copy-paste and IT never got around to delivering solutions. The worst was when this lead to massaged data, because then this would take a data conversion which was always a huge high-risk job that would never, ever get done. Any operational error was of course because operations was sloppy, not because it was an incomprehensible maze of exceptions, corrections and special cases. The leader types didn't help, it was as much dom/sub as you get in the corporate world.

Sadly I think the collapse was so noisy from other reasons nobody truly saw ITs role in it, the high operational costs drove us away from the mass market to few customers and large orders - which is a very competitive market with a low tolerance for error. And because of the many lacks in error checking we had errors. So just to sum it up, it's not just other departments playing "somebody else's problem" with IT, I've seen IT play that game just as well themselves. That's what happens when one department gets to rewrite the business like it serves them, not as it serves the company. Your head of IT is probably a work suck hole, thinking the more IT do for the company the more powerful he is. Reality is you're just everyone's bitches.

Re:Does anyone know the Happy Medium?

[NoSig]

If your formal process takes up 100+ pages then you've failed already unless you are working at a NASA-type organization with unlimited resources and where bugs simply is not an option. Not a single person in your organization is going to know what those 100+ pages say, not even supposing a single person wrote them - that guy won't remember all of it. If your entire formal process cannot be reasonably explained in 5 minutes, and gone over in complete detail in an hour, then you don't have a formal process. What you've got is a mish-mash of pieces of a process where some bits are not followed correctly and some bits are ignored because no one remembers. Monstrosity 100+ page processes only exist because someone is too incompetent to know the consequences of their decisions on process, or worse they are competent and their goal is to have enough process that no one can be 100% in compliance with the process. Then when something goes wrong, they can accuse someone of failing to follow the process. There is also the rare third option of really having a project that simply must not have any bugs and where the immense resources required to achieve that are fully provided - that is rare indeed.

Re:Does anyone know the Happy Medium?

[divisionbyzero]

First of all, veto privileges are fairly common in software companies. The best known example of this is at Microsoft.

Secondly, a veto needs to be backed with a well reasoned argument or you aren't likely to be a senior developer at the company for long. Prima donna opinions and attitude aren't going to be tolerated by management or fellow developers.

I agree with your last point though. Product people with a software background aren't likely to request ridiculous features. These typically come through sales who promise customers mind-reading software delivered on a pony.

I have only vetoed once and the feature came back in a later iteration in a much better form and was accepted by the entire team.

If management takes on a more dictatorial stance, they are going to lose their developers. Good developers are like good salespeople in that there are always opportunities out there for them.

Fair enough. I suppose in large part one's perspective depends on where you've worked previously. I have seen just as many instances of engineers coming up with gee-whiz features that can't be sold as sales people promising mind-reading software. From the outside I think google suffers from the gee-whiz problem. Again, there seems to be a happy medium somewhere but it's not easy to find.

Re:Does anyone know the Happy Medium?

[khr]

I can maybe see Senior Developers being able to veto process changes... but features/functionality? Maybe my workplace is different, but usually when someone from management asks for a list of features, they expect it to be complete. It usually happens at the expense of the timeline, but they want all the features no matter how long it takes.

Sure, I can see Senior Developers having some veto into the features and funcionality. Most senior ones have gotten there because of their experience working with computers and users, and can have some valuable insight into that. I've been in situations where managers have asked for things and I've known that they won't fly. Or might be too costly to build, or might be incompatible with something else in the system, or might be redundant.

A senior developer should also be able to explain it with more than "nope, won't do that...".

e.g. a manager at a company that made medical software once told me that since every patient has a sex, we had to make it a mandatory field for users when making a new patient, that it would increase the accuracy of the data. But having worked with customers I knew that it wasn't uncommon for patients to come in and buy some medical accessory (like eye glass cases) and knowing the patient's sex wasn't needed for that kind of transaction. Or that data entry people talking to customers on the phone wanted to get the minimal information to schedule the appointment and sex wasn't needed to get that task done, but making it mandatory would mean they'd have to have callers repeat information, or write it on paper to enter it in a different order...

Re:Does anyone know the Happy Medium?

[Eponymous Coward]

Google is interesting. Douwe Osinga recently left Google and he has written about his experiences on the inside. Check out the "Google Thinks Big" section on this page.

He says this about Wave:

Wave started with some fairly easy to understand ideas about online collaboration and communication. But in order to make it more general and universal, more ideas were added until the entire thing could only be explained in a 90 minute mind blowing demo that left people speechless but a little later wondering what the hell this was for.

Re:Does anyone know the Happy Medium?

[drb226]

Sorry, but you were expected to die 1 - 4 years ago.

Re:Does anyone know the Happy Medium?

[rjstanford]

I think you nailed it. The only thing I would add to your list is a dedicated QA team. And perhaps the ability for senior developers to veto management decisions on features and functionality.

Vetoing features and functionality is often needed. What cannot happen with any success is vetoing business requirements. Admittedly those requirements should never be presented directly as functionality requests; still, writing beautiful elegant solutions to the wrong problems will put you out of business a lot faster than nasty crufty solutions that actually apply to the problem at hand.

Business users get to define the problem. Technical staff work with them to define an acceptable solution, and then implement it. Business staff then gets to explain the solution to the customers and find out what the next pain points are. Lather, rinse, repeat.

Over my head

[Apple Acolyte]

70% unit coverage? CCN complexity 20? Pre-sprint grooming of stories? This article is apparently not meant for a general geek audience. Can someone translate these terms into something that non-professional programmers can comprehend?

Re:Over my head

[Derkec]

70% Unit Coverage:
        -- You've written code level tests that flex 70% of your code checking for regression failures.

CCN:
        -- Technical term you can look up, but basically it's a measure of how many decision points are in a block of code. Less decision points is simpler. Too many and you may have something difficult to test and difficult for a programmer to understand. Higher complexity generally means more risk and a higher need for testing of various types.

Presprint grooming:
        -- A "sprint" is a time block set aside for development. Usually 2-8 weeks. The goal is declare what you're going to get done in that time and not change the requirements during that time. Between sprints, you can change your processes, "groom" stories (tasks that describe things in a user experience way generally).

Test driven dev:
        -- When writing a new feature, right a test for that feature first and you're close to done when the test passes and you haven't broken other tests.

Re:Over my head

[stewbee]

"70% unit coverage" means that your production code before being integrated should have touched at least 70% of the code. The complexity number is related to how many branching statements there are in a single function/program. You can look up how they compute it, but it involves basic graph theory. I have no idea what "pre-sprint grooming" is though.

Re:Over my head

[mevets]

70% - out of every X lines of code, 70% of them are actually executed in test. For example, code might contain something like:
if (month == 12 && day == 25) {
          printf("Joyous Season of Light\n");
}
However, without cheating, this holiday greeting would be untested on most testing days.

CCN - a way of expressing the logical complexity of program code to flag potential trouble spots. Google Cyclomatic Complexity

pre-sprint grooming - google agile development.

[imho]
Coverage is a good thing, although 70 seems pretty lowball.

CCN is well dressed snake oil; however it has a more interesting side effect - low CC# programs are often easier to read.

Agile is unpretentious snake oil.
 

Re:Over my head

[Anrego]

I have no idea what "pre-sprint grooming" is

It's from the agile crowd. Basically the idea is you have periods where you do just dev.. and periods where you look at processes and make changes to your use cases and such. Sounds weird to me (if I find something wrong in week 1 of dev, I want to fix it now, not plow through and go back later) ... but my understanding of it is foggy and even the most messed up approaches can claim a success story _somewhere_.

Re:Over my head

[heathen_01]

70% - out of every X lines of code, 70% of them are actually executed in test. For example, code might contain something like: if (month == 12 && day == 25) { printf("Joyous Season of Light\n"); } However, without cheating, this holiday greeting would be untested on most testing days.

Using techniques like IOC and Mock classes that snippet of code could be tested every test run.

Re:Over my head

[Anonymous Coward]

That is not TDD.

You are allowed to break the feature down into tiny pieces and write a test for a tiny piece before doing the implementation for that piece.

At the feature level it's more like "test in parallel" than "test first", unless it's a really small feature.

Re:Over my head

[HelioWalton]

Not exactly. During a sprint (A set period of time, but constant, ie 2 weeks always), you work on something from start to finish (lets make this button show a pie graph with blah blah balh), dev to test to documentation and whatever else. Pre-sprint grooming is essentially refining tasks that are yet to be done to ensure they can be done in a sprint, or ensuring you know how much of a sprint the task will take up. You might for example break down the previous "press a button, get a pie graph" into "press a button and get a pie graph with constant values" and "make the pie graph actually retrieve our current values" (Not the greatest example, but whatever.)
Of course, this all depends who you talk to...

Re:Over my head

[Imagix]

if I find something wrong in week 1 of dev, I want to fix it now, not plow through and go back later

From the Scrum perspective there's two problems with this (assuming the "something wrong" isn't because of the stories that are on the current sprint's backlog). First, this becomes a distraction away from the stories that are on the current backlog. The amount of work for this sprint was estimated based upon the stories that were committed to in the current backlog. Adding a "bugfix" will cause committed stories to slip and not be completed in time. Second, your bug may not be in anything that is immediately important. The current backlog may have been planned by the product owner to meet certain requirements for a deliverable by the end of the current sprint. Adding your bug may cause other stories to fall off the sprint which may have been critical to the deliverable. (Reminder, that your bugfix also means a draw on QA and documentation resources too)

Re:Over my head

[psmears]

Sounds weird to me (if I find something wrong in week 1 of dev, I want to fix it now, not plow through and go back later) ...

This makes sense to me - provided it's used for the right kind of cases. The case it'll likely apply to best is when, during a coding phase, you discover you don't know whether the widget should be red or yellow. Your alternatives are to:

• Enquire from the appropriate people (business sponsor, customer, stakeholder etc) what the right colour is—only to get caught up in discussions over whether the widget should actually be blue, and in fact should be square and not circular, and could you possibly make it blink twice when I move the mouse over it? ... or...
• Just make it red for now, but keep a note to raise this next time you are having conversations with the customer

The idea is that, by taking the second option, you keep making progress on the parts of the spec that you are sure about, rather than losing focus and being diverted onto endless discussions about minor matters.

Re:Over my head

[nschubach]

Obviously, the code he posted would be part of a method that accepted a date value so you could test it with any date. It was just an example to explain the point. I myself am having a hard time trying to find a case where you couldn't test 100% of your code, but I understood the explanation.

Re:Over my head

[isj]

CCN = Cyclomatic complexity number

It is basically the number of of different paths in a code block. It indicates how many tests you have to perform to reach 100% coverage. But more importantly it is also a reasonable indication of how complex the code is to comprehend. A cyclomatic complexity of 20 is quite high, unless the 20 different paths are 20 different cases in a switch statement.

Look for "cyclomatic complexity" on wikipedia - it covers the subject quite nicely.

Re:Over my head

[toxonix]

You don't need to be able to read and comprehend every article on /. Just go back to basking in the warm glow of your touchscreen, vehemently coveting its sleek aluminum shell, its pulsating Apple logo, and leave the thinking to the engineers who have to deal with this kind of stuff. In other words, STFU.

Re:Over my head

[Derkec]

True enough from a developer's perspective. But I was trying to keep it simple for the lay person.

Re:Over my head

[AuMatar]

Test an out of memory error, without crashing the test harness.

Test a race condition.

Test every error condition that could possibly occur in your program. This includes every OS error return of every OS call.

Test all network errors, on every socket read.

All of these are reasons why 100% isn't possible. High 80s/90s is only possible in languages and code that make heavy use of exceptions, and then are artificially high (you aren't really fully testing all the error generating code, you're testing a small amount of it but the automated tools can't tell).

In a nutshell

[theshowmecanuck]

Yes. When you need to spend almost as much time learning how to and using the tools, processes, and configuration than actually producing code... then yes. And no it doesn't always help make better code. A lot of the time it takes time away from making code good.

Re:In a nutshell

[Anonymous Coward]

the problem is always the same:

how can manager get bonus lowering staff costs?

they get cheapo developers and throw the top notch methodologies at them in the hope those will make up for the devs inexperience and plain incompetence.

that gives in turn a bad name at methodologies and tools, because cheapo devs cannot understand nor benefit fully from them. what use is a pmd output to a guy who's never got the difference between passing by reference and passing by value?

and there are, lots of them. just check any java forum.

yet, the truth is that some of the methods and tools are a good helper for good and also top notch developers. if you work with the tools, and not by the tools, you can catch way more common errors, be notified of piece of code that may need some attention because grew too much in scope and features and lot of other small things that make your life easier.

but it has to be a cereful choice, based on your experience and knowledge, not something imposed because it's the last buzzword.

The solution is simple...

[Old Sparky]

Get rid of management!

The one process to rule them all

[Tribaal_ch]

Karma be damned, this is relevant to TFA:

Re:The one process to rule them all

[syousef]

I'm passionate about programming, but not enough to have sex with my mother. I guess I'll never be a programming motherfucker.

Re:The one process to rule them all

[durdur]

I'm out of development now (retired early). There's nothing wrong with many of these principles per se. But IMHO Scrum and similar methodologies have got a cult-like hold on some organizations, where it's become something you can't challenge. If I hear another techie go on about pigs and chickens and stuff like that, I'm gonna barf.

TDD and getting your unit test coverage up are reasonable ideas but all they do is make sure you don't have goofy small problems like a sign inversion - they don't prevent the really big mistakes like design errors and security holes.

Code review in my experience can be a horrible pain in the ass. My recent project required two reviews before each checkin, and some reviewers were just too anal about it. It could take two or more days to do a review cycle and that's too long.

Re:The one process to rule them all

[Tim C]

Does it have to be *your* mother, or just *a* mother?

Re:The one process to rule them all

[smelch]

I'm passionate about programming, but not enough to have sex with my mother

I to have this problem. Perhaps would could arrange some sort of swap?

Re:The one process to rule them all

[balbus000]

His mother is the only one likely to come down to the basement.

Re:The one process to rule them all

[mcvos]

My son's mother would work quite well for me.

Re:The one process to rule them all

[smcdow]

Agreed. I just want to get the fucking thing working.

Once the fucking thing is working, I'm done. Ready to hand it off to some janitor, beancounter, or some such other process-loving dweeb. Let them work out all the corner cases and test coverage. That way, I can more quickly move on to some other interesting thing.

Re:The one process to rule them all

[LordArgon]

Interesting. I'm a big fan of scrum. Tellingly, I've also never had a serious discussion about pigs and chickens. I think there are a lot of useful tenets of scrum (e.g., timeboxing, group scheduling, bottom-up planning, iterative learning) and I've seen people pick-and-choose simply because don't see the usefulness of a tenet (even though they haven't tried them). If somebody is harping on some aspect of scrum, I would hope it's because they don't think your org has ever done it right. If your org really tries it, it doesn't work, and they're still mindlessly harping on it, well, yeah, that's dumb.

BTW, if code review is a horrible pain in the ass, you've got a culture problem, IMO. Assuming there aren't real concerns with your code, either you're not following the standards (perhaps there are no standards), the reviewer doesn't trust you, or the reviewer is a jerk. I wouldn't say any of those things reduces the value of code review; you've just got a larger problem that needs solving.

Re:The one process to rule them all

[AuMatar]

Formal code reviews are worthless- nitpicky, political, and way too slow. Here's the rules for a good code review:

1)There must be a web based tool to upload the code to, which automatically emails when comments are added.

2)Said tool must have an easy way to say "Approved".

3)Code reviews are to look for logic bugs and possible side effects. They are not the place to complain about how someone likes to name variables (unless an extreme problem) or that someone forgot a space before a + sign. That's just a waste of everyone's time.

4)Code reviews are 2nd in priority only to ship blocking bugs. Get them done fast so the guy can move on to the next issue.

5)Code reviews should go out to a wide audience (so they can get potential bugs from anyone who knows the code), but do not need more than 1 approval. That approval may not be from a junior.

6)If you have roles in your code review other than submitter and reviewer, you're doing it wrong.

This is why I left development

[tripleevenfall]

This is why I left development. After 5 years working at a software company I found that only 10% of my time or so was spent writing new code, which is the only thing I really liked about the job. The rest of the time was spent in meetings, wading through red tape, reviewing others' code, doing maintenance on the (junkpile) legacy system, doing remedial training for the front-line tech support staff, and the 5 million small tasks that have nothing to do with why I went into the career field.

Re:This is why I left development

[Vectormatic]

What did you switch too? i find myself doing pretty much the same thing (lots of non-coding compared to actual coding), yet i cant think of something else i'd really enjoy (and which would provide me with the same income)

I'm not fed up enough to switch careers by a long shot, but those meetings do get a bit old..

Re:This is why I left development

[tripleevenfall]

I went to work for one of our clients, administering the system I used to do development for. The starting salary was a bit less, but negligible. The regular hours and good work environment more than made up for the small loss of income.

Re:This is why I left development

[mevets]

I used to deplore meetings; listening to some preening jackass and thinking about how far we were getting behind by this. It isn't just the time sitting in the room, but depending on how bad it was, focus could be lost for the rest of the day.

Then I became freelance. Meetings took on a whole new significance. These jackasses paying my rate because I'm good at a few things; but rather than have me do those things, I'm sitting in a meeting, bored, but well paid.

Look at your contractors in the next meeting; if they aren't in the scapegoat chair, they are the only happy ones in the room.

Re:This is why I left development

[Anrego]

Maybe try a smaller company. Find a nice 5 to 7 person operation, bonus points if software isn't the main objective. When you are programmer 1 of 1, you find very little process, and what process there is, you define.

Re:This is why I left development

[vlm]

This is why I left development. After 5 years working at a software company I found that only 10% of my time or so was spent writing new code, which is the only thing I really liked about the job.

Something I've never understood, even after being in the game for three decades this year, is the proprietary field demands the majority of your time be spent doing all manner of foolishness. In the open source field, the programmers pretty much program 100% of the time they're "working". And the market has shown that any serious OS project absolutely crushes any proprietary project in reliability, featureset, security, documentation (At least in the internet / google / blogs / mailing list era).

Even the densest bean count-er-y management should be able to trivially see what I do at home "works better" than what is tried at the office. Yet they generally insist on doing it "wrong" despite staggering costs to the company. You'd think Darwin would apply, but the herd-like mentality of "everyone" copying whatever fad they read in a magazine prevents Darwin from weeding them out.

I've always found that puzzling.

Re:This is why I left development

[sosume]

wish I had mod points to give to parent .. so true.

Re:This is why I left development

[flightmaker]

After nearly 30 years working in electronics and software, getting made redundant for the fourth time, losing the missus, being sick, fed up, pissed off with working for idiots who sign any old spec to get the contract even if it is approaching impossible to meet, I decided I'd had enough. Next month I take the Part 2 driving instructors exam. I still enjoy writing some software - I did some C++ for myself only a couple of months ago to run on my Linux server but so far as doing it for anybody else is concerned they can all go take long walks off short planks. I don't care any more.

Re:This is why I left development

[Vasheron]

One cruicial difference between OS and commercial projects is the existence of tight deadlines and unstable customer defined specifications, which often have implications for reliability, featureset, security, and documentation.

Re:This is why I left development

[Kjella]

These jackasses paying my rate because I'm good at a few things; but rather than have me do those things, I'm sitting in a meeting, bored, but well paid.

What's wrong with:

These jackasses paying my salary because I'm good at a few things; but rather than have me do those things, I'm sitting in a meeting, bored, but getting paid.

You make it sound like employers care about the work to be done and contractors don't. That isn't true - or at least shouldn't be true unless you're one of the idiot employees who do unpaid overtime to finish. If it takes more hours of your life, of course you should be paid more. But then I hear the US has really abused the "exempt worker" so be roughly any white collar worker. Around here the definition is fairly strictly limited to management and others of exceptionally free positions that have great liberty to prioritize, delegate and decide when and how to work. If practically you have to be on time every morning, you're not exempt and even if you don't pay could easily get sued for back pay when they leave.

Another thing is that you get less stupid meetings. Consultants is a visible, billed expense so if you're measuring "whose time is more valuable" then the contractor is pretty high up. And unlike salaried people somebody is going to look at that bill and ask if you actually needed to be in that meeting. I did come to a place once and things weren't set up, and he was all apologizing. I was shrugging "Well the meter's running, if you want me to come back another day just say so but that won't be until I have another opening in my calendar." That's when he realized he wouldn't be apologizing to me, but to the PM for either billing extra hours or causing a delay in the schedule. If I don't get my work one, it's not my problem it's yours. Right now I'm employed but I still think the same way.

Re:This is why I left development

[hitmark]

http://www.ted.com/talks/jason_fried_why_work_doesn_t_happen_at_work.html

There is also a mention somewhere that a core lib of Amiga os was written by one guy over 2 days in virtual isolation. He only had to get one design issue clarified with a superior during the process.

Re:This is why I left development

[wrook]

Here's how I look at it (although I've also left development ;-) ). The author of TFA (and it seems you as well) is a victim of badly run software development. It is all too common, unfortunately.

The *only* thing that pays the bills in a software company (assuming it isn't a consultant) is functioning software. An hour in a meeting is an hour where you aren't creating functioning software. Therefore meetings should be minimised. On the teams I organised I insisted that meetings were *only* attended by management, unless direct developer feedback was required. Managers were required to summarise meetings in emails and to answer questions on an ad-hoc basis.

The only meetings developers attended were a 15 minute stand up meeting in the morning, where management was forbidden from attending and the bi-weekly planning meeting. The stand up was merely a meeting to coordinate who expected to be touching what code that day, or to request help from other members of the team. The planning meeting was a traditional "planning game" meeting from XP (estimation, or a request to split stories up into smaller pieces).

Status was automatically generated in the form of a graph at the end of each day. Either a story was complete or it wasn't. Developers checked off the story when it was completed (we also had QA on some of my teams who would check off when the acceptance tests that they wrote were passed). We usually used a wiki for this, but an issue tracking system would work fine. Managers were generally forbidden from asking "When will X be done" questions, because it was blatantly obvious in the charts.

Legacy code sucks, but every system has legacy code. All developers were encouraged to put untested code under automated tests if they were working with it. Any code that was under test and was not optimal for what the developer wanted was encouraged to be refactored. Developers were highly discouraged from modifying code that wasn't related to the story that they were currently working on (this enforces the scope of refactoring). Putting legacy code under test and refactoring it is a challenging skill and any programmer who doesn't want to do it isn't welcome on my team.

There was never a difference between bugs and new development. All work was prioritised the same way. A bug is simply a lack of functionality that was expected to work previously. This expectation should not affect its priority. Very often I have new work that has a higher business priority than fulfilling my expectation that something else already worked. All work is prioritised, developed, tested and tracked the same way.

On training, developers are not good resources for giving training on how to use the system. This is mainly because they don't know how the system works, only how it is developed. QA and/or Documentation are *much* better resources for giving training. If QA and/or Documentation don't know how the system works, then you have massive, massive problems that you need to address immediately.

I once worked in a job where preparing for meetings, attending meetings and following up for meetings took 24 hours a week. That was a crappy job and a fucked up development team. On my teams, apart from the stand up meeting (which the developers always insisted on, I never cared), developers were expected to spend up to 1 hour a week in formal meetings. I expected them to spend *all* of the rest of their time directly on producing code or helping others produce code. Everything else was handled by management (because, that's *why they are there* -- to make sure that BS is not stopping the developers from producing code).

Code reviews, whether formal or informal are directly related to development. It doesn't matter if you are doing pair programming, desk reviews or formal inspections. The end result is the development of a consensus on how problems should be approached. You are also developing a set of idiomatic expressions that will help everyone mesh properly wh

Re:This is why I left development

[Ol Olsoc]

We're hiring a new team of accountants to look into this.

Two sides to everything

[Haedrian]

On one hand, I can understand the creative process that goes into coding and the 'fun' you have in making it your thing.

However if you're likely to have millions of people depending on your code, which will alwso be modified by other people, then you had better have a good process as well.

I used to work at a company which required that pretty much all the important pieces of code have JUnit tests for them. Whenever someone else touches your code - which is bound to happen (I was modifying code which was written years ago - and the author wasn't employed anymore), it'll be a good thing if you know you haven't smashed anything.

So there's a time and a place for everything. If its very important code, yes please, strictness. If its something small and silly, then go creative on it.

Re:Two sides to everything

[w_dragon]

Except all you actually know is that you haven't smashed anything in a manner they thought to test for. In my experience while unit tests do decrease obvious bugs, they also decrease real testing after developing a feature where the developer runs the feature a few times to verify it actually works.

Re:Two sides to everything

[mrjatsun]

> However if you're likely to have millions of people depending on your code, which will
> alwso be modified by other people, then you had better have a good process as well.

A good process means nothing. Understanding why a process wants you to do something
is what's important. Checklists are bad. Making sure the right things get done is what
is important. Writing a test plan isn't important. Making sure you have good test coverage
is important.

I have been writing code professionally for over 20 years now. I have been at a handful of
different companies, and gone through a few handfuls of various processes (2167a was
the first one I had to use coming out of school). I've worked on very large projects and
very small projects.

For small to the small side of medium projects, it comes down to the people
working on the project, period.... In my experience, a standardized process doesn't
effect the success of a project either way. If you have a good project lead, and good people,
you will have a "process" or a somewhat standard way to do things. It doesn't have
to be documented in a formal document. But it will be there even if you don't realize it.

Large projects are a mess. You need a standardize process as painful as it is. One that
allows you to integrate early and often. The first thing you need to realize that the % of
incompetence rises the larger the project. It's easier to hide in a large project. It's easy to
hide in a process that generates a lot of information. You have to identify problem areas
in interfaces, and people, early and often..

Re:Two sides to everything

[AmiMoJo]

The problem with creativity is that you can end up with code which is difficult to debug or come back to in a year's time. I think it was Ritchie who said "Debugging is twice as hard as writing software, so if you write the cleverest code possible you are by definition unqualified to debug it". That doesn't mean no innovation. Chrome is a good example of some innovative but still well designed and written code.

Of course often you don't have a choice. The system I work with at the moment uses a custom language that was designed when the average program length was literally 3 lines. Really simple industrial control type stuff. Now we are churning out 2000 line programs and it is simply impossible to produce anything vaguely like good code. Embedded systems are the other classic example where limits on resources are tight.

Re:Two sides to everything

[angel'o'sphere]

In the open source field, the programmers pretty much program 100% of the time they're "working". And the market has shown that any serious OS project absolutely crushes any proprietary project in reliability, featureset, security, documentation

Seems you are bad in math.
If those programmes would programm 100% of theri time they had no: reliability, security and documentation. Because they had tome to think about it or to work on it.
angel'o'sphere

Re:Two sides to everything

[angel'o'sphere]

Writing a test plan isn't important. Making sure you have good test coverage
is important.

A test plan and having unit tests are for two different things.
How do you testg business requirements and system integration if you have no test plan?
How do you show the custoemr that your software is doing what it is supposed to do if you have no requirement that is matched by a test plan and a test protocoll showing that the test was successfully executed?
angel'o'sphere

Re:Two sides to everything

[mrjatsun]

Yes, I am aware of that. Your missing my point. I wasn't trying to iterate over
everything that a test plan will cover.

Writing a test plan doesn't ensure you have met any of your requirements or
than even if your requirements are worth a crap to start with.

The point being writing a test plan does not mean you have a good plan on
how to verify your product (from unit test through deployment and maintainance).
That is what's important. If you want write a test plan which references
to requirements, great. Usually required for any contracts anyway.

But it's the plan that is important, not the document. I haven't see a
checklist item yet for a good plan/bad plan. And I've seen many
really bad test plans go by which fulfilled their process checkmark.

Beh

[Hognoxious]

I've worked on projects where they had procedures as thick as a phone book, and it was still possible to write crap code. In fact, due to the incompetence of the person who wrote them , they sometimes pushed you into writing bad code.

On the other hand, some programmers produce good code, whether they're specifically ordered to or not.

Management should be the ones executin the process

[evanh]

That about sums it up. If they're so keen on doing it their way, then they can do it!

Mutual dislike between managers and coders

[140Mandak262Jamuna]

People who are passionate about coding, rarely make it to the management cadres, partly by choice, partly because they lack the other skills needed to be managers. So the management is filled with folks from sales, marketing and mediocre programmers.

I think may be the guru-disciple system of education practiced (allegedly) in ancient India might bear better fruits. Guru lead teams with disciples learning from them might turn out better quality code, but the system would be expensive in the short run and takes a while to take root. The quarterly bottom line obsessed corporate world is as far away from the system of stoic ascetic guru living in a hut in a jungle accepting princes as students romanticized in Indian mythology.

Re:Mutual dislike between managers and coders

[tripleevenfall]

There's a lot of truth to what you say. I am one of those. I think I was a "B" grade coder. Not a star, but adequate. I developed the skill, met deadlines and specs. It just wasn't my calling like it is for some.

Frankly, I don't think it's bad to have somewhat mediocre programmers in your management structure. We understand at least a good chunk of what developers are doing, and when we don't you can explain it to us and get understanding. If I'm a McManager who used to be in HR and has never written code, I'm not going to understand your basic needs as an engineering team. You won't be able to explain to me why a certain architecture isn't workable. I understand you and what you need 80% of the time and I can go fight those battles and leave you to code.

I think it's good to have mediocre programmers become managers if they have the management skills necessary (and aren't simply promoted because everyone else is irreplacable). Most of the time those skills are not common to the skillset of the best developers. It's better to have average developers become good managers than having good developers moved out of programming and into management, leaving only mediocre programmers writing mediocre code.

Re:Mutual dislike between managers and coders

[140Mandak262Jamuna]

When it comes to ego, the airforce combat pilots are really in the stratosphere. The ones with 5 or more kills who get to be called ace combat pilots? In the WW-II in the Pacific Theatre, you would find Japaneses aces with tens of kills. On the USAAF hardly anyone scored 10. How come?

In Japanese Imperial Air Force, ace combat pilots stayed in combat, much like star super duper programmers who eschew management promotions with disdain. Often times the ace would actually out rank the base CO-in-C. Even if he does not, he would be treated with so much of deference the ace would act as though he does. On the USArmyAF the moment a combat pilot scores 5 kills, he gets transferred out to Colarado to transfer his skills to a new crop of young pilots. Japanese aces eventually got killed, taking their skills with them.

If we can shoe horn air combat aces into training roles and play guru, we should be able to find enough non-ass***e super programmers to become productive useful gurus. BTW does anyone know the personal temperament of Donald Knuth or Dijkstra?

Re:Mutual dislike between managers and coders

[Xest]

But this is really a fault of coders, in this respect they're their own worst enemy.

If you implement one of the various decent Agile methodologies like Scrum then this is precisely what happens, when done right, with a team full of developers capable of being rather introspective and accepting where things went wrong then they really can learn off each other and their mistakes particularly at each retrospective stage.

Get a good Scrum with say a couple of really good programmers, then a few not so good programmers and over time the standard of the weaker programmers get pulled up towards that of the good programmers in terms of ability. The problem is all too many developers are so up their own arses they're just not willing to accept there are things they can do to improve, even those who are actually quite shit.

The attitude is rife, you only have to look at Slashdot for the blatant fanboyism on different topics that flows through to their jobs, and I have seen many programmers like this, there is a suggestion that perhaps the tool or methodology they've been using for the last 10 years isn't the best for the particular task at hand, but no, they just can't cope with that concept, the tool or concept they've always used is clearly the best thing in the entire universe and perfect for doing everything ever.

Ultimately the problem is a recruitment issue, and whilst I disagree that there is a shortage of developers in general, I think there's an absolutely crippling shortage of competent developers, who may not be the best initially, but at least have the mindset required to learn, to improve, and to consider other ways of doing things.

Re:Mutual dislike between managers and coders

[140Mandak262Jamuna]

Looks like the other poster in this thread worked in your shop. http://slashdot.org/comments.pl?sid=2143358&cid=36093330 You seem to have serious angst against managers, my sympathies to your staff, if you are really who say you are.

Re:Mutual dislike between managers and coders

[MarkvW]

Problem is that you've got to pay the good guru. Really pay him or her. Otherwise, you've got to find yourself a good guru.

Re:Mutual dislike between managers and coders

[AuMatar]

There's also the people who got promoted who thought they were star programmers but were really B or C grade. Those are the worst of the bunch- they think they're much better than they are, and don't listen to their programmers. The B and C grades who are realistic about their skill levels will defer on technical issues.

Re:Mutual dislike between managers and coders

[wrook]

In a much too long comment above, I advocate for a "coach" role in development. This is someone on the team who is responsible for the performance of the team, but who is not part of the management. It is pretty much identical to a sports coach. They spend their day watching what the programmers are doing, pointing out problems, offering advice, etc, etc. Of course, this has to be someone with a very deep understanding of programming. Player/coaches also work well, but in my experience manager/coaches do not work. This is mainly due to the issues you point out where the skills necessary for management are orthogonal to the skills needed for programming. A good manager is also way too busy clearing the path for their programmers and making sure that they can concentrate 100% on programming.

Process

[Anonymous Coward]

Only about 10% of programmers are "artistes" who create new things of beauty...process stifles those.

80% of programmers come to work more-or-less on time, sit where they are assigned, work using the tools and processes given, and produce most of an organizations output. Process is beneficial to those, in that it makes their work useful in the overall "whole."

10% of programmers should be fired before they do any more damage. A better HR process would help with those.

Re:Process

[roman_mir]

your comment is mostly wrong.

Only about 2% of programmers in a large company actually do most of the real work.

20% are good for chit chat.

78% should be fired immediately.

Re:Process

[internerdj]

I've come to understand this. When I go to my boss for a performance raise, he evaluates me based on me compared to everyone under him. It is my advantage to have the 78%. When he sees this and he says that we need to break the company policy on raises over x% so we keep this great employee, his boss evaluates his request based on my bosses perspective of best one out of 62 rather than one of 11. It is my advantage to have the 78%. Not to mention that bigger groups need bigger funding, so when I need something to do my job I'll have a bigger funding pool to draw from.

Re:Process

[konoking127]

It amazes me how many people believe they are part of the 'elite' 2% (or whatever BS percentage people like you choose). At the end of the day they are lucky to churn out the same quality work as most of the people around them. The only difference is your delusion.

Different time, different place, different plan

[bytesex]

The first version needs to written with passion. You then throw it away, and write the second version seriously. Also, code for your next php-driven social-thingy must be written with passion. Code for the satellite, not so much.

Re:Different time, different place, different plan

[Entrope]

Don't confuse impetuosity with passion. I write code with non-trivial safety outcomes if it fails to do its job. I manage to do it with both process and passion. Sometimes I even surprise myself and have passion for the more tedious parts of the process, because I know they make the product better given a certain schedule: The systems I work on are complex, and even though I am a smart guy, I cannot keep all the details on all the systems and subsystems I have to work on in my head long enough to analyze designs and changes without following some processes.

Re:Different time, different place, different plan

[dkleinsc]

Same here - I'm passionate about code reviews and unit tests. Seriously. Especially on the stuff I've written, but also for other people's code.

Unit testing is fun if you approach it with the right attitude: I'm going to explore a system and see how it behaves. Each test is a representation of the question "What should happen if I do this? What actually happens when I do it?" The first question is the joy of creating functional specs and design. The second question is the joy of experimenting.

I'm also passionate about code reviews because they're a fantastic teaching and discussion tool. You end up with conversations about "Why did you do things that way?" or "Have you thought about doing ..." which makes it easy to educate developers on approaches that others are doing.

Re:Different time, different place, different plan

[rwv]

Also, code for your next php-driven social-thingy must be written with passion. Code for the satellite, not so much.

I want to write a php-driven social-satellite. Should I be formal about my passion?

Or maybe I should switch from php to python.... import antigravity

Re:Different time, different place, different plan

[superposed]

Another relevant xkcd comic: Good Code

TDD

[Anonymous Coward]

"We all know by now that Test Driven Development is a best practice."

No, TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

Re:TDD

[tigre]

TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

No, TDD is painful, but it's a long-term investment. If you code once and never have to touch it again, you can get away with making it work right without tests. But if you want to be able to grow your system, you want something to make sure that you don't kill the old functionality in the process of building the new. And sometimes it's just helpful to spell out the expectations of new functionality ahead of time so that you know when you've achieved it.

Process for the sake of process is pointless. And if you have good reason to work around or jettison a given process, go for it. As I told the other programmers at my job, the only piece of our process we would probably consider an absolute necessity is source control. Code review is highly recommended, tests are very strongly pushed, formatting standards are there for good reasons, but there are times when they are not helpful.

The problem always comes down to whether or not your developers are adults with good decision-making capabilities. Process is too often employed as a way to allow you to get stuff done with people of average or less ability, but process also hurts them because they can't figure out when the process is unhelpful. And if process is imposed by fiat, those who could figure that out are frustrated by having to go through the motions. But if you have trustworthy people and you actually trust them, you institute the process for their benefit, and when it's not useful they know not to use it.

Re:TDD

[heathen_01]

I don't (usually) practice TDD. I do however write automated test cases for as much of the system that I'm developing as practical. Usually thats around 80% or more of the functionality.

The major benefit of automated testing comes with maintainence of the code. Making changes 6months or more later can be more difficult as you've forgotton all the intricate details and exceptional cases. This is where test cases can make your life eaiser and much more productive.

Re:TDD

[vlm]

"We all know by now that Test Driven Development is a best practice."

No, TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

Failures of TDD I've seen, censored for NDA reasons:

Neutering the tests by reviewing/paying/promoting people on a metric of test success rate graphs, etc, leading to purposefully idiotic tests, basically subverting the technique and actually using a different dev methodology (usually waterfall) with the TDD overhead as a millstone around everyones necks. Basically everyone gets a great review as the project fails.

Tail wagging the dog where the spec and the testsuite diverge fatally and permanently. "I don't care if your real world application fails, this code passed the test suite and it will be released as is!".

Test authors have no idea what the application is, and cannot even remotely guess what a sensible test would be, so they test the code they are planning to write instead of a "real world test" (obviously a problem in classified projects, but also big orgs where all the depts hate each other)

Everyone agrees idiot coders write idiot code. The insight is they ALSO generally they write idiot tests. Guess what, idiot tests often miss detection of idiot code and occasionally fail good code. Your tests / test authors have to be more sophisticated than the code / code authors, not just in theory or on an org chart, but in reality, and unfortunately they more or less come from the same pool.

Re:TDD

[TheRaven64]

The problem that I've found with TDD is that it encourages people to write code that tests the wrong thing. I like using it, because I gradually grow a set of regression tests and can be reasonably confident that I haven't broken anything, but I've seen a lot of people write tests that check for specific details, so you make a 5 line change to their code and then have to modify 100 lines of tests - not because you've broken anything, but because their tests were checking for implementation details rather than valid semantics.

Re:TDD

[angel'o'sphere]

"We all know by now that Test Driven Development is a best practice."

No, TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.

Obviousl you neither know what a best practice is nor what TDD in fact means.

Do you really think your non TDD way of doing stuff does not leave bugs behind that are not picked up? So, how do you magically make sure that all bugs are picked up by test cases ... just wondering?

angel'o'sphere

Re:TDD

[tigre]

I don't think you understood a word of what I was saying. The whole point of TDD is the reality of code change.

Re:TDD

[tigre]

The problem that I've found with TDD is that it encourages people to write code that tests the wrong thing. I like using it, because I gradually grow a set of regression tests and can be reasonably confident that I haven't broken anything, but I've seen a lot of people write tests that check for specific details, so you make a 5 line change to their code and then have to modify 100 lines of tests - not because you've broken anything, but because their tests were checking for implementation details rather than valid semantics.

I don't think it's TDD per se that encourages such bad practices. It's a complete misunderstanding of the purpose of tests. I guess I'm not big on low-level unit testing because it generally ends up breaking things down so much to the level that you do end up testing implementations rather than interactions. And thus you end up with Development-Driven Testing, an anti-pattern if I ever saw one.

So the problem is again with the application of processes and best practices without any understanding of what they're for and why they work. Because the people doing the work aren't good enough to understand and the people managing them aren't good enough to explain.

Measuring Code

[rwv]

The main issue is that measuring whether code is good or not is impossible. I promise you that I can write code that has the prescribed level of unit testing and complexity that also doesn't work. Software reliability/dependability was a problem in the 1960s. It's still a problem today. No silver bullet, and all that.

Re:Measuring Code

[heathen_01]

While I agree that there is no solver bullet, I would like to think that some of us have moved on from the 1960's.

I promise you that I can write code that has the prescribed level of unit testing and complexity that also doesn't work.

What doesn't work, the unit of code tested? In that case your unit test is just plain wrong. If its the application that doesn't work then that is a different issue, one which unit testing is not designed to address. Don't fall into the cargo cult trap - the right processes need to be used the right amount for the right reasons.

Re:Measuring Code

[rwv]

I think we agree. Part of my point was that as a developer, I can route around the processes while writing either good or bad code. Generally, I write good code. Occasionally there are mistakes which eventually get caught by process. On occasions, code with bugs can survive "in the wild" for an amazing amount of time before it's discovered.

In general, I think code inspection, test coverage, and integration testing are a powerful recipe for consistently developing high-quality code. I've never scrummed or worked a project following an agile process. In general I've made a career working towards the next contractual deadline... which is actually what the article says to do when there are contractual deadlines.

But in the space between... there are definitely places where worrying about code inspection and testing is idiotic. When you are mucking around with a new concept... KISS. Keep It Simple, Stupid. No burdensome process required in those instances. But with any piece of code, I wouldn't dare assign it the label "mature" without the inspection, coverage, and integration quality milestones. And I'd caution that even "mature" code might have defects waiting in the lurch.

Re:Measuring Code

[Jack9]

> I promise you that I can write code that has the prescribed level of unit testing and complexity that also doesn't work.

Then your testing is woefully inadequate.

Inappropriate reliance on process...

[shic]

What really cripples things is when process is deemed a substitute for understanding the specifics of individual situations - where a one-size-fits-all-problems approach is adopted and imposed - usually by people who have no practical experience with the processes they espouse.

If software development could be successfully reduced to a process, I'd have automated it. Where there's a considerable burden of process, either the process is inappropriate - or developing the software itself is inappropriate as it amounts merely to re-inventing the wheel... an exercise in task creation that benefits no-one.

We should think of software development techniques and apply them judiciously - and the more techniques a developer masters, the wider their skill-set and the better they will adapt to new challenges. The critical question that needs to be asked is this: why is a technique being used and is it providing tangible benefits? If this question can't be adequately answered, everyone involved is wasting their time.

Re:Inappropriate reliance on process...

[gbjbaanb]

you got that right - I think most of these 'best practices' are more fashionable things rather than ways to achieve excellence.

I've been writing software for some time, and I used to write code well before TDD was invented. My codeworked back then, so why would I need TDD now? Sometimes I think maybe its because the tools and development practices are geared towards that - we do a lot of fast-development methods whereas once upon a time we'd take time to do design and validation before getting stuck in. Today it seems getting stuck in is the first and only thing, so I'm not surprised we need TDD to fix that, and that leads me to wonder if the time spent writing tests makes the fast-dev cycles so much slower that you might as well be faster overall doing it the old-fashioned ways!

I know we have started doing some of these things, not because the code is bad and needs fixing,but because someone read about it on the internet and thought that we shoudl do it because everyone else is doing it - ie, entirely due to fashion.

Re:Inappropriate reliance on process...

[MonkeyRobo]

Quite. Having a process is not, and can never be, a problem. Having a bad process is a problem. If your process involves regular retrospectives, where everybody can contribute to making things better, then your process should tune itself over time. (Well, assuming you are empowered to change things for the better, that is.)

Darth Vader, bring balance to the force

[laffer1]

There is a balance between process and actually getting things done. Most companies never find it. The biggest problem I have is when this is used to audit progress on a task. For instance, I am a consultant for a company just starting to get out of startup mentality. There's been a push for more process and they've implemented their version of Agile. One of my first tasks was to write a script to dump data from Agile Zen so they could run reports on how fast developers are finishing stories.

Fail #1: All stores are not the same size!
Fail #2: Each team has different rules for stories and use agile zen differently. How do you compare them?
Fail #3: About half my time is spent in meetings, writing stories, fighting with QA to test things (not enough people). I'm supposed to do the same amount of work in half the time? ...
Fail #n: You're doing it wrong

The worst part is that when we had PRDs, at least there was a big picture loosely discussed. Now we get stories that don't integrate with each other and a big ugly mess.

Re:Darth Vader, bring balance to the force

[Spy der Mann]

I completely agree. I've come to believe that Agile development is a fad invented by some marketing genius to get big loads of buck from gullible enterprises. While TDD might be useful for, say... a linux kernel module, there's very little use for it in your standard "make me a module which reports in detail our budget surplus and deficits" project.

It's much more efficient to hire a small team of beta-testers available on-demand ("Jim wrote this new model, can you test it please?") than wasting hundreds of man hours per-month in "agile" development.

Re:Darth Vader, bring balance to the force

[angel'o'sphere]

It's much more efficient to hire a small team of beta-testers available on-demand ("Jim wrote this new model, can you test it please?") than wasting hundreds of man hours per-month in "agile" development.

And you have to retrain those "beta testers" every new release?
If you are wasting 100ds of man hours a month in agil development ... then you are obviously not agil, or?
Seriously, if you can not write test code for your code, no matter if you do it before coding (TDD) or after coding (unit tests) then you can not code at all.
angel'o'sphere

Passion isn't important

[Uruk]

Passion isn't important. Cost and risk are important. The processes are put in place to (attempt) to minimize cost and risk associated with software development. Experience teaches us that cost and risk are very high when building software.

When it's your money paying for the development effort, feel free to structure it so that you can chase your passion.

I sympathize with the idea that this kind of bureaucracy can suck the life out of developers, but guys, this is work. If it were that fun, they wouldn't have to pay you to do it.

Re:Passion isn't important

[Entrope]

People who have passion about what they are doing are usually much more productive than people who view the job as a way to get a paycheck. They also tend to want to stay longer, meaning the organization saves on replacement and retraining costs. There are also just wrong processes (of which "too much process" is a case) for an organization, so organizations need to do cost/benefit analyses on their processes.

Other than that, I agree with you. Developers have three choices when they are faced with a process requirement that annoys them. From easiest to hardest: (a) suck it up and do it because it's part of the job, (b) quit and look for greener pastures or (c) try to convince the right people to revise the process. (Sometimes (c) is easier than (b) if the process is new, and the cumulative effects of (a) can make (b) easier when summed over many annoying process requirements.)

Re:Passion isn't important

[Derkec]

I firmly disagree. Passion is key. People who don't give a damn and people who don't enjoy coding tend to write crap. We try to hire for passion and smarts (knowing both are hard to interview for).

Passionate people given good processes and tools are ideal.

Re:Passion isn't important

[LordLucless]

And if you kill the passion, your cost (of hiring new developers because you drove the old ones away) and risk (of losing corporate knowledge in process) increases. So you can say "my money, my rules" all you like, but actually, it's reality that dictates the rules. And reality says if you burden people under a weight of bureaucracy disproportionate to what that bureaucracy is intended to accomplish, they'll leave. And if they don't leave, they're probably afraid of not finding another job, indicating that they're not good enough at their job to be confident in their abilities.

Like pretty much everything, it's a balancing act. You need to provide direction, or the goose is going to go wandering around the yard instead of laying its damn eggs, but if you stifle it too much, the thing's going to croak.

Re:Passion isn't important

[jbengt]

People who have passion about what they are doing are usually much more productive than people who view the job as a way to get a paycheck.

I work in engineering, not software, still, the last time I encountered a passionate IT employee, he got fired for taking long, passionate lunches at the hotel down the street with an electrical engineer and fixing their timesheets to cover up their absence.

On a more serious note, you don't have to choose between passion and apathy. I'll take conscientiousness over passion any work day. Passion can at times be dangerous to objective thinking, partner relations, or a professional standard of care.

Re:Passion isn't important

[sjames]

The opposite of passion at work is known as "phoning it in". You ACTUALLY advocate 100% phoning it in in all workplaces?

MS tried that once and the result was Windows ME.

Re:Passion isn't important

[GreyWolf3000]

I don't think you're aware of this, so even though I think I'm stating the obvious, I'll do it anyways. Here it goes:

Passion != fun

Re:Passion isn't important

[jawahar]

I think using the right tools will circumvent the need for process/bureaucracy.

Yes, for proprietary software.

[VortexCortex]

As more and more DRM "solutions" are deployed, and the DRM systems themselves get larger, more complex, and more expensive, I think you'll find it not far from the truth that: "Process killing" IS the Software Industry.

Only individuals can be passionate

[joh]

And most software is not written by one individual.

As soon as you have an actual team writing software and as soon as there are others telling this team what they have to code, you need every bit of control you can get. There's no way around that and every anecdote "proving" you can get away with passion and good coders just proves that you can be lucky.

Devs should own the process

[Derkec]

This is really the key insight of most Agile methodologies. Development should own the process and change it to suit their needs during regular retrospectives. The team (not the whining individual) should be able to say, "You know what, I think we're not getting bang for our buck out of this many unit tests, let's shift to 50% coverage." As long as that same team is taking ownership of the regression failures and making an informed trade-off their comfortable with, all is well.

If you get a good team together, you're going to get good code. You'll get better code if you empower them. Experienced and good teams will usually have a lot of these processes and tools in place because noticing things like high code complexity automagically alerts them to "bad smells" that can be examined and either accepted as necessary or invested in to address or test more thouroughly.

Generally, I think development is most fun when you're on a new project and don't give a damn about breaking things. Then it's pure creation. But once an app is older and there's some weird code you're staring at you have to decide, "is this probably a bug, or is this a bug fix for some weird situation or platform?" That's when you wish that the guy having fun three years ago had written some damn tests.

Re:Devs should own the process

[gbjbaanb]

That's when you wish that the guy having fun three years ago had written some damn tests.

But today, the guy who writes the tests (and hopefully keeps them updated) is the same one who agrees with the "my code is its own documentation" principle and doesn't put a single comment in there. He won;t have written the design specification either, or the install guide so you still won't know how the damn things works.

There's no silver bullet for doing things right, TDD isn't any different. Good devs can tell bad code from looking at it, they don't need CCN reports to highlight it, and as someone else said - I can write code that passes all unit test coverage requirements and still doesn't actually work.

You're quite right about the difference between new code and maintenance - sometimes I think the onyl answer to it all is to put the 'rockstar' coders on maintenance duties and let the old sloggers work on the cool new stuff. Too bad things are often the other way round!

Re:Devs should own the process

[khr]

same one who agrees with the "my code is its own documentation" principle and doesn't put a single comment in there

I sure wish more people agreed with that and didn't write so many comments in code... Last week, in fact, I introduced a bug in some code because I foolishly read the comments for someone else's code... Too bad the comments weren't updated when the code was changed earlier this year...

I run into this frequently on this 10 year old code... It's pretty heavily documented and a fairly significant portion of the documentation is simply wrong. It may have been right at one time, but the code's been changed so much over the years the two no longer match.

Re:Devs should own the process

[gbjbaanb]

Some people think comments are just a bit of extra information.
Some people think comments are as important as code.

Guess which group writes the best code... ?

My point is with people who write *no* comments. none. nada. zip. zilch.

competition

[roman_mir]

Like in everything else, from economics, to love life (whatever it means for different people), there needs to be competition in software development. This means performance based compensation, this also means getting rid of those, who cannot achieve.

Developers should get goals in terms of business questions, then they should be let to do their own estimates and they should 'bid' on projects. There should be tracking of the success of different projects - from accuracy of estimate time and cost to number of features, to number of bugs per unit of time, to overall user satisfaction, to amount of time it takes to fix a bug or add a feature (this really would measure complexity and cleanliness of the code), to the average amount of time it takes the support team to deal with user issues (cleanness and usefulness of documentation.)

It is possible to do this, of-course large organizations rarely if ever achieve this level of 'enlightenment' even to start thinking about things this way, that's mostly because people do not understand economics, it's not about software.

Re:competition

[geekoid]

You seem to fall into the same trap other developer do. Mainly that you are an island. You are not an island. Other need to maintain your code. Add new features, etc.

Re:competition

[roman_mir]

Didn't I explain that points would be added for code with better maintainability and more documentation?

Anyway, as to being an island - for the past 2 years I've been an island. Doing all my projects all alone, just me, a store chain and 50 suppliers.

Not the problem

[asherlev]

Process isn't the problem, environment isn't the problem, language isn't the problem. The problem is that everyone looks to all of these things as the silver bullet that's going to "fix" software. Test driven development doesn't make good software any more than I-495 makes New York City. There is no silver bullet.

Distortion: construction is free

[dazedNconfuzed]

In most engineering disciplines, the process of actually building something is long, hard, expensive, and persistent. If the project is build a bridge, you spend a lot of time making sure the design is right; why? because the process of actually building the bridge takes months or years, costs many millions of dollars, and once built is not easily replaced. There is no room for error, so process is taken very seriously as a central part of ensuring timely cost-effective correctness.

In software, the process of actually building something is instant, easy, free, and transient. Type "make all" and go get some coffee; find a bug? tweak a couple lines and do it again. This distorts the development process; "process" gets snubbed as a costly distracting annoyance instead of the means of assuring that what gets delivered is correct, because it's just so dang easy to fix and rebuild in seconds ... losing sight of the long-term cost of not doing it right the first time.

Re:Distortion: construction is free

[Tridus]

A bridge is also a fixed, well-known thing. It's not going to change radically in design between when you start and when you finish building it.

Software on the other hand is written for customers who themselves don't know what they want, in a market that is probably rapidly changing. It WILL change between the start and the end of the project in a lot of cases. Sometimes it changes because the customer changes their mind. Sometimes it changes because the market changes. Sometimes laws change. Sometimes the customers were just flat out wrong in everything they told you and the entire design is wrong as a reuslt.

When you're dealing with that, process does just get in the way.

Re:Distortion: construction is free

[John Hasler]

In software, the process of actually building something is instant, easy, free, and transient.

Then you should not use processes designed around the assumption that building something is slow, difficult, and expensive (which is not to say that you should not have processes).

Re:Distortion: construction is free

[trout007]

Good post. In Civil and Mechanical Engineering you typically have two choices when building important structures. You can use established building codes. These cover almost everything you need to build a structure and have made the ability to build safe structures using standard components and techniques very economical. Such standards are ASCE 7 which gives you standard loads, AISC Steel Construction Manual which gives you standard materials and shapes and techniques to analysis the structure. And there are the AWS Welding codes that give standard ways of welding steel joints that all certified welders are able to do. The downside of this is that you are stuck with standard looking buildings for the most part.

If you want something extremely cool or complex you have another option. You can do all of the engineering yourself. You can come up with your innovative design and then do a full analysis that everything will work out. You may have to build wind tunnel models since standard load data won't apply. You will have to do your own welded joint calculations and test them to failure. Then certify that the welders can make those welds reliably and most likely do x-ray inspection on those welds to verify they are sound.

Both of these options are available to a customer for the structure. A cheap off the shelf cookie cutter version or a very expensive original idea and all of the engineering that goes along with it.

Re:Distortion: construction is free

[geekoid]

except programs also take months, cost millions of dollars and aren't easily replaced.

"In software, the process of actually building something is instant, easy, free, and transient"
Yeah, if you're writing hello, world program.

Re:Distortion: construction is free

[geekoid]

" It's not going to change radically in design between when you start and when you finish building it."

not true.
In the process things gt changed, but it's part of the process. This includes new features added while building.

You don't see it becasue:
A) engineers can attach real world costs to any change and need to get someone to sign off on the costs.

B) You don't build bridges, so you don't see any development issues.

Properly engineered code would mean when a change comes up, you say 'it will take this long, and cost this much'. PLease sign the document stating this. You make the sign off a requirement for the developer. Meaning they get fired or fined. This way you protect them and have someon to take responsibility.

When you go to the customer and say, sure we can do this,. It will take 2 months and cost you 20K extra. they may rethink their 'minor' change.

Since the industry doesn't have that, I do it in email. They few times someone tried to call me out, I just forwarded and email and said 'I told my manager it would take this much longer and cost this much more and he approved it.'
While not a lot of legal protection there, it has worked.
Right now I work with a bunch of civil engineers. They get all the same crap software developers do BUT they have legal requirements for sign off, and protection if someone is trying to build something that isn't structurally safe.

When dealing with that, the process will help you because as you go you will have a base of knowledge about how long something takes and what it costs.

If someone wants a , without changing in parameters you need to call them out. Have a process means you have factual backing you can document.

They say that if software developers built house, civilization would fall when the first woodpecker should up.

I wish that was true, because after the last house fell, we would fix it. Instead the industry just slogs through this miasma of low quality slap it together crap over and over again.

Re:Distortion: construction is free

[Max Romantschuk]

When you're dealing with that, process does just get in the way.

All development follows some process. The trick is to find a process which supports the needs of the (development) organization. Process that gets in the way is the wrong process. This doesn't mean it's impossible or pointless to formalize the bits of ad-hoc process that are determined by trial and error to work.

s/w dev != manufacturing

[sridharo]

Processes are defined in the manufacturing industries to prevent slippage of 'achievable/defined' quality. Unfortunately the extension of processes from these industries (six sigma?) to software development doesnt reap the intended benefits.
Blame it on the guy who thought the role of a programmer and the line technician are the same!

fun and games until the revenue stops coming in

[alen]

i've seen supposedly passionate coding in practice. good people but it's almost impossible to work with them since they social outcasts.

a software release turns into a 4 hour nightmare because everything worked on their dev laptop but in the real world it doesn't work and they have to figure it out. but they don't give you all the information and you have to wait on the phone at night while they tinker and fix it in production. meanwhile employees can't take customer trouble tickets.

or the billing system can't make customer bills because it's now multi-threaded but there is no system to control which bills are processed when and the different threads lock each other out in the database because some selects lock down most of the table. yet the code is good and can't be changed

Cholesterol

[geoffrobinson]

A wise elder at a defense contractor once told me that process was like cholesterol. You can't have none. But too much will kill you.

Detroit: Where the weak are killed and eaten.

[bshensky]

While all you process-laden geeks with your SCRUM teams and unit tests are getting the life sucked out of you in the Valley, we here in Automation Alley live by the seat of our pants, Montgomery-Scott-style, delivering barely-passable code on time, with the enhancements and 80/20 fixes to be pushed ahead of next year's model refresh, and a keen eye on the Bugzilla board, waiting for our user community to serve up the bug list. Thrilling, I tell ya.

Didn't yo mama ever tell you not to buy a first-year model?

I just wanna have fun

[SlashRdr]

Of course doing everything to an extreme isn't a cost effective way of working. If he has his way the other developers on his team would leave or kill themselves: "But, for example, maybe junior (or specialized) developers should be writing the unit tests, leaving the more seasoned developers free to concentrate on the actual implementation of the application.". Sounds a little arrogant to me.

Coming soon to a dev team near you...

[Enter the Shoggoth]

Are you finding the majority of your code is actually made up of unit tests?
How can you be confident that your tested code is safe when delivered into the clients hands?

Introducing Meta-Test(tm) - with this breakthrough in software engineering methodology your drones can increase their coverage numbers without writing a single extra line of application code!!!!

Hurry! Don't miss your spot at our upcoming seminar where you will receive details on how to send your drones to our new training sessions that you can bill back to your clients and put your project even further behind schedule!!!

discipline & partners

[achowe]

I've always been passionate about my code and attempt to test as much of it as possible, but I've found what saves on testing / process is discipline in coding style, commenting, consistent use of language idioms, good function / variable names, checking all function returns, lots of debug logging... Lots of little things that a programmer can do to help them read and diagnose their code during development and testing. It also helps to work with a "testing" partner (an element of Extreme Programming). I don't formally follow Extreme, but I do know from experience working in pairs or on occasion threes works really well: let the passionate go nuts with the support of a tester who should be equally passionate about that aspect of code (it is possible to find), and maybe the third is simply a mentor to overview and act as a sounding board for design ideas and problem solving. My partner and I will often discuss and compare ideas again and again; I'll code something, test it, then pass it to him to test it in depth. In many ways it becomes a game to write the code clean and for him to fault it. http://www.dailywav.com/0600/kaboom.wav

I like reading good code.

[drolli]

And i love programmers how manage to encapsulate good ideas in a way that they are not harmful.

Look at government for process gone amok

[Tridus]

If you want to see what happens when you keep layering more process on to try and fix perceived problems, just take a look at how any bureaucracy makes decisions. Government is well known for doing things that don't make any sense in the real world, and the reason why is that our (I work for a government) decision making isn't based on reality. Or even decision making, for that matter. It's all based on following a process and doing whatever comes out at the end. With enough process and a process-focused group of managers, the results don't even matter. Literally, this is the way people think: "Whatever happens has to be right because we followed the process, and if it goes wrong it's not our fault because we followed the process!"

There's always going to be some level of organization and control that is needed when several people are working on a common project, but you have to balance that with the need for people to actually think. When you pile up process on process on process you remove that and instead get a bunch of people who can't see what's actually going on because they're too busy complying with 14 different processes.

This particularly applies to software development. It's really important for a team to pick out what they specifically need for their project and what works for the team. Throwing in something like (for example) Test Driven Development just because it's currently a buzzword-compliant process won't get you anything except a lot of wasted time. Sadly, bad managers love to do exactly that because adding more process is itself a good CYA (Cover Your Ass) process.

Methodology == Religion

[syousef]

The real problem is that methodologies are followed, taught and practiced like religion instead of science. A new buzzword or buzzphrase and a new way of doing things will never substitute for thinking through whether what you're doing is actually working to attain the goal you've set out. Unit tests for example are brilliant when the developer gets control and is honest about using them. This almost never happens in practice. In practice unit test coverage becomes some bureaucratic check box to tick. What's worse is that there are dozens of industry expert consultancies wanting to sell you their own particular brand of excrement which may have some merit but the way it's sold - as a fix all for everything - will never be anything but a pile of poo.

Re:Methodology == Religion

[geekoid]

" taught and practiced like religion instead of science. "

absolutely correct.

I have had many success using unit testing.

Re:Methodology == Religion

[SoftwareArtist]

Exactly. Just look at the list of "best practices" in the summary:

We all know by now that Test Driven Development is a best practice.

According to who? I certainly don't know that. There are a limited number of cases where TDD makes sense, and a much larger number of cases where it doesn't make sense and shouldn't be used.

And 70% unit test coverage.

Who came up with that number??? Really, it depends enormously on what you're doing. For low level computational code, I feel uncomfortable with anything less than 100% coverage. For UI code, unit tests usually don't make sense and I'm quite happy with 0% coverage.

And keeping your CCN complexity numbers below 20.

You've got to be kidding.

Re:Methodology == Religion

[sjames]

Frighteningly, your subject line can become all too true. When "Process" takes over, you can actually hear employees earnestly discusing "the manual" in such a way that if you replace "the manual" with "the Bible" and page/chapter references with books and verses, it still seems to fit.

I have seen that crazy religious cults can be recognized by the way they attach some special significance and odd definitions to otherwise common words and give specific instructions for what are otherwise common everyday activities.

Passion

[SwashbucklingCowboy]

Depends on what the software is doing. If it's controlling my car or my mom's pacemaker I want the developers having a passion for making sure the code is 100% correct.

"process" should help focus competence but

[a2wflc]

Too often "process" and tools are used as a substitute for competence. It's VERY easy to follow process and design patterns and use all the latest tools and end up with a brittle design & implementation. It's also easy to create a very similar design/implementation that's is robust and extensible and easy to maintain. And 90% of the people I've worked with over the last 10-12 years wouldn't be able to look at them and determine which is better (they'd most likely say the first is better if they were told the name of the methodology and design patterns used). After a year where the first fails repeatedly (crashes, can't meet new feature deadlines, etc) and the 2nd has been a huge success, they couldn't tell you why.

Developers

[mbrod]

The creativity should come from Designers now. Developers just implement the design. It is the Designers that need the passion, Developers don't have that role anymore.

Re:Developers

[jeffc128ca]

Developers just implement the design

You sound either insane or too young to know what goes on in development. Development encompasses a lot of things and many projects don't even have a "designer" involved at all. Properly tackling how interfaces connect back to back end databases and other services takes smarts and passion. There is a whole host of things that go on behind that pretty screen that you "design". If you don't have smart people thinking creatively how to do that the project will be crushed by bugs and poor performance.

Budget

[mrops]

My problem with process has always been budget. Folks higher up budget on the basis of minimal process and expect full process. If you want 70% unit-test coverage, that is twice the time the code would have taken if you did not write unit test. Add 3 times the time if you want good integration tests to go with it.

Unfortunately, this makes a project costly. The problem occurs if the PM then demands full process when the time is not accordingly budgeted for it.

Release cycles also become long.

Re:Budget

[angel'o'sphere]

If you want 70% unit-test coverage, that is twice the time the code would have taken if you did not write unit test. Add 3 times the time if you want good integration tests to go with it.

Serisouly if that is the case for you you must either be a genius (the unit tests are for nothing and don't ever reveal anything) or you do really a lot of stuff wrong.

Having >80% code coverage ashould cost less than 10% of the developmen time.

And having the unit tests should save over the whole team a lot of time as it basically makes it impossble to check in non compiling code or code that does not pass the tests, saving not YOUR time but saving the time of ALL OF YOUR TEAM MATES.

angel'o'sphere

Re:Budget

[AuMatar]

80% code coverage costing less than 10% development time? Only if you're either ignoring all errors, or have massive try catch blocks that catch everything generically (which means your code coverage numbers are garbage- if you aren't testing each way of generating each type of error, you only have a fraction of that coverage).

I'm not going to deny the usefulness of unit tests, but in my experience writing a good unit test suite takes 50-200% of the time to code the function, depending on the complexity of the code being tested.

Re:Budget

[Fuzzums]

Writing unit tests costs time. It probably makes a difference what kind of application you're writing, but 10% I find, well, "very fast". For the last big application I worked on writing coding v.s. unit tests was about 50-50. You have to think about input conditions, test them, expected outcomes, unexpected outcomes, conditions for exceptions and so on.

There was more than enough complex math involved that without them, development would have been impossible. Regression testing saved our asses in more than one occasion. Having unit tests saved time in those occasions indeed. But not before we write the tests and prepared the datasets that are needed for them.

Together with other self-imposed processes like documenting and reviews, this lead to an application that has a minimum of bugs in it and a very satisfied customer.

Re:Budget

[angel'o'sphere]

When you need so much time for unit tests then your code is badly written.

For clarification, I was not meaning unit tests but tests on usecase/scenario or story level, automated UATs, or how you want to call them. If you can do that you are very fast in reaching 80%-100% coverage.

A "classical unit test" is a test for a compilation unit, in Java that would usually be a single class.
If I can avoid it I don't do "unit tests" at all. In statically typed languages like Java or C++ they are not really necessary/usefull, unless you programm for embedded systems and can not run scenario driven tests.

If you need more time to write the test than the actual code, you should consider TDD. So thinkling about "how to test" improves your way how you write the code. Adding tests to code that "just evolved" is ofc more difficult.
The next thing is: design your code to be easy testable.
E.g. nested structures of loops and conditions should be refactored into their own protected or private methods (best is usually package scope). Then you can write tests for each method, which is easyer than designing test data that leads to the else block in the middle of a loop that is nested in another if.

angel'o'sphere

Re:Budget

[AuMatar]

Actually, the smaller and more modular your code is the more time you need to properly unit test it as a percentage of time. For example, the C code

bool isNumber(char *data, int len)
{
  if(!data)
    return false;
  for(int x=0; x<len; x++)
  {
    if(data[x]<'0' || data[x] >'9')
      return false;
  }
  return true;
}

Took me about 2 minutes to write, more time spent on formatting then I would in a real text editor, due to writing in slashcode. Now to test I need to hook it up to the unit test framework (unknown time depending on framework, lets say it's negligible). I need to write a test for the null case, a test for starting letters, a test for embedded letters, a test for trailing letters. The same for non-letter characters. If I wanted to be truly cautious I'd want to do the same with non-printable characters, in case the idiot who wrote it did something really stupid. I'd want to embed a few nulls (0 values) just to make sure the author didn't assume that they ended the data stream (unless it's supposed to- in which case you should test that it does). I'm assuming ascii here, if that's not the case the above code is wrong, and the test suite should include tests for non ascii utf-8 characters.

I spent more than 2 minutes just thinking up the test cases for a trivial function. Actually writing them all would take me a good 10 minutes with a poor framework, probably 2-5 with a good one (admittedly I'm slower at coding unit tests than writing code, because I find unit tests boring. Consider that a personality flaw). And that's about the smallest number of test cases I'd consider for this trivial function to have remotely good coverage.

Writing unit tests is slow and takes time to do it right. There's payoff in the end, enough to make it worth it. But don't assume it's negligible time, it's really a huge investment.

Re:Budget

[angel'o'sphere]

Well,

you are right on this example.

and you missed obvious (well, or not) cases which emphasize your point.

However I think perhaps the more bare bone your routines are (working on bytes and chars) the more challenging the tests are.

OTOH if you have highly abstracted OO code it might be easier, never thought about that.

So, you have to test for negative len. Also the routine is not "robust" as it has no true relation between data and len. In fact without seeing the requirements it is nor really possible to test the function. (OFC from reading the code the intent is obvios).

Nevertheless it should not take to long to write "normal" test code.

However you are right that for such trivial code the test writing time might be in the 200% range of the original code.

angel'o'sphere

Re:Management should be the ones executin the proc

[Bing Tsher E]

That's fine, but what will you do with all your spare time without an income?

Regarding unit tests...

[jcr]

I know that one particular development group at Apple found considerable gains in productivity from using unit tests consistently, where every time they fixed a bug, they wrote a test to confirm the fix. The upshot was that they spent far less time chasing down regressions than they did before they adopted the policy.

-jcr

Re:Regarding unit tests...

[hibiki_r]

Yes, but one thing is to write a lot of unit tests, especially in areas of major need, and another is to use TDD. TDD treats tests like a specification: You don't write production code until you wrote a test case that fails. Also, you are supposed to make the test work in the simplest way possible, whether you know it'll need refactoring or not.

The problem of course is that just defining and understanding specifications is the hardest part of programming: The fact that you write a spec doesn't mean it's the correct one. And even in the remote case where you know what it should do, that doesn't mean that turning it into a battery of tests will be worth the effort, as many code changes don't come from a program that was written incorrectly at first, but because the requirements have changed. All in all, it can lead to the ossification of interfaces and unwillingness to make changes, as the amount of work required to, say, change the signature of a method in wild use could require tweaks to hundreds of test cases. It also makes changing the code to be easy to unit test so important that testability becomes the one and only test for good design, which isn't quite right.

The value of high code coverages varies greatly with the languages you use. How much does the compiler do for you? How much ceremony is there in your code, compared to important, complex code? Writing an app in, say, groovy, without an extremely large suite of tests is far riskier than doing the same on java, where the compiler stops you from doing many things that would cause runtime errors in a more dynamic language.

Re:Regarding unit tests...

[GreyWolf3000]

Also, a lot of times you don't realize that a specification is completely broken until you're balls deep into implementation.

I personally don't write tests firsts, ever. I tend to work in three steps:

1. Write a proof of concept for the nontrivial parts of the solution.
2. Build tests/specs.
3. Write your production code.

This still isn't perfect, and you're trading a constant decline in productivity for a reduction in the number of times a sprint turns into a disaster because you were solving the problem the wrong way. Also, #1 can often be done in parallel while others solidify the feature requirements more, or supply important assets like graphics comps, branding, copy, etc.

Process is designed to compensate for bad hiring

[cryfreedomlove]

If you hire only Rock Star level developers then you need very little process. Most process is about mediocre managers looking to minimize the impact from mediocre developers.

I've heard this for years

[Virtucon]

Seriously, there's a place for creativity in the development process but there has to be some level of validation. There have been many attempts at coming up with the best process including Agile however they all endeavor to separate the artistic part of coding from the brass tacks of getting the work done in an accountable fashion. More businesses today look at developers as "resources" rather than folks who have to literally create something out of thin air. With "Design Patterns" we had a set of templates now that architects could throw on a UML diagram and behold "that's what we use, don't divert from the path." With TDD we now had "code coverage" so we could think about what the testers should be doing rather than actually making the algorithms and business logic work. I remember, many years ago, sitting in a room with our VP of Software Development. He had just bought a new tool that analyzed code. It wasn't Coverty, it was something else. Anyway, we used to do peer reviews within the Architecture team and we didn't do it on check-ins either, we did it on random sections of code with developers who we thought could use the help. It was meant to be a learning experience, both ways. Well he plops down this tool which naturally looks at all the wrong shit "you only have 2% comment coverage, our standards say a minimum of 25%." To which I promptly retorted "who picked that number? Our standards are that we comment logic that isn't intuitive." To which the meeting then went on to focus why the developer didn't use exception based handling in the code vs. checking return codes after every system call.

That's the problem with a lot of these tools, you have some dipshit in authority setting the standards. Things like burndown rates are there to give the project managers something to do. Now, while I think that this article is a pile of shit, there is some truth to letting developers use their creative skills. Certainly, prototype. Create. Work till midnight on that new bubble sort algorithm and test. Of course you can write a better bubble sort, you know you can, and if you just did it there would be a 50% savings in CPU cycles for this splash screen in the first 1/2 second of elapsed time. Do all of this but also realize that there's deadlines, standards and testing that has to be done. Also realize that you won't be the only person who has to read that code and while you may think that it's perfect, in a year or so you'll think it's ugly because it hasn't been written in the latest language or you just learned that neat new template class mechanism and could eliminate 50% of what you wrote and just use templates! Sometimes you just have to leave it alone and call it done, which is the biggest problem in software development.

Have the bean counters gotten in the way? Yes, but remember they're the ones who have to show management that you're tracking on schedule and delivering something that will work; keeping you employed.

Re:I've heard this for years

[jeffc128ca]

Have the bean counters gotten in the way? Yes, but remember they're the ones who have to show management that you're tracking on schedule and delivering something that will work; keeping you employed.

At what point does it become excessive tracking and managing by the bean counters? I've watched over 15 years a shift from wild west to overly bureaucratic. What if all those bean counters cause quality to go down, not up? Yes we need to avoid the wild west development style to avoid those problems but we seem to be solving that by going in the absolute opposite direction. I've seen bug fixes for simple things take years to work their way through the process. I have witnesses simple side projects with budgets of $50K turn into millions of dollars annually due to process overkill. There has to be an in between where we have some process over site but allow some flexibility for developers to do their jobs. At what point do those bean counter be the reason the company goes under and every one is no longer employed?

Re:I've heard this for years

[Virtucon]

It's all about accountability. Companies will spend $20 in process to track $5 in cash burn. You could blame a lot of things I guess:

1) MBAs like to put everything into little buckets and chart down to the penny where the spend goes. I guess that goes down making sure they give maximum shareholder equity?
2) Project Management is now the new field for people who couldn't make it in Real Estate. Blame PMP certification I guess. If you can manage a construction project, you can manage a large development project. The best project managers in a development project have a background in field.
3) Simple Side projects failing could be from a lot of things but the biggest is usually lack of definition (the Requirements thingy) and scope creep. Understanding what you're trying to solve is the first step.
4) Lack of resources. Even with MBAs, PMPs and all the other horseshit I still see projects put together on a shoestring with resources. Don't go and try and do a $2M project and then try to save $50,000 to avoid that tester. I'm just using that as an example, but I've seen projects fail because some bean counter thought he was doing the right thing for the company.

You're right, there is a balance. That's why I kick Scrum people out the door when they start preaching their mantra. Likewise I won't hire a PM who hasn't had software project experience or been a developer. Sure, I've been involved in failures, who hasn't but the successes have been where we knew what we had to do up front, we had the budget to do it with and we weren't interfered with by the political echo chamber or by the bean counters. Yes, there's always accountability, there always will be but management has to realize that writing software, the actual task of doing it, is creative. It takes structure in math and logic but it takes more creativity than people will give you credit for. When you stifle the creativity you stifle the productivity.

These "code coverage" tool people are selling snake oil. TDD has it's place but like so many things is it something that you can deliver or just have a set of books on your shelf talking about it?

Good Development Process

[Foofoobar]

You can use mediocre programmers if you have (and enforce) good development process and train them in best practices. Being able to build code that is easy for the programmer to read (and others to read), separation of sql/html,css,js/ etc into their respective parts of the MVC/ORM pattern, consistent inline documentation, consistent comments on code checkins, etc., following the same rules for writing code (development docs)

A good development process can alleviate many of the problems with having to do all that testing as you test because you have a bad development process wherein you are unable to easily have the developer look at his comments, his documentation and his clearly written code and go 'oh thats the issue'

Not saying this will ELIMINATE the need for testing but this reduces the need for at least 30-50% of it if everyone is properly trained in a good development process and it is enforced.

Re:Good Development Process

[MobyDisk]

I question this concept. Part of the issue is in how you define a mediocre programmer.

Suppose you train them in all the things you listed, and they grasp them all and can do them, then how is the programmer mediocre? Sounds like a top notch programmer to me! In my experience, the mediocre programmer is the person you train on these things and they just don't get it. Worse yet is that they think that they understand it - but then they go to do it they put things into the wrong layer, or didn't realize they could have used standard pattern X here, or thought they wrote good documentation but it is not helpful when you go back to it. Perhaps they wrote a unit test but it didn't cover the things it should. Or maybe it even reached 100% code coverage but still didn't test all the possible scenarios.

Try to use process to manage *that* person. Even worse, they may be getting top marks in their reviews because they are following all the steps while your most talented programmer who writes bug-free code is being passed over for raises.

Re:Good Development Process

[geekoid]

The need is exactly the same, It reduces the time to test.

It's a really important point that many people don't seem to get.

Re:Good Development Process

[evil_aaronm]

I can almost guarantee you, then, that your product will also be mediocre. Apple - or any other example of "exemplary" design and execution - does not stand out because their developers are interchangeable. You can't possibly get "insanely great" products from mediocre people, or from processes that necessarily drag everyone down to the lowest common denominator.

Re:Good Development Process

[Foofoobar]

Suppose you train them in all the things you listed, and they grasp them all and can do them, then how is the programmer mediocre? Sounds like a top notch programmer to me!

Well thats the point isn't it? A mediocre programmer can become more adequate if they are put within a good environment with good development process and good management to follow through on their responsibilities.

Re:Good Development Process

[MobyDisk]

No, that's missing the point. You have confused mediocrity with inexperience. You constructed an idealized developer who is a blank slate and can be taught anything and can grasp it. So they were never a mediocre developer, they were just a new developer. The mediocre developer cannot be taught those things. Yet managers think that adding process and training alleviates that problem.

Processes don't affect professionals...

[cshamis]

Software professionals aren't deterred by software processes; some may B&M that writing test cases for a UI is a PITA and a waste of time, but a professional software engineer doesn't mind this extra work, because it's just part of the job. Architects, Doctors, and structural engineers have processes that they need to follow, and so should software engineers. Software engineering has only really been coming into its own in the last 30 years, so it's still a young engineering discipline, and the entire field is still struggling to come up with a guide-book. From the stakeholder's perspective: software is crushingly expensive. It takes many man-hours to develop quality software. Project management processes used by "metal-benders" who actually produce a tangible product didn't always map well to software development practices; but 30 years ago that's all anybody had. The last 15 years has seen a huge improvement in software design processes. Much of what was thought to be the last word on the subject (SEI/CMMI, and ISO-9000) turned out to be impractical, inappropriate and/or counterproductive in the real world. Newer mentalities like Agile, and XP are much better suited, but are usually snubbed by middle management tiers because the control gates require qualitative decisions not quantitative. Any idiot can sign off on a status report if X-many bugs per Y-lines of code is met. ---But when decisions need to be made solely on subjective opinion and experience, then suddenly nobody wants to sign off on anything; requests are denied, decisions are postponed, and the project suffers. That's not a problem with processes, that's a problem of ineffective leadership, and it's endemic in the software industry. --But that's the subject of an entire other essay...

Re:Processes don't affect professionals...

[russotto]

Software professionals aren't deterred by software processes;

And no true Scotsman wears BVDs beneath his kilt.

some may B&M that writing test cases for a UI is a PITA and a waste of time, but a professional software engineer doesn't mind this extra work, because it's just part of the job. Architects, Doctors, and structural engineers have processes that they need to follow, and so should software engineers.

Ah, the process is good because it's the process ("part of the job") and some totally inapplicable analogy.

Software engineering has only really been coming into its own in the last 30 years, so it's still a young engineering discipline, and the entire field is still struggling to come up with a guide-book.

So many falsehoods and questionable assumptions in one sentence: That software engineering is only 30 years old, that it actually is an engineering discipline, and that "the entire field" is even trying to come up with a guidebook.

Much of what was thought to be the last word on the subject (SEI/CMMI, and ISO-9000) turned out to be impractical, inappropriate and/or counterproductive in the real world.

A fact which was dead obvious to many, but then we were told that it was "just part of the job" and we were somehow not professionals if we objected to it. In other words, nothing's changed.

Not only killing the software industry...

[geir.isene]

Process is killing much more than the software industry. It is hacking its way through creativity everywhere. And personal initiative. And responsibility. When the process gets the responsibility, then you need corrective processes to cater for process failure. And "dynamic processes" to improve process improvement. Etc.

Although I am involved in several ITIL implementation projects, I have stopped believing in the CSI (Continual Service Improvement) and a few other processes along the way. Instead one should foster a culture that aims to improve.

It is unfortunate that Toyota's success has made processes the Universal Truth in business. It is not. There are a whole list of items much more important than processes in any company, such as

Is the business operating in the right market
Does the company deliver the right products or services?
Do they have the right personnel?'
Does the company have the right management?

I recently came home from Tanzania where I delivered a talk on ITIL to some 60 people that had never seen a PC or a white man before, and interestingly enough; Processes was not among their chief concerns in life. And I betcha that if we tried to enforce processes on that village, their spirit of play would not get nourishment.

Some food for thought:
http://isene.com/processability.pdf
http://isene.com/liquid.pdf

Re:Not only killing the software industry...

[geekoid]

" Processes was not among their chief concerns in life"

yes it is, and I bet they have process buried into everything they do.
All tribes do. It's how they survive.

Process is different from need to need, and it's only one part of how you need to operate.

I will note that Toyota applies all the questions you asked as well as the process.
" It is hacking its way through creativity everywhere"

no, it doesn't. Thats a huge lie. One I am getting sick of. Creativity is pounds out crap code.

Code is NOT ART. never has been, never will be. It's math and engineering. The idea that it's art is what is strangling the industry.

Process should just be...

[mario_grgic]

an agreed upon way to communicate and it should be chosen by the team rather than imposed on them. People who think otherwise, especially people who think they can reduce software development to a process are really claiming that they can implement artificial intelligence using people pushing paper as their computing model. Any computer scientist will laugh at the idea, but it does not stop pure managers trying to do exactly that.

Software Isn't Unique In This....

[tungstencoil]

So, in the article he does suggest some process is probably good. That's good. But then he suggests that "better" coders might need less. In practice, that is probably true, but his implication goes farther than "let's hold the hand of the junior guy a bit more."

So... how to architects, civil engineers, electrical engineers, graphic designers, technical writers, project managers, and, I dunno... just about every white (and blue, for that matter) collar employee stay "passionate" in the face of some level of process and bureaucracy? How would you like to go on a bridge (or a plane, as others have mentioned) built by folks who are so smart they don't need any process to validate their genius?

This sounds more like "I'm so good I don't need to do these dumb things that slow me down" and less "perhaps we should keep the process just at the point it is beneficial".

I just spent two years with a team of 6 refactoring a critical legacy application built and maintained by people who were "smarter" and "better" than code reviews, TDD, planning, documenting, or hell, breaking that 8K header file into header/class and maybe even into a couple of objects. And truth be told: they were ALL really, really smart.... and really, really willing to cut corners just to "get something done", resulting in an unmaintainable mess that cost us somewhere in the neighborhood of 1.8MM to clean up.

To all you guys who are WAAAAAY too smart to have processes: that's awesome. Please just go work somewhere else, preferably someplace I'll never work.

blacksmith vs factory

[speculatrix]

a lot of software is still developed in the same a way a master blacksmith might craft a piece of work

many companies think they can treat software development as a factory mass-producing generic products with ISO9000-alike paperwork

Need an excuse? Follow a process

[hyperenator]

So, your code is a pile of s**t? It doesn't matter, you followed the company approved guidelines, created the necessary documentation and so on... Oh, yes... and you'll fix it in the QA phase ... well, may be.

Re:Need an excuse? Follow a process

[geekoid]

That is why code review is part of any good process. It allows you to mentor.

It's not either/or, BTW.

Re:Need an excuse? Follow a process

[hyperenator]

No offense, but a good mentor does not need a process to play that role.

My software process is...

[pauljlucas]

Programming, motherfucker.

Re:Management should be the ones executin the proc

[osgeek]

Please, come work for me!

I always love it when the developers working in my group tell me to shove my directives up my ass!

Not processes kill.

[Tanuki64]

Bad processes imposed by incompetent non-developers, who do not understand what they are doing and why do.

Process Is Supposed to Facilitate Work

[Greyfox]

Not impede it. From what I've seen, team size seems to be the biggest factor in actually getting something useful done. Having over 10 people on a team really seems to detract from its capability. If you have people on your scrum who don't know or care what another team member is doing, then your team is too large.

The most productive teams I've worked with didn't take the entire process, they took what they needed to stay coordinated and discarded the rest. Actually several teams evolved a process that looked very much like agile long before the whole thing was defined. Short release cycles, short meetings to stay coordinated and small, focused teams. Most teams that adopt any form of testing at any point don't ever stick to it (Even though it'd really help.)

Kinda

[MBGMorden]

I've seen both sides of this coin, and I can sympathize with both viewpoints. The "Agile" nonsense is the absolute most buzzword-tastic pile of crap I've ever had to mess with. It's a bunch of people more focused on playing a word game than actually getting work done.

On the other hand, I was recently given the task of doing some housework on a PHP application that a former colleague of mine had written on his own (it was a system designed to track purchases at local pawn shops for review by law enforcement for stolen merchandise).

Lets just say I wasn't impressed. Single character variable names everywhere. Virtually no comments. Code that should have been put into a function was just cut and pasted each time an operation was done. Tons of stuff (like the actual list of organizations we were receiving data from) was HARD CODED into the code. The passwords for the users were stored in plain text in a database.

What's sad is that I wasn't allowed to actually correct most of this. Management explicitly dictated that I was only to correct a few outstanding issues - NOT really delve into the code. The reasoning was "how much work" that program is because based on the workload of that previous guy (who was maintaining this nearly full-time) put into it. They figured we'd need another full-time developer. In reality the function of the program was relatively simple - it's just that it was so fucking poorly written that any small change took a mountain of work.

Just IMHO, what I think companies need more than feel-good processes is a small set of very thoroughly enforced coding standards. Developers who don't follow them need to seek employment elsewhere. No amount of process is going to make a bad programmer good.

good code quality somehow is opposite of passion a

[devent]

Why is that, that good code quality somehow is opposite of passion and creativity? I agree that pointless process is killing that, but good practice can be good.

I for myself like writing tests, like having easy code that have low complexity. That is why I can be creative in the first place. With my tests in place I can write my code and be sure I won't break anything and with low complexity I always have the overview over my code. Both are making sure that I can be creative.

The real question is why developers are not writing testable and low complexity code in the first place. Why is nobody teaching them how to write good code? Nobody is arguing that in the architecture or machine design, because if you not following good practice there the building will collapse or your machine will blow up.

Please define "passionate programmers" and "great code". If by that you mean people that just code away and leaving behind a mess of a code that only them can understand and maintain, then nobody wants that kind of programmers anyway.

True, there was a lot of stupid things in the past, like the Waterfall Model. But this is why the software industry is still a very young industry. Other industries have over 500 years of experience, like architecture or machine building. But the software industry is only about 50 or 60 years old.

Yes and no.

[DdJ]

If you've got a small team of extremely competent programmers, is excessive process likely to do much more harm than good?

Sure.

If you've got a humungous team of barely-competent cut-and-paste code monkeys that their managers don't know how to evaluate, do you need astonishing amounts of process to get any work out of them?

Yeup.

Can you stop all business activity while you try to change from an organization of the second type to an organization of the first type?

Nope.

Re:Yes and no.

[russotto]

Can you stop all business activity while you try to change from an organization of the second type to an organization of the first type?

Are you likely to hire and retain competent programmers if you
1) have a humungous team of code monkeys
2) have an astonishing amount of process that your new competent programmers will be expected to follow?

Nope. The process just holds you where you are. It prevents success as much as it mitigates failure.

Re:Yes and no.

[DdJ]

Yeup, I agree.

The path through it might be to keep the two groups isolated from each other and not apply processes for the one group to the other group as the change proceeds. Not sure if that'll work, but I figure it may be worth a shot.

"At night, the ice auditors come."

most "creativity" is unnecessary

[h4nk]

As someone with over 15 years of professional soft dev experience, my experiences have proven that unnecessary creativity destroys the reliability of software. Technical inaccuracies, misunderstood and misappropriated patterns and strategies, "tricks" and undisciplined programming are all the best ways to kill a piece of software before it even hits production. Engineering has plenty of room for creativity. Solving complex problems requires such creativity that it often translates to pure talent. But this work is based on a foundation of science and mathematics. Too many times have I seen developers do something tricky solely because they thought it was a nifty technique. These people scare the shit out of me and they permeate the industry. As web development has matured, I have seen the wheat separated from chaff. But for every 2-3 serious sw engineers, there's the 1-2 people that are obtuse, asinine and out of their depth. Most often these people have lingered for some reason or another, the Peter Principle usually. One thing that has proven itself to me: Only the fool suffers a fool. These fools are often the demise of well-intentioned projects. If you want to be "creative" in software development and ignore 40 years of research and science, go to fucking art school and please quit trying to convince the rest of the world to take you seriously.

Dear god yes it is

[jeffc128ca]

I've been programming since the 80's and getting paid to do it since the mid 90's. It was the mid to late 90's when I got introduced to life cycle methodologies and at first I applauded them. I was even selected for a committee to help firm up our "best practices". I thought I would improving things by stopping bad practices by bad programmers. After a while it turned into a nightmare as those bad programmers started using the "best practice" rules as cover for the crap they produced. "But it passed unit testing and UAT, it must be awesome". Our project budgets went up and implementation time increased while our customers were visibly pissed at the final product.

The problem with SDLC process management has been sufficiently skewered by Joel Spolsky in a blog post several years ago. By putting a bureaucratic process in place you replace rock stars with compliance monkeys. Actually delivering a solid product is far less important than meeting checkpoints. And when your a programmer who's dedicated to a polished final product it's like murdering your soul to watch mid level managers (who know nothing of development) work hard at micro-managing you to ensure those check points are met on time. The actual outcome is irrelevant, the surgery was a success even though the patient died.

I have worked in both environments, the loosy goosy no rules and the highly rigid SDLC bureaucratic world. I understand the need for review and verification and the problems of wild west development. But development process methodologies are dangerous enablers of bad middle managers and "consultants" that sell you management processes that don't really help any body.

Re:Dear god yes it is

[geekoid]

Joel has done more harm to the industry then BASIC.

And he's wrong, again.
A) If you have a true Rocks(hint:you don't) then put them in RnD, and have them do proof of concept.
When you realize it's just another guy who happens to have a little charisma, get rid of him.

B) I love the Ad Hom attack against people who want to do there job and follow the process.

C) process and micro management are DIFFERENT issues, don't conflate them.

You problem is bad management, not bad process.
When the process bring to light bad code during code review, they need to be taught why it's bad.
If they can't be taught, you don't want them on the team anyways.

" check points are met on time. "
That statement is almost always a sign of the wrong process. Assuming your goal is good code.

Re:Dear god yes it is

[jeffc128ca]

You problem is bad management, not bad process.

The process is an enabler of bad management. The biggest assumption that all development life cycle methodology is that management is competent and wants the project to succeed. In the real world of cubicle politics this is nuts to assume. The "process" now becomes a weapon in an agenda war between managers and departments. Competent works and managers become targets and victims of the "process".

Bad Developer == Bad Processes

[jason.sweet]

I'm an experienced developer, I don't need any of these new-fangled practices to make my code good.

I'm glad I don't have to maintain this guys crap.

Developers in general won't accept processes that amount to busy work. You can tell when you have this kind of process because your manager is constantly harping on it but eventually gives up. Good developers define the processes.

Re:Bad Developer == Bad Processes

[geekoid]

Good developers define a process that confirms to their bias of what 'good' is.

In my decades in the industry* I have learned developers are like driver. Ask any of them and they will tell you they are better then 90% of the others.

I have heard hundreds of excuse on why code isn't bad, or that sloppy code was all they had time for. It's never really true, it just comes does to lazy developers writing lazy code.

No, the needs define the process. The needs dictate good process or bad process.

Letting just the developer develop the process means it will be inconsistent and most likely useless in the long run.

*not meant as an argument from authority, just giving the reader and idea that I've worked with a lot of developers. It's still anecdotal.

Re:Bad Developer == Bad Processes

[jason.sweet]

Good developers define a process that confirms to their bias of what 'good' is.

True. If the developers are truly good, instead of merely claiming to be good, the outcome will reflect that.

Ask any of them and they will tell you they are better then 90% of the others.

The problem is that most developers are wrong when they say this.

I have heard hundreds of excuse on why code isn't bad, or that sloppy code was all they had time for. It's never really true, it just comes does to lazy developers writing lazy code.

Most developers are lazy. The bad developer cuts corners and generates sloppy code. The good developer automates. The good developer gets help from someone with more experience. The good developer works hard to get things right the first time, because doing otherwise only creates more work.

No, the needs define the process. The needs dictate good process or bad process.

The needs shape the process, but they don't have to shape the implementation of the process.

Letting just the developer develop the process means it will be inconsistent and most likely useless in the long run.

I agree. There are a lot of other players with investments in the process. The developers do not work in a vacuum. However, if "the business" is pushing processes that are useless or disruptive, developers need to treat the processes like they would bad software requirements. Demonstrate the flaws. Offer alternatives. The pride a developer has in his code should extend all the way to the product sitting in front of the customer. This means that the good developer has to take responsibility for more than the code he commits. He has to be invested in the whole software development process. When he does that, he will not blindly accept bad processes.

Yes, but...

[bradley13]

Every passionate developer I have ever know hates "process". At the same time, they all understand that at least some process is necessary. It all depends on two things: what kind of work you are doing, and what kind of management you have.

- What you are doing: (A) If you have a small team working on an internal application, you need a lot less process. (B) If you are a huge team working on safety critical code, you need a lot more process.

- Your management: do you have managers smart enough to realize that A != B, or do you have PHBs who want to fill in boxes on some checklist?

Essentially the "agile" manifesto: as much process as necessary, as little as possible. That said, speaking as a passionate type, I would never want to work on a huge project within a bureaucratic company, precisely because I want to do technical work, not paperwork. I imagine many other technically passionate people feel the same way...

Re:Yes, but...

[geekoid]

A) wrong. the application will become more and more critical. People who did not write it will eventually use it, and need to change it. If it is a slapped together piece of code, they will need to rewrite it. It will cause a lot of money and man power later.

B) You need the SAME process. The process and testing will be faster or slower depending on the code base, but you still need it.

Developers seem to forget that OTHER will use and modify their code

Re:Badges, we don't need no stinking badges...

[countertrolling]

Yep.. excess process is nothing compared to the licensing problem.. copyright and patents will bury us... They already are

History of process and people

[scamper_22]

Let's explain the typical history of process.

A group of relatively skilled people come up with an idea oh how to improve software. Things like unit-testing, agile... They try it out with other skilled people and notice that the process does create an improvement.

This process becomes the latest and greatest craze and everyone in the software industry thinks it is going to result in better software... yet it doesn't.

The reason... you need skilled people to begin with.
I never turn against process. I just firmly think you need good people first before you start on this process binge.

For example, unit-testing is a great idea. I use it all the time. Yet, when half the people don't know what a 'unit' is, unit testing just flops. Unit-testing works great when you have your code well abstracted in clean units.

So for unit-testing to work... you need, abstracted, clean code. How many 'bad' developers write abstracted and clean code? You also need developers who can write proper tests and think of the cases to test for unit testing. Again, if you have a 'bad' developer who isn't checking for errors in actual code... why on earth do you think they're going to write useful unit-tests?

The same is true for every other process. Agile needs good managers, product people, developers, testers...

It's not even about time. Believe me, if all the code I worked on was clean, abstracted... written by good people... I'd have no problem writing unit tests for everything possible and following process 100%.

But typically, you have poorly written code with many mediocre/understaffed people and then some mandate to implement process which doesn't work well given the poor code and people... and just eats up time... and you get frustrated at the process.

In reality... the process isn't the problem perse. It is just exposing the poor code and mediocre/understaffed people.

Re:History of process and people

[geekoid]

Your issue is with training, not the process.

Every need to put their ego back into their pants, and learn to do good code review to bring up the inexperience developers, and get rid of bad developers.

I know I beat this dead horse to a pulp, but software developer need to learn how to engineer code, and should got through the same process as engineers. Including needs a software equivalent of a Professional Engineers license.

The fact that ;people are basically writing the same thing over and over again throughout the industry is a failure on the industry.

Re:History of process and people

[scamper_22]

"In reality... the process isn't the problem perse. It is just exposing the poor code and mediocre/understaffed people."

Didn't I just say that.

All depends on the team and the project

[degeneratemonkey]

N/T

Creative!=Good

[mrbored]

Software Programming should be handled like all engineering disciplines. Imagine if all engineering firms started building bridges, apartment blocks or cars the way software developers write code. To put it simply, we'd all be dead. The next time you're restarting your computer, upgrading your drivers, installing a new firmware, just think to yourself, when was the last time something purely mechanical stopped working for no apparent reason.

I wish all software was written with the same engineering philosophy which NASA employs. They Write the Right Stuff. Sure we wouldn't have as much software as we do today and it wouldn't be able to do as much, but what it did do would work flawlessly and I wouldn't feel like throwing my keyboard through my monitor at least once a day while at work, only to arrive home to have to restart my Sky HD box because it's crashed or has incredible menu lag.

Bad management

[degeneratemonkey]

is not knowing when to use process and when to throw it away.

Re:Bad management

[geekoid]

If you pick and choose when to utilize the process, the it has failed. Both management and developer.

Learn to adjust and modify to meat your goal for having the process while still getting code out.

James Turner has not much clue ...

[angel'o'sphere]

... if you read his article (the one linked in the story) you see he does not know anything about "agil processes" or Scrum (e.g. tracking hours for burn down charts, lol)

Read the comments on his web site to his "article" ... most are "insightfull".

angel'o'sphere

Two words...

[dacro]

http://programming-motherfucker.com/

false rpemise

[geekoid]

I have seen many passionate developers right crappy code. Code that cost money at the end of the day.

Code is engineering, use engineer practices.

Short answer:No.. longer answer: Yes

[gosand]

I've been managing a testing team for the last couple of years and we've been doing Agile. We had a core team of people attend an Agile Boot Camp for a week, and it was one of the best things I've done in the last 18 years in the industry. For the first year Agiles was great, then the "real world" of doing software in a very large company started to creep in. It is still functioning much better than Waterfall, but here are my thoughts:
      1. The first rule of process improvement is that in order for a process to work, it has to be useful. Because we were trained on Agile, we made it useful. The trainers were able to quickly shut up those few project managers who thought they were already experts in Agile and wanted to "improve" it.
      2. As a career tester, it was nice to hear testing portrayed as a positive thing instead of what always happens - people blame the existence of bugs on the testing team. It was really great to see developers, project managers, and product development people to think like testers. It was also great for me and my team to see into those worlds as well.
      3. Communication! It was really great for the first year. We really collaborated with Product Development, Testing, and Development teams. We worked really well together. Everyone was focused on this project. Then people's attention started to get diverted and shared between the many things that come up outside our project. The general quality of things deteriorated.
      4. We had to tweak things. Simply because of the environment we are in, and the project particulars where we aren't a stand-alone application and had to interface with other apps/groups who follow Waterfall, we had to modify how we do things. It hurt us, but was unavoidable. I think Agile would be awesome if you could do it in a bubble.
      5. Transparency. This goes with communication, but initially everyone was open. If something didn't get done, people would just say that. There was no blame, let's just address why it didn't get done and move on. But I honestly think that developers have that "god complex" where at the same time they want to be told what is wanted (removing any creative control) and not letting anyone else know what is really going on (controlling the reins of things). I really liked not having to play any politics or that other BS that always happens - finger pointing. We're still doing OK, but it's crept in from time to time.

Agile as a process works - provided that it's applied to projects that can effectively use it. Agile isn't the right process for every project. And I think that is ultimately what "process" needs to be seen as, an effective means to an end. The process is just there in order to get things done, if it doesn't work then it needs to change. Too often, people don't want that (or they want it too much). People like to blame the process - and rightly so. For a process to work, it needs to be usable, and there is no panacea when it comes to process. You just have to find the right one for what you're trying to do.

It is not process it is...

[cfulton]

Software development is both an art and science. It is creative but exacting. The processes that are put in place around development are there to make sure the developers generate a bug free artifact that delivers on the requirements.

It is not process that removes creativity or productivity from developers. Metrics and processes can be used to level up developers from mediocre to good to great. Developers need to be able to see in the metrics and review what is “wrong” with their code and what they can do to improve it. This post is really about Management and Leadership not software development processes. Any process from just go do it to Agile Extreme Programming can be used to generate good code or bad.

The real culprit is a lack of understanding on both sides of the aisle. A certain type of developer think they are above the process. Their “creativity” is so important that they lose sight of the fact that they are being paid to deliver a product that meets a set of requirements and quality specs. On the other side of the coin are project managers, business analysts and corporate managers that don’t understand how software development works or even what a good set of requirements are. Their ignorance leads them to over process the development cycle. The development becomes about process instead of the process being about development.

When diva developers and poor management come together you can bet that the project will fail.

Article Title Is Worst Title Ever Written

[The Living Fractal]

Notice how I used hyperbole there, in my subject line? That's because the title of this article is ridiculous. Nothing is going to kill the software industry. Why play Fox News here? The real issue is efficiency, not existence.

the end result isn't the "process"

[murdocj]

Process is fine, as long as it's always aimed at producing better software. When the goal becomes the process itself, that's when you have a problem.

If you are having lots of trouble with changes introducing bugs, it's time to extend test coverage, because that will help the goal of producing reliable software on time. If the goal is just "90% test coverage" without any relationship to how that improves the software product, you've got a process problem.

I've been in situations where everything is going fine, we are producing good code sprint to sprint, but the scrum master gets focused on getting a smooth burn down chart. The burn down chart is NOT the product. If the product is being produced successfully & on time, focusing on the burn down chart is a distraction that is hurting, not helping.

Re:Better question

[cfulton]

Here here. Spot on.

all about talent level

[Pro923]

Do you think Mark Russinovich used development processes when he created the SysInternal suite of tools for Windows (perhaps the best set of developer tools ever made)? For great developers, process just gets in the way. For the other 95%, process is necessary.

Re:all about talent level

[Pro923]

I would argue that a single talented software coder could produce better software than the entire Boeing team and all their process could in about the same amount of time. Process is a way to take talentless people and still bang out a product, pretty much by brute force.

If you don't get it you shouldn't be a coder

[Viol8]

"What creativity is needed to code a crystal clear requirement or specification? Sorry I don't get it."

I sure as hell wouldn't hire you then - you sound like a standard issue glue coder doing lego brick style programming. Algorithms and solutions to tricky problems don't just invent themselves, someone with creative flair and intelligence needs to dream them up. Thats obviously not you.

Re:If you don't get it you shouldn't be a coder

[angel'o'sphere]

Algorithms and solutions to tricky problems don't just invent themselves, someone with creative flair and intelligence needs to dream them up. Thats obviously not you.

As I said before, in the real world such situations only exist in very very rare cases.
I did not see a programmin challange the last 25 years, I did not need to "invent" an algorithm etc etc.
And I really doubt: you didd. Otherwise you had understood my previous posting and did not start bashing me.
The only industries where programming requires creativity are: gaming, astronautic instruments, complex control for robotics or automated systems etc. perhaps modern compiler design, e.g. for groovy.

Let me bash you a bit: if you need creativity to build a typical business application then you can not programm, and thus I would not hire you ;D

angel'o'sphere

Re:If you don't get it you shouldn't be a coder

[rjstanford]

"What creativity is needed to code a crystal clear requirement or specification? Sorry I don't get it."

I sure as hell wouldn't hire you then - you sound like a standard issue glue coder doing lego brick style programming. Algorithms and solutions to tricky problems don't just invent themselves, someone with creative flair and intelligence needs to dream them up. Thats obviously not you.

What you're missing is that the vast majority of all problems arenot tricky. At least, not unless someone decides to make them so to satisfy their creative urges. Most companies are, indeed, solving very much the same problems as hundreds or thousands of other companies. The twists in business logic that (may) make them unique are often not even visible at low levels of software development.

This is the same urge that caused someone at a company I worked at (for a whole 6 weeks) to spend several months re-designing the way that a Java-based web application handled translation. Creative? Sure. Solved the company's technical problem slightly better than a "stock" solution would have? Yep. Worthwhile? Absolutely not. Nobody else knew how to deal with it, no other libraries would work with it, and it probably ended up costing the company around $100,000 in the end in lost time and productivity - while saving them a few nickels a day in unneeded efficiency.

Most problems are not unique. Learning to recognize patterns and apply conventional solutions to most things is a very valuable skill. Knowing where to apply conventional patterns to unconventional problems is even more helpful. Correctly identifying the 2-3 times a year you actually need a solid, creative solution to a really unique problem is priceless - and very hard to do if you haven't made them stand out from the sea of normal boring problems by glorifying those into things they didn't need to be.

in balance processes can save lots of time / bugs

[Dan667]

sure you can over do lot of these practices, but I find that I spend less time fixing bugs when I do them. There is certainly a balance and diminishing returns though.

On the flip side...

[SCHecklerX]

Too much software is written without any thought as to how it fits within a process. Management tends to just throw technology at a problem, vs. first analyzing what they are trying to achieve and molding the software to that process to increase efficiency. Too often, software is chosen, and the business then molds their process around it, making things a pain in the ass for everybody except your "passionate" developers, who likely don't know a whole lot about the existing (or lack thereof) process. Retarded.

Stifle creativity?? Give me a break

[SpikeTheRight]

Stifle creativity?? Give me a break. Yes thats right, programmers creativity should be stifled. What programmers don't understand is that no one is interested in their dumb ideas. Don't be creative, code jockeys, just do what you're told. But of course thats not enough for the massive egos of most programmers. They need to have something more important to do than just writing code they've been handsomely paid for. The fact that you think programmers need to be "creative" says a lot about your misunderstanding of software development. I would like to spend more time on this but I actually have real work to do, developing software like a professional, following a process. Grow up.

Re:Stifle creativity?? Give me a break

[Anonymous Coward]

Thank gawd for the hackers or we'd only have boring, uncreative professional types and small advances in software. Yes, if you're doing the 1% mission critical stuff, you need to be very strict and creativity is only necessary for solving problems. But for the rest of software, creativity is an important element in rethinking what computers can be used for.

Would we have Linux without creativity? Would we still be using Fortran and Cobol (LISP being too much in the hacker, creative camp) because nobody was allowed to think of any other way of designing a programming language? Maybe we'd be stuck with assembly or even machine code if people couldn't think creatively about how better to program the machine.

Would Alan Kay and company at Xerox Parc have invented SmallTalk and pioneered the windowing environment without creativity? How about Douglas Engelbart's Mother of all Demonstrations and seeding the ideas that would turn into being the personal computing revolution in the 80s and 90s. Would we even had an internet if someone hadn't thought creatively and wondered if you could connect two machines and have them communicate with one another?

I don't understand this mentality that programming should be done without creativity. Why would anyone of you be in the profession unless it was for a paycheck?

Re:Process is designed to compensate for bad hirin

[geekoid]

Rock Star programmers are a myth.

Re:Need moar people

[geekoid]

No it's not. Have a series of test you automatically run before check in is a good way to catch the more gross bugs.

When you are in an environment where the code is too big to be in your head, then a test can catch bugs you may have created due to other changes in the code.

But yeah, the developer should not write tests.

Management doesn't want creativity or passion

[n7ytd]

Management wants shippable code, predictable results, and the ability to swap developers in and out of programming roles as needed. As the life force gets drained from drone A, they want to be able to replace it with drone B without downtime or a learning curve.

A strict process tends to help this.

Um no, sorry.

[Viol8]

"But don't expect to write code that keeps a 777 safely in the air."

I'm sorry, but thinking up code that keeps a 777 flying safely requires someone with creative thinking as well as good process. You won't always find the answers in he Dummies Guide to Writing Avionics Software.

Re:Um no, sorry.

[caywen]

I use lib777, so my code looks like:

while(!wantToLand)
      my777.StayUp();

my777.Land();

There is no crash in the code.

the industrial revolution marches on

[PJ6]

A lot of so-called standard software development processes treat its workers like the machines they use. This sentiment has permeated not just programming, but all industry for the last two hundred years.

Yet attempting to automate programming misses the whole point of what it is; it's akin to trying to reduce second-order logic to first-order logic.

All these process-related failures we see in software development are misguided attempts to recreate a traditional computer with people.

Agile for the win!

[CosaNostra Pizza Inc]

Agile for the win!

sw dev != sw engr

[Anonymous Coward]

Not to throw stones, but I think a major point that is missing here is that software developers are not the same as software engineers. Software developers do not need extensive process and procedures to function, software engineers do. Someone spends 80% of their day writing code is a software developer, while someone who spends 80% of their day writing requirements, verifying against requirements, capturing design, etc is a software engineer.

I'm neither, but have been labeled as both.

Killing the software industry?

[caywen]

To the question about killing the software industry: No way. Seriously, do you see software companies becoming any less prolific today than ever? I don't. It's incredible to me how much (quality) software is churned out on a daily basis, processes in place.

That said, the question about whether heavy process can kill ones creativity - yeah, absolutely. Some programmers become frustrated more quickly when there are barriers between their ideas and their code hitting the download sites. That's normal, and there isn't anything wrong with that. I'm one of those programmers, and that's why I always pursue my own side projects. That's our outlet, and once in a blue moon one of us makes something phenomenal, without the processes. Once in a while.

But to imply that such programmers are so great in number that it would kill the software industry is just not realistic. There are simply a huge number of creative developers who can work within the constraints given to them.

Re:Developer quality

[GargamelSpaceman]

You are wrong. H1Bs are not low quality developers. However the ability of management to obtain software development at cheaper rates either because of the H1B system enlarging the market of domestic software developers, or throught outright outsourcing, has enabled the higher ups to get a short term boost in 'productivity' at the expense of good software development practice.

Being expensive gives software developers SOME small bit of ability to say no, I don't want to do it this way just because it's quicker now, because it will take me hours and hours and hours in the future to maintain it, fix it, and eventually do it over later, and make future development more time intensive.

Being able to get hours of software developer's time for cheap, means working smart is less of a priority for the higher ups. ( Note this is not the foreign software developers who don't value working smart, it's the higher ups ).

Of course, even with cheap software development available, not working smart will soon lead to software development costs equalling and yes even exceeding previous levels, in much the same way that moore's law has failed to keep computers from being resource strapped. Cheaper resources means they will be used with less regard - see http://en.wikipedia.org/wiki/Jevons_paradox

The computer, instead of being a tool that allows few people to perform tasks that would have required millions, each software developer will become able to maintain a smaller and smaller bit of code because the codebase a developer can comprehend is proportional to it's quality until low paid software developers are developing code that does less than ever before!

We'll have ( offshored or whatever ) coders working for less than 'minimum wage' performing tasks using coding and computers that could have been done by hand in the 1940s for less!

Ok, maybe a bit of an exageration but you get the idea.

Cowboys Wrangle. Butchers Process.

[istartedi]

I don't mean that as an insult to Cowboys or Butchers. I'm a Cowboy and I'm not ashamed of it. I can bang stuff out. I can get things done. I welcome and recognize the other guys--the Butchers. They're the maintenance coders. They're the guys who don't mind writing design documents post-facto, knowing that nobody will read them. They don't mind meating with managers. Of course, if you want to disparage the Butchers you could say that Butchers, well... butcher. You didn't hear that from me though. In order to put meat on the table, you need Cowboys and you need Butchers.

Re:Cowboys Wrangle. Butchers Process.

[PPH]

They don't mind meating with managers.

Freudian slip: I think you are jonesing for a nice juicy burger, dude.

TDD

[Mongoose Disciple]

We all know by now that Test Driven Development is a best practice.

I certainly don't know that, although I do know that some people think it is.

It's kind of the stereotypical mistake of process: a technique that makes a lot of sense in some cases and produces great results in those cases is generalized to a lot of situations in which it's either useless or counterproductive.

Is your team really agile?

[farker haiku]

My team is agile. We have a burn down chart. We have a weekly iteration planning session. We pair program, we test first, we write our cucumber tests, javascript tests, junit tests, and we code 35 or more hours a week. We're also in one of the strictest change management environments I've ever worked in, and we're still churning out high quality, highly tested code in high volume. In our stand up every morning we talk about the stories that were completed the day before and we mark them on our burn down chart at that time. Our Kanban board helps drive us through the process and it keeps everything visible to all the team members. Documentation for our semi-monthly releases is a chore - it takes us about 8 man hours a month, but were working to automate a lot of it.

Agile means adapting, and it sounds like you're not.

Re:Badges, we don't need no stinking badges...

[PPH]

Been there, done that.

The reason you can do what you do is that when the project approaches impending doom, management is willing to toss out all the crap and let the workers get on with the job.

The problem is that, they only pull the bean counters and PHBs off of you until things are up and running. Instead of taking them out behind the office and shooting them in the head.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Design != build

[dazedNconfuzed]

You are suffering the same confusion I'm trying to point out.

Software engineers (and their management) confuse the blueprint with the finished product. Your source code is not the final product, it is the complete detailed blueprint of how to build the final product - from which your compiler chain then builds the final executable (in a manner nigh unto instant and free).

This confusion leads to misapplication of processes, which in turn lead to silly assertions such as TFA. As build time approaches zero, so does the imperative to do correct design.

It's an art, not a process

[paulxnuke]

I'm an old school developer (1975, anyone?) who still views code as an art rather than a bean-countable process and has no interest whatever in management (job security through non-productivity.) The endless processes I have to deal with nowadays do suck most of the joy out, and despite all the typos I've seen caught by code reviews, I'm not convinced they make better software. What they do is:

- Make managers, who have never known what we do anyway, happy by giving them numbers to look at. Their ultimate (and hopefully futile) goal is to systematize development to the point that monkeys can do it. Then programmers will be screwed just like teachers and firefighters are now.
- Make old programmers who saw the way things were going and (unlike me) converted into process inventors while their colleagues were being outsourced, secure and/or tenured.

I'm behind the power curve (none of my startups made me rich) but I still enjoy writing code; my hope is that, by the time I get to stop pumping out crap for corporations (e.g., my dogwalking business takes off), that I still like writing code.

The developers? lol

[rgviza]

It's been my experience, at least in a bank, when you shove excessive process down everyone's throats, the developers are the least of your worries. We'll happily spend half the day doing the necessary paperwork to update an address on the web site, chasing down people for approvals, etc etc.

The real issue is once it takes 2 weeks to change a phone number on a fucking web site, because of excessive process, change control meetings, qa sign off etc etc etc. your business customers (the departments you write your software for) fire your IT organization out of frustration and go outside the company to a contractor that's not hampered by such processes, procedures, and bullshit. When you ask them to change a phone number, it gets done before the phone is hung up, just like we used to do it.

Then your entire IT development unit gets fired, largely because of some busybody manager that had some great ideas about shoving their stupidity up everyone's ass.

True story...
When I worked at this bank, there was one change control meeting a week. You had to first get the idea approved at a CC meeting. Then you had to go code it, get it signed off by QA, who had to schedule you at least a week out, because they were busy too, and present the qa results at a second meeting. So it quite literally took a minimum of 2 full weeks to change a fucking phone number on a web site. They tried to move me to a different state. I found another job and quit because I didn't want to move. 6 months later the entire team got laid off because the business units went to an outside contractor that could get stuff done in minutes instead of weeks.

Process is GREAT!

The four easy steps...

[frank_adrian314159]

... of succeeding as a QA functionary (With bonus metrics for more quality!):

1. Come up with a metric which can be tenuously tied to quality. Ten bonus points for metric being ill-defined.
2. Make developers (and your QA team) jump through hoops to measure the metric and make it go up/down (as appropriate). One bonus point for each percent in increased time used to calculate/track the metric.
3. Point to increase/decrease (as appropriate) in metric to show your "effectiveness". Five bonus points for the delta being statistically insignificant.
3.5 (optional) If metric does not change, speak out forcefully about how developers are "resistant to change" or "don't care about quality". Twenty-five bonus points for each employee sacked during this time.
4. Never measure customer satisfaction. Minus 200 bonus points for doing this.

Repeat as needed and celebrate your success in improving quality!

"Look at how much I'm improving things"(LAHMIIT) metrics are the bane of my existence.

Processes make sense ... or don't they?

[garry_g]

The problem IMHO is when processes get out of hand - exceeding the complexity of the original project. True, there are projects that require near-perfect code as e.g. people's lives might depend on the quality. But just as programs suffer from excessive feature-growth (of stuff hardly needed, see M$ Office products e.g.), the same way process definitions tend to grow over time. Plus the compexity of a program if also often not easy to judge. For experienced (or "genius") programmers, some problem might be trivial, and quickly coded, while management (not understanding the problem) might see it as a difficult problem, endangering the company, which then require complex planning and QC. All the while the programmer could have finished the project less than the time required to even get the plans in order.

This reminds me of a Dilbert comic, where the PHB would come up and require some plan on how to impllement something. Dilbert would reply, he'd already solved the problem. To which the PHB replied, he'd still have to make the plan ...

Personally, having programmed for something like the last 30 years (starting with ZX Spectrum and C64, via Amiga through Unix/Linux), I can pretty much code pretty decent code up to a certain complexity without doing much of planning ahead of time. Anyway, I do "fall back" to doing some more or less intricate planning at times, when I "feel" like the problem might have some tough points that could mess the whole thing up. Do date, I barely ever had major problems with this way of working. But then, I'm also just about the only person in the company doing any serious programming in that area ... so no team to keep in line ...

Grammar nazi alert

[Intron]

"fleshed out", please.

flushed out is what happens to the project just before it is completed.

not necessarily

[Chirs]

It's impossible to cover every possible scenario in unit tests. Also, unit tests don't cover the system as a whole.

Data collection and improvement

[urusan]

I have a mostly academic perspective of this debate.

Basically, the goal of process is to achieve high quality code at a low cost, or to put it another way the goal is to make development efficient. How this objective is achieved isn't really that important. Anyway, due to the massive complexity of a development team (member personality and interpersonal chemistry are complex affairs) and the plethora of different software objectives/metrics (nuclear plant safety software is measured mostly by lack of failures while social networking software is mostly measured in adoption rates), there's not really one right process.

However, just leaving it at that isn't very informative. It turns out there is one key element to process that everything good about process flows from...which is collecting data on what the developers are doing and taking the time to analyze that data to find areas where the process can be improved, leading to constant improvement in the efficiency of the process. Collect (Quantitative) Data -> Analyze Data -> Adjust Process -> Goto Collect Data.

Keep in mind that the data collection and improvement analysis are also part of the process. Collecting too much data (or the wrong kind of data) can be very detrimental as it takes time away from the developers doing real work to handle paperwork and constant navel gazing sucks up otherwise useful time. On the flip side of the coin, too little data collection will not allow you to construct an accurate picture of what's going on and good analysis of that data requires real time and effort...but this effort will pay off in the long run by making the real work more efficient for many years.

This process also takes some experimentation. "Let's try adding/dropping/modifying this practice and see what happens." "Oh, the data looks good after we implemented this, so let's continue to do it." "Hmm, this caused a slight decrease in performance due to time usage, I don't think it's worth it so lets drop it." It also takes some deeper thought. "Well, there was a slight negative impact in the short run, but this should have a positive long term impact, so lets hold out on judging it just yet." "This software is needed for a one-shot event, so we could drop all our long-term processes this one time to save effort." etc.

Existing processes and best practices are great places to look for ways to improve your process. "Code reviews have well known positive benefits across a wide range of situations, so we should incorporate that into our process." This really cuts down on the amount of experimentation needed to find a good process. However, following these suggestions blindly will lead to all sorts of trouble because as several other posters have mentioned it's a form of "cargo-cult engineering". One might adopt a practice that's good for high-quality high-cost systems when your goal is to make an acceptable-quality low-cost system quickly. One might adopt a practice that has a subtle negative effect, like driving away a good programmer or wasting small amounts of developer time for no benefit. If the boss is blind enough, they might even adopt a practice that completely destroys the team and project.

Constant analysis and review also helps reduce problems like valuable people leaving due to process. If they know the new practice they hate is only experimental and they'll be able to make a case against it at review time it makes it easier for them to tough it out for a while.

Once the process of continual improvement gets started, the team will constantly get better and better. Realistically there will be ups and downs, but the team will recover from the downs (such as a valuable person leaving), putting them at a nearly constant state of high efficiency...and the extra work put into maintaining and improving the process will be worth it.

Now of course there's more to good process than just striving to be in a state of constant improvement, but that would take far too long to talk about in depth. It includes things lik

Whatever you do, do test.

[k2r]

From my experience the optimum process just depends from company culture and team culture and how clear the requirements are.

The _only_ constant observation I made is that every project that did not test on every level, passionately, very early and continuously was doomed. If you don't define and prove quality you will not deliver quality. If you test you will define what you build. The more complete your tests are, and the higher levels they reach, the more you and your customer know where you are heading to.

"Just build your idea" may feel good at first, but your team will suffer and burn at the most inconvenient point in time.

DO TEST. A LOT. AND THEN MORE. AUTOMATE.
EMBRACE YOUR TESTERS. OR ELSE.

It's way easier to be creative if when can be sure that your project will not die a horrible death.

Re:(Design != build) but (Code == Build)

[Platinumrat]

Code is not a blueprint. I can't point a customer at code and say this is what you are getting. I can, however, point at requirements documents, functional specifications, user manuals, acceptance test specifications and architectual drawings. A savy customer will be able to make sense of those.

The code is really the nuts and bolts, concrete and other raw material that the people use to assemble the product; just like in a bridge building project. If I have already poured half the concrete and the piers are already standing, it's a little difficult to turn what was a suspension bridge into a truss style or move one of the ends

It doesn't end there. Where I see a lot of confusion, is assuming that an executable is the end product. It's an intermediate component in a much larger system. Example: Why am I building the bridge? To join two roads togother? To join two cities? To link two countries? It's always as part of something else. It all depends on scope, but it's never because I saw a river and said..."Hey I think I'll build a bridge". The same with most complex systems. The 777 code modules fit into some part of the final system. You can point at the avionics software in isolation and then plug it into the entertainment system.

Process is the problem

[funky_vibes]

Developers can create solutions without process.

Process only creates problems, like someone auditing your code when not needed. Or noone auditing your code when you do need it. Same goes for testing, unimportant (old) parts get rigorously tested, but the latest development may be ignored entirely.

Process is something you need in a kindergarden, not when dealing with grown ups who are supposed to be trustworthy.

Differences between developers

[npsimons]

How can you tell the difference between a good, experienced developer and a mediocre, inexperienced developer? The good developer already has "processes" (habits and self-discipline) that she has learned and honed through hard experience, knows they work, and resents when someone tries to force her to change for the sake of change or waste time on things that she has already tried and found wanting. Processes will only hamper good developers. The bad developer also resents "processes", but that's because they don't have any self-discipline. Processes probably won't help bad developers either. The trick is to hire good developers that get along; the only way to tell a good developer from a bad is to look at their code, which also requires . . . good developers. A good place to start, though, is to make sure they can actually program. Another good sign, though, is a good developer will occasionally approach you and suggest possibly trying a new "process" to see if it will make your shop more productive or reduce bugs.

Selling consulting

[jawahar]

Selling software != Selling consulting
IT Consulting companies business is to sell process to their clients.

Management would like it known...

[bregmata]

... that the floggings will continue until morale improves.

Creating Quality Requirements

[Hyperhaplo]

Ok, there's now a few hundred comments here and no one has linked to http://www.processimpact.com/articles/qualreqs.html or similar. Try the PDF version - http://www.processimpact.com/articles/qualreqs.pdf

Have a google around. There's a full document of which is the summary.

Read the Book of 5 Rings.

Read the Art of War (alternatively, listen to it..) ...

and start applying this to your job.

Requirements management is something everyone tries to avoid because it is 'too hard'.

---

On the darker side, I work with people to whom writing a short (2 to 3 pages) 'technical specification' is a 'waste of time' as the 'code documents itself' and 'it is obvious what the code does from reading it'.

Funny part about this comes when 'simple utilities' need to be updated.. and only the original coder knows how they are meant to function.
Yes, it reads from that file.
Yes, it writes to that file..
Yes, it does that processing..
But, what is the actually business requirement?
No idea, because that coder is gone. The system has changed.. and now we don't know if it should be doing x, y or z. What do we do now? No technical documentation, no business spec, and over 400 people use this (currently broken) utility on a daily basis.

Wait! I know! Let's do some requirements work and determine what it *should* be doing now.
No, we can't do that.
Too much work.
Let's just 'fix' the code.

Hang on. You have just made a code change.. slapped the code back into Prod... how do you know that your change 1) is what users need and 2) actually works?

So, they brought the code down again, and made another change. This went on for several weeks, annoyed users to no end to the point of end users not using the utility... and our reputation goes further down the drain.

All because no one wants to do requirements analysis, change impact analysis, business specifications, technical specifications or actually properly document their code.

That rage you feel when someone says "the code documents itself!". It means it is time to fix the processes, or leave.

Meanwhile, yes, we have a review process. Test plans are required. Testing is 'required'. Why don't these processes work? Well, it's called "rubber stamping".

If you can't beat them, join them or leave.

Apprentice, journeyman, master?

[ResidentSourcerer]

Guru as team lead probably is NOT a good idea. Again, team lead is at least partially a social skill. I've known several guru class coders, and I wouldn't like to have any of them as bosses.

I can see merit in having a master/apprentice system too.

An experienced coder has a young apprentice. The apprentice does the scut work: Making the code pretty to whatever the flavour of the week is, reading the inline documentation, and adding to it as needed. (Or writing it from scratch...) Writing test suites. Assisting in the debug process.

Apprentices should be moved from journeyman to journeyman, say every 6 months. In this way they become familiar with various parts of the code base. They see a bunch of different work styles. They get practice at coming up to speed with other people's code. In addition they act as pollenators. "Mike showed me a neat trick with...."

After several years of this, they become journeymen. But now journeymen are paired to journeymen. The PAIR has to sign off on a milestone. For each segment, one codes, one documents/tests, both debug.

After a few years of these, if there is merit, they become masters, and are paired with an apprentice.

Some journeymen may not be master quality. At this point they are diverted into sales, and HR, and support....

Alienation of labour

[silasdb]

Ha, and you thought this was new: Theory of alienation

Commitment.

[mevets]

I found it very difficult, as a permanent team member, to not throw my whole shoulder at whatever we were doing. I resented the people that got in the way of our success; I wanted to do nothing more than make the best fucking in the world.

When I watched the 2000-2001 tech meltdown, and saw lots of colleagues who were equally passionate and dedicated ejected like horses in a still sea, I got pretty jaded about it all. These people were wildly committed to their teams and projects, and the company they worked for (at the time three letters starts with S) had zero reciprocal commitment. The horses were drowning at sea; and the survivors were ordering $300 bottles of port at pointless team dinners.

The best lesson from that company was that reciprocal commitment is fair game - I have something a company wants, and they have something I want ($$$). The relationship ends there; they get what they pay for.

No lessor quality - often in fact better, because repeat business is sweet - but complete emotional detachment. That is the way it should be; its not a marriage, it is a hook-up. If businesses moved back to giving a rats ass about the people that make them a success; well that would be different. Try and find two.

Process vs People

[greywire]

The problem is not Process (either the lack of or too much of), the problem is People.

No Process + Bad People = bad results (see GIGO)
No Process + Good People = good results
Processes + Bad People = bad results
Processes + Good People = good results

See the pattern? Now, in theory, good Process + Good People should > No Process + Good People

I've tried implementing various processes and the problem is always the people. Some people get it, some don't. Some refuse to use it, some dont understand it. Some people's internal workings just arent compatible with the process. Experience is important.. more experience usually means more acceptance of process.

You have to match the process to the people or vice versa.