Software Accountability Made Real?

An Anonymous Reader writes "In a recent presentation and post, Kent Beck (eXtreme Programming, Embrace Change) highlights Open Quality Dashboards as a means to make software development accountable. Many different approaches attempt to reduce the number of issues creeping in all along the development process. Whether a shop abides by the rules of up-front UML design or test-driven development, or a methodology somewhere in between, the ongoing burst of popularity for tools enabling continuous integration and frequent releases shows the need for unit testing to appear earlier in the development process. In this context, quality dashboards could well establish a credible benchmark for software accountability."

Comments

We do this internally already.

[Dr. Bent]

At my company, most of our products are built daily (at a minimum) and the metrics are published to an internal website. Things like ugly code, unit test failures, bad JavaDoc, poor test coverage, and findbugs problems are visible to everyone in the company.

This makes it a lot easier for developers to do the right thing (and fix these problems). Nothing like a big red bar to motivate you!

Re:We do this internally already.

[Anonymous Coward]

At my company we just whip the developers when they make mistakes. It works wonderfully!

Wha?

[superpulpsicle]

Software development accountable what?! How about making Management accountable. The developers are not the ones screaming for new features every release with these ridiculous time lines. The decision makers are accountable. Aka executives.

Re:Wha?

[neiras]

That's EXACTLY it. At our shop, we got sick of being blamed for "taking too long on projects" - so we got together, got up to speed on Personal Software Process and Team Software Process, and started a development lifecycle and process improvement team.

There are a number of interesting benefits to this. The best one so far is that we maintain a 'responsibility trace' right from individual stakeholders in Management, to each requirement, to each design element... we can actually tell who in management has a stake in a particuliar _block of code_.

The other neat thing is, the execs can make changes all they want. We really don't care. Because we're on a fixed 3-week development cycle (all the way through the cycle each 3 weeks, culminating in a release) we can either say "sure, we'll do that in the next build" or "scratch the current cycle and we'll do that now". In the latter case, we only lose a maximum of 3 weeks work. Not bad at all, and if management complains, well, we can show them WHY we lost 3 weeks. They shut up pretty quick.

Unfortunately, convincing management that the paperwork we end up doing to improve and maintain our process is a Good Thing, is difficult. If we aren't coding, we must not be working, right? Wrong. Now we have nice graphs showing number of defects in our software falling through the floor, time spent fixing defects falling through the floor, developer productivity skyrocketing... It's fantastic.

Bottom line: Management in some places doesn't WANT responsibility. They want to hand down directives from above, and we are the magical little gnomes who make their projects at 1/4 their salary, if we're lucky. If they go sour on a gnome for whatever reason, they want to be able to fire with impunity. Process is the way to make them eat their own crap whether they like it or not. They WILL end up liking it, and you get your life back.

Re:Wha?

[Anonymous Coward]

Hehe, I know. Nothing like your boss coming two days past deadline and uttering: "I just dreamt up these cool new features we must put in there!". Sigh.

Re:Wha?

[chris_mahan]

Please tell me your company name, so I make sure NEVER to apply there.

Movie trivia. Which movie is this from?
"Attitude reflects leadership".

Re:Wha?

[EnronHaliburton2004]

His scenario reflects 2/3 of the places that I've worked.

However, his solution might work well in many places where feature-creep happens, even when there isn't as much animosity between developers and management.

Re:Wha?

[chris_mahan]

Yeah, I know, and same here. I'm just picky of my work environment.

I think his solution stinks to high heavens.

You know what writers say? "The story needs to be as long as it takes to tell the story well."

I say that trying to make all software development fit the 3-week cycle, is akin to making all software development fit the J2EE Way. I need more flexibility.

Re:Wha?

[cratermoon]

Remember the Titans?

Re:Wha?

[EnronHaliburton2004]

Yeah, you can't force all development into a 3-week cycle, but it can work pretty well for some projects where pieces of the project can be postponed until the next development cycle.

A 3 week cycle could work pretty well in a web-environment (which is what I work in).

Re:Wha?

[neiras]

We evolved a lifecycle that fit our company and our needs. You're free to do the same, and that's what I would recommend to anyone who is serious about writing software.

Just don't discount the value of _having_ a lifecycle. I never said there was One True Way, I just presented our shop's journey as an example.

If it sounds like we're in chains, you're dead wrong. We've never had more freedom. The hard part is deciding you're actually willing to do what is necessary to get it when you work in an environment that needs, but does not understand, software development.

Re:Wha?

[swillden]

"Attitude reflects leadership"

"Remember the Titans".

I'd write it as "Attitude reflect leadership", since that's the way it was said, and because I like the combination of bad grammar (or poor enunciation) and sharp insight. Julius Campbell (Wood Harris), a star black player on the team, said it to Gerry Bertier (Ryan Hurst), the white team captain, when Gerry complained of Julius' bad attitude.

Yes, my kids have watched it so many times I can recite most of the dialog. Good movie, though.

Re:Wha?

[mutterc]

I think this is a market failure, and so clueful management can't help you. From what I've seen, the market will simply not bear the cost of software done right.

Nobody will pay extra for quality software (I'm talking about business customers; individuals don't have any kind of realistic influence on software development). If you (as a developer) tell a customer they'll have to pay $X and wait Y months for the software they want, they'll just buy it from someone else who promises it at $0.5X and 0.6Y months. Therefore, as long as any one company is cranking out crap, all must.

(Perhaps it's software buyers (typically PHB's, after all), who exhibit the kind of inability to learn from experience that we programmers typically attribute to those making the schedules and budgets).

This forces for-profit software development firms to live on the edge, cranking out crap as fast as they can, trying to stay afloat. (And, of course, don't forget that in the U.S., for-profit companies must make ever-increasing profits, so you have to cut something as time goes by. Can't raise prices, oh no, so you gotta cut costs).

Open Source software is immune to these market forces, so it's possible for OSS to not suck.

Re:Wha?

[eraserewind]

As a professional you have a responsibility to the company you work for to make your development process as reliable as possible. If "screaming for new features every release" is throwing your development into chaos, then you don't have a good system. If you can't display a measure of the impact on the codebase of those new features, you have no justification for the any delays caused by them. If you can't show daily measurement of the maturity of the code, how can anybody make a decision about schedules, resources, or when to ship? It'll all be just wild ass guesswork, and you'll suffer as a result.

The unfortunate thing about software development that I've found is that the so called "quality systems" (ISO 9001, CMM, etc..) do nothing to improve the quality of anything that goes on. You have to do it at a lower level than a bunch of documents, and that means build and test automation, document automation, regular release cycles, etc... Anything that has to be done by hand is a waste of time. If it's not automated it's close to worthless. Any "quality engineer" who defines a process for a company should be obliged to supply a tool for automating it.

Re:Wha?

[chris_mahan]

Yeah, and that's why I say it stinks: That their management is so obtuse that the devs felt like they had to create a layer of politics to insulate themselves from irresponsible slave-drivers.

The worse is that the devs think it's normal.

Bitter? Nah...

Re:Wha?

[Mr. Slippery]

If you can't show daily measurement of the maturity of the code, how can anybody make a decision about schedules, resources, or when to ship? It'll all be just wild ass guesswork, and you'll suffer as a result.

You can have all sort of measurements, but the question of whether they are accurate or meaningful is another question...

Re:Wha?

[legirons]

"How about making Management accountable."

Indeed.

"High risk change. Requested at 6.30pm one day, needs to be available by the following morning."

Hmmm, wonder why there's no high-quality code on that project...?

Rise of software liability

[G4from128k]

The Wallstreet Journal has a page B1 article (free via this link?) on buyers trying to hold software providers liable for flaws, damages, bugs, etc. It seems the old EULA disclaimer is not going to hack it anymore. Buyers argue that each software patch is equivalent to a product recall and that vendors should help pay for the cost of patches (AT&T says it sends $1 million per month on patching).

If General Motors can be held liable for damages caused by a defective car part, some argue that software makers should be held liable for damages arising from buggy code.

Re:Rise of software liability

[justasecond]

OK, riight. Thing is, if the code crashes, whose fault is it?

This happened today: customer calls claiming my software is broken -- he's getting "invalid opcode" messages trying to run the thing. Well, noone else gets the same message. Turns out he's running the software on some crap 386. So, is it my software, or: a) an old Windows95 so loaded up with spyware, viruses, registry crap from software uninstalled 5 years ago, outdated network clients, etc. that it takes 5 minutes to boot, or b) bad hardware (memory going bad, drive cache toasted, etc.) or is it my software?

How can you prove in court that these kinds of problems are due to the software and not due to user error and/or incompetence in operating their computers?

It's akin to suing GM because your 50-year old car has bad brakes.

Re:Rise of software liability

[pmike_bauer]

Buyers argue that each software patch is equivalent to a product recall and that vendors should help pay for the cost of patches

Ok, fine. I'll just gouge my customers up front rather than sticking it to them later by not reimbursing them for patching their systems.
Software shops can then sell protection plans along with the product that guarantees a payout in the event of patching.

Do you wanna pay now or later?

Re:Rise of software liability

[G4from128k]

Ok, fine. I'll just gouge my customers up front rather than sticking it to them later by not reimbursing them for patching their systems. Software shops can then sell protection plans along with the product that guarantees a payout in the event of patching.

Do you wanna pay now or later?

Absolutely true! But if a competing software company actually creates quality code, then it won't have to gouge its customers, won't have many pay-outs for defective/patched software, and won't have to sell a protection plan. The other company that can't or won't produce good, low-defect software will find that it is at a price disadvantage.

Re:Rise of software liability

[smileyy]

That would be, I believe, the point of a court of law.

Re:Rise of software liability

[justasecond]

And how is a court of law to determine this? Google "junk science"+lawsuit to see how guillable courts are when it comes to culpability.

Re:Rise of software liability

[smileyy]

...and the better solution is? Yes, the legal system may suck, but that scenario described in the ancestor post *is* was courts are meant to adjudicate.

Re:Rise of software liability

[justasecond]

Not intendending to be sarcastic -- I'm just curious: Software *has* to be different than cars somehow, right? Otherwise, why haven't people been suing MS et al. into oblivion over the last 30 years, as has been done to auto and (especially) small plane manufacturers? In my mind, you are a little tenuous in asserting that software disputes are the domain of the courts.

Re:Rise of software liability

[smileyy]

That's a good question. I imagine part of it has to do with MS software not directly killing anyone. I still don't know where else failure-to-perform-as-advertised or breach-of-contract or whatever disputes would be settled, *besides* a court. Or arbitration, which amounts to about the same thing.

The most important thing is common sense

[FullMetalAlchemist]

The most important thing is common sense; depending on your team, different methodology is needed.

The most important aspect of development for my team today is requirements reuse, sound silly but works great. By following this simple methodology we have made errors nonexistant; it beats unit testing by a mile in efficiency, plus it matches the results.

Most other teams fail with this approach though, and hard. It simply comes down to what the team is made of, mine love it.

Re:The most important thing is common sense

[Anonymous Coward]

A couple of thoughts:

1) As someone said, "Common sense is not that common" :-).

2) IMO unit testing IS common sense. As a matter of fact I can't think of a more sensible thing to do that to test units of code before integration. It's very logical approach that is finally getting the recognition and importance it deserves.

Re:The most important thing is common sense

[FullMetalAlchemist]

I totally agree, for us though, unit testing has already been done; thus, requirements reuse allows us to pick the parts that worked and no need for (much) unit testing, just the final build tests.

Dart

[bill_mcgonigle]

If you want to implement one of these, check out Dart from the guys at Kitware.

I've seen their in-house dashboards and they're quite impressive. These guys eat their own cooking.

What I don't like about XP

[The Slashdolt]

Managers tend to think that gathering proper input, leading to proper requirements, is "hard". But doing this upfront work is required to properly analyze/design/estimate a programming effort. Along comes XP/Agile whatever you want to call it. They say, you don't need everything up front, you can change things as we go, we're "agile". This is what managers want. Every month along the project the requirements change, the design changes, we adapt, this is great. The part they keep leaving out is the fact that change is not any cheaper. With any method you pick, as everyone knows, the later in the project you make changes the more they cost. They always leave off that part.

I can't recall where, but I remember reading the quote somewhere, "you can't refactor an elephant into a cheetah". I don't think many managers truly understand that...

To me XP/Agile is just an excuse that allows marketing and management to not have to do their job.

Re:What I don't like about XP

[chromatic]

With any method you pick, as everyone knows, the later in the project you make changes the more they cost. They always leave off that part.

The fundamental premise of XP is that there are ways to reduce those costs dramatically. A secondary premise is that exposing the costs of those changes at the point of change gives stakeholders more information to evaluate the necessity of those changes.

Re:What I don't like about XP

[The Slashdolt]

Your comment is doing exactly what I complained about above...

...there are ways to reduce those costs dramatically.

I don't doubt there are ways to reduce those costs dramatically. But reducing those costs increases costs in other places. Change is not free. You aren't reducing overall costs, you're just moving them around. You can simplify your design, make it completely decoupled and resilient to change. But, again, this is not free. These decisions have costs. Especially in terms of technologies, performance, etc, etc. XP offloads up front work onto developers later in the project. They don't tell you that the entire project will cost more and take longer, they leave that part off. Management only see, "As requirements change, your software changes". Your comment is an example of how XP fools management into thinking that reducing costs in one area do not impact costs in other areas.

Re:What I don't like about XP

[swillden]

The fundamental premise of XP is that there are ways to reduce those costs dramatically.

Yes, as long as you keep in mind that (a) those costs are still much greater than zero and (b) those costs are also much greater than the cost of doing the up-front analysis.

Designing and implementing for change is a good thing, but it's still more cost-effective to get it right the first time wherever possible.

IMO, some of the ideas in XP are valuable for every development approach, but they don't change the fact that if it's possible you're *much* better off doing solid requirements analysis up front.

A secondary premise is that exposing the costs of those changes at the point of change gives stakeholders more information to evaluate the necessity of those changes.

True, and that's generally a good thing. There is a downside, though -- if the users are cost-sensitive the resulting software is often much less effective than it could have been if the requirement had been understood up front and implemented to begin with.

Incremental requirements analysis coupled tightly to incremental development leads to hodgepodge systems unless the developers are really given free rein to refactor on a whim. And often, even if cost isn't an issue, users are resistant to refactoring in the UI (assuming the system is in production). Even when the reorganized UI would be significantly easier to work with, users are likely to resist having to put in the effort to learn the new UI and change their work habits. It takes a farsighted manager to be willing to repeatedly shake up his peoples' short-term productivity in order make them more efficient in the long term. And even if management is supportive, the users may learn not to request changes because they don't like the relearning their tools.

The more requirements can be understood up front, the less cost needs to be spent by developers, testers and users in managing change. If the requirements can't be understood up front, and if a small prototype or two isn't enough to help the users understand their needs, then XP's approach is the only way to make the development manageable.

But the lack of requirements should be considered a problem, and every attempt made to solve it before starting serious development.

All IMHO, of course.

Re:What I don't like about XP

[chromatic]

...those costs are also much greater than the cost of doing the up-front analysis.

Only as far as the business requirements do not change after the analysis or the initial analysis anticipates changes in business requirements accurately.

That's possible, but I've never seen it happen.

Incremental requirements analysis coupled tightly to incremental development leads to hodgepodge systems unless the developers are really given free rein to refactor on a whim.

Indeed; that's exactly what XP suggests!

In a situation where I had complete, sufficient, and accurate specifications up front, I'd still use most of XP: frequent iterations, customer prioritization, test-driven development, and heavy refactoring. I haven't seen anything else reduce my bug counts or improve my code's maintainability and efficacy as dramatically.

Re:What I don't like about XP

[The Slashdolt]

Only as far as the business requirements do not change after the analysis or the initial analysis anticipates changes in business requirements accurately.

I think you're talking about two things here. The first is the reduction of scope of a large project. Where we know we'll have to do X, but we'll only do x for now. We'll incorporate into our desing the knowledge that we will eventually have to do X. The second is anticipation or prediction of related requirements. As in, I am doing an email client, of course they will want a spell checker so we can anticipate that.

Neither of these things is what I am talking about, or have experienced in my career. My experiences are more along the lines of, "lets build a transaction processing system to do these overly broad things because we really don't know what is going to sell". The development team works on requirements as they see them, very little customer input or involvement. We're told to "just get something together so it can be sold to customers and we can figure out what they really want". A few months later a "product" is produced and sales can't sell it because it doesn't align with the customers needs. They come back with statements that it needs feature X, Y, Z. And of course, every sales person needs different X, Y, Z features. High priority. Management now says, "OK, we've determined that these three features are the highest priorities, now get them done". These three features are usually products in and of themselves. An example would be, adding a complete realtime business intelligence layer on top of a transaction processing app. When we try to explain that these two systems are fundamentally different architectures management will usually say that we should be "agile" and our design must be insufficiently robust to handle "change".

Don't get me wrong, everyone knows requirements change. Requirement changes within the scope of the application are manageable. The issue is when people take the XP/Agile ideas and use them as a replacement for solid upfront product research or customer input.

Re:What I don't like about XP

[cratermoon]

it's still more cost-effective to get it right the first time wherever possible.

Even if that were true, it's never possible; not in the real world. Even in the pretend world, the effort needed to get the theoretical correct up front and for all time requirements is prohibitive. In the same theoretical play world where getting it right the first time is possible, you'd still find that the company would be bankrupt and the business problem to be solved would be moot before you were done.

Re:What I don't like about XP

[bruthasj]

You're forgetting that the main driver of change is the customer/user. And, trust me, they will find every opportunity to squeeze stuff out of you. Designing everything up front will only make you obstinate about every little thing when the customer comes to ask for something. If you're obstinate or keep telling them it's a PCN (project change notification) or CR requiring lots more money, there are plenty of other vendors to step in and do otherwise.

XP/Agile is a way to capture this process. You don't have to do it *all* to gain benefits. Take XP as a menu of things to do. Pick and choose what you want to do and then be off with your project. Hating it *all* based on marketing's and management's laziness would be a shame. (Developers, marketers, and managers will all be lazy anyway no matter what method you use.)

Re:What I don't like about XP

[Anonymous Coward]

The point with XP is so that you can quantify the cost of the changes - because mgmt is going to make changes anyway. The big XP practice that points this out is the Planning Game, where you get to ask mgmt "If you add this, you need to take out X tasks - which ones can you give up?"

Re:What I don't like about XP

[swillden]

Only as far as the business requirements do not change after the analysis or the initial analysis anticipates changes in business requirements accurately. That's possible, but I've never seen it happen.

Me neither, not 100%. But my point is that doing as much as you can up front to really understand the problem pays off, and the payoff is greater than linear.

If you can get the requirements half right before you start, your project will be much shorter and much cheaper than if you just start building incrementally.

If you can't get any sort of handle on the requirements before you start, because no one knows what they want, then agility is your only hope.

Indeed; that's exactly what XP suggests!

Of course, which is why I mentioned it. But see all of my other reasons why it often won't really happen, particularly if the decisionmakers are cost-sensitive.

Re:What I don't like about XP

[swillden]

Even if that were true, it's never possible; not in the real world.

Of course not. Not 100%. My point was that you're money ahead by getting as much as you can up front.

In my case, the software development I do is on a contract basis, often for firm fixed price. So I'm very accustomed to having to get the requirements very close to complete and correct up front before planning the development effort (and including some padding for the inevitable changes!).

Re:What I don't like about XP

[Beek]

One of the cornerstones of XP is to have a customer onsite that knows what they want. You won't find out everything you need to know if you only gather requirements up front; having the expert always available to you allows you to fill in these gaps. And since you have the customer onsite, it's sensible to do less of the upfront work. You just get right into implementing the most important feature.

Whether or not you'll be able to get an expert is another question. But if you read Agile Project Management and XP Installed, they both offer solutions to that... If you can't have a customer onsite, find someone to act on behalf of the customer. Highsmith refers to this person as a Product Manager.

Regardless, with XP/Agile, you get MORE customer input. And you still get it early. If you don't have the tight feedback loop, you're not following the Agile methodology.

The example you've given is what XP/Agile are trying to solve. Get the customer to tell you what they want, then do the simplest thing that will work to deliver that value to the customer. Overly broad solutions don't happen. Current solutions are broadened when necessary.

You can't do XP/Agile without a customer telling you what they want. The first three chapters of Extreme Programming Installed drive this point home, I suggest you read them.

Re:What I don't like about XP

[chromatic]

Ah, I see our differences now. My experience is in building software for specific clients. Yours sounds closer to off-the-shelf software.

I suspect that having greater knowledge of what your target market wants and needs will be helpful, which is similar to what you said earlier. You're right; if you don't have a clear customer capable of identifying and prioritizing actual business needs for the software, XP's planning features won't work very well.

Re:What I don't like about XP

[cratermoon]

Re: Padding for the inevitable changes.

That comment makes it abundantly clear that you know that it's never right up front. Why pad then? That's either a) cheating the customer, when your estimate turns out high or b) a gigantic risk of your company's profits if you guess low. Do you think the customer doesn't know you are padding? Of course they do -- negotiations up front on scope/price/schedule then end up being a struggle to push the padding one way or another. XP and similar agile methods want you and the customer to honest with each other and admit you don't know how big the thing will be, but that you will start delivering business value from very early, and allowing the customer to steer the project as it goes.

Consider two scheduled airline flights. On the first, the pilot points the nose in the direction he believes is towards the destination at take off and then after X hours begins a descent and landing expecting to be in the right place, and if not, hoping he put in enough extra fuel to be able to make it the rest of the distance to the airport. On the second, the pilot monitors the course, navigates along the way, and steers the plane with the necessary course corrections. He took off with sufficient fuel to make a normal flight, but all along the way he is sure he stays within range of safe landing sites in case of emergency.

Which flight would you take?

Re:What I don't like about XP

[swillden]

Customers don't hire my company to get the lowest possible price, they hire us to get a guaranteed-to-work solution, and sometimes to get a guaranteed price. They know that we have the resources to do the job (whatever it may take) and when they sign a firm fixed-price contract they know that we've added plenty of contingency to cover the risks, and that we intend to walk away with a very healthy profit, and hope to have an insanely high profit, but that's okay because they've run their numbers and determined that the benefit to their business of having the software is worth the price they've agreed to. There's no dishonesty here: Both parties know what the other is doing, and it makes sense for both.

Yes, my company takes a risk -- there's no way to sign a firm fixed-price contract without taking a risk! We try to limit the risk by nailing down the requirements as tightly as possible beforehand, and by detailing every assumption we can think of that went into the estimate. If the customer changes the requirements or violates one of the assumptions, then we have an opportunity to renegotiate, if we think we need to (in order to maintain high customer satisfaction, we try to head off those changes rather than looking for renegotiation opportunities).

But, at the end of the day, we take on significant risk. We price that risk in, knowing that we'll lose on some contracts, but trying to balance things so that we're profitable overall, yet without pricing ourselves out of the market.

According to the XP theory that requirements simply cannot be known in advance with sufficient accuracy, we should be broke. Our services business is quite profitable.

Really Make It Real

[4of12]

These are all good ideas, the unit testing, the automated frequent testing, etc.

Having experience a few crashes of bleeding edge versions of evolution and firefox with the automated calling back to the developers about the crash symptoms got me to thinking that having actual use (and abuse) be automatically incorporated into test suites might really abet the development of less crash prone code.

Despite the capability of automated testing to test many more features than can be done by hand, new applications have so much context and so many options that we need to test for what the users are actually doing with the application. Not just what we think they're doing, what we hope they're doing, but what they're really doing.

The most important bugs would be the ones that happen to the greatest number of people the most times.

Harvesting application interactions and sending them back to the test suite has a lot of value, but it's up to the developers to do this in ways that are sensitive to the user's need for privacy, too.

thinly veiled ad for a startup

[idlake]

This whole story looks like a thinly veiled ad for a startup. Their main product isn't the overall reporting of test coverage and other metrics, their main product is an automated test case generation tool (one that seems to generate oodles of data, just no anything that looks very useful).

Should you collect statistics on your project, bugs, test coverage, and all that? By all means. And there are lots of tools to do that, free and commercial.

Re:thinly veiled ad for a startup

[Anonymous Coward]

Have you taken the time to look at the recorded webinars before pronouncing this tool not useful?

Re:thinly veiled ad for a startup

[Anonymous Coward]

I gave my first impresion ("it doesn't look useful") after reading the materials on the site.

What I find off-putting about the whole site is that it mostly seems to emphasize reputation ("worked at Google", "worked at Sun"), not technical information or facts.

If a startup requires listening to a Webinar before any convincing technical information comes across, well, it has a problem.