Avoiding Mistakes Can Be a Huge Mistake

theodp writes "No doubt many will nod knowingly as they read Paul Graham's The Other Half of 'Artists Ship', which delves into the downside of procedures developed by Big Companies to protect themselves against mistakes. Because every check you put on your programmers has a cost, Graham warns: 'And just as the greatest danger of being hard to sell to is not that you overpay but that the best suppliers won't even sell to you, the greatest danger of applying too many checks to your programmers is not that you'll make them unproductive, but that good programmers won't even want to work for you.' Sound familiar, anyone?"

Comments

Perhaps

[Thelasko]

Perhaps programmers that have consistently good code should have some value placed on them. We'll call it "Karma". Programmers with good Karma get audited less often than others. If they fail an audit, they loose some "Karma" and have to write a bunch of excellent code to get it back.

Re:Perhaps

[Anonymous Coward]

Code reviews teach the reviewers as much as they check on the author. Why would you deny the lesser programmers the joy and experience of looking at good code?

Re:Perhaps

[YourExperiment]

Do programmers also loose karma for being fast and lose with their spelling?

Re:Perhaps

[Anonymous Coward]

Do programmers also loose karma for being fast and lose with their spelling?

/irony

Re:Perhaps

[Pogue Mahone]

That's exactly the kind of check that is harmful, according to the article. Who determines what is "excellent"? Against what benchmark? Who performs the audits? Who checks that you have spelled "lose" correctly?

Re:Perhaps

[enjo13]

How do you identify "good code"? That's one of the great problems we have as software developers. Quantifying 'good' code is extraordinarily difficult. Code reviews do an excellent job of identifying clever code, but rarely capture the full utility of what is being written. You may think you know good code when you see it, but over the course of my career I've become convinced that is not true at all.

Really the problem is that the only way to truly measure code quality is by seeing how it runs in a production environment. Even then I can easily quantify the quality of the teams overall output (does it work? does it work consistently?), but tracing that back to an individual programmer is often nearly impossible. Systems tend to interact with each other, and placing blame is not an exact science. The gulf between 'good' and 'good enough' is not nearly as wide as it seemed when I was a novice programmer.

Great code almost never breaks. Good code works most of the time. Poor code is another matter.

Poor code is easy to spot. Poor code never works. It's ugly. It's complex. It's stateful. It's jump off of the screen and practically begs to be put out of its misery.

That's precisely why companies have processes and checks. They are an attempt to catch marginal code and make it 'good enough'. The problem, as the article points out, is that in the process they often inspire great coders to deliver marginal code themselves.

The secret is to spot (through some mixture of science and art) great programmers and provide them with the freedom to write great code. If circumstance requires you to hire marginal programmers, then by all means put the process in place to make sure that what they do doesn't detract from the work your best and brightest are doing. Separate them as best you can. Limit how their systems interact.

But whatever you do... don't limit your best programmers, as they are far more valuable than hundreds of poor ones.

Re:Perhaps

[postbigbang]

To wit:

Excellence can be determined in various ways, often through documentation, the great allergen of programmers. If you can't explain it, it isn't really done.

The benchmarks can also be defined as well. They need to be met. Make the bars well known, and what must be done to meet them.

There are great references for auditors, too. Feeling a little pain, are you? Had to throw in the grammar nazi reference?

Re:Perhaps

[ChromeAeonium]

Do programmers also loose karma for being fast and lose with their spelling?

/irony

They can be docked karma that way, but when they're not sure about something, they can cover their asses and submit anonymously. That way, if something totally whooshes over their heads, they're in the clear. They can later correct their own dumbass mistakes unanonymously and whore karma instead of losing it. What a perfect system!

Re:Perhaps

[syousef]

Perhaps programmers that have consistently good code should have some value placed on them. We'll call it "Karma". Programmers with good Karma get audited less often than others. If they fail an audit, they loose some "Karma" and have to write a bunch of excellent code to get it back.

That's awful in so many ways.

For starters look at how poorly Karma works here. It serves to re-enforce awful sheep mentality. Just try putting down Google, Apple or Linux. Or try praising Microsoft.

Next what you're proposing creates a negative feedback loop. A developer codes well and gets through a few audits. Now they're trusted, they can afford to let things slip for some time before anything is caught. There's less incentive to keep producing good code, and there's more of a chance that an error will slip through. No one is perfect and mistakes will happen. The way to protect against them is to ensure there's some redundancy, and taht is exactly what a code review provides.

Also consider retention rates and the average time a developer spends at a company. Does an expert or lead programmer start off having every little thing reviewed? Who's qualified to do that? Or are they trusted based on heresay and a resume? If so how long will it take to find a dud programmer?

Next consider what effect it will have on the morale of a struggling programmer, or one that doesn't cope well with reviews. Especially a junior one whose abilities can be salvaged. A co-operative might work, but constantly giving more and more high pressure code reviews is just about guaranteed to break such an individual.

Finally, you should realize that such "karma" already exists informally and that making it a more formal process achieves very little. In other words developers very quickly get a feel for what the strengths and weaknesses of another developer are.

Re:Perhaps

[mevets]

The "robustness" test isn't good enough either. Sometimes poor implementations of good ideas spurn enough innovation and demand that its marginal quality is irrelevant. The early web browsers, email programs, etc.. were likely neither robust or well implemented. On the other hand, some solid, robust good implementations of ideas are so intractable, that at best they merely serve the original purpose, and at worst are like an albatross around the necks of future developers ( usb anybody ?).

David Parnas proposed that a poor programmer could create demand for dozens of programmers; a unique software situation, as a poor mechanical engineer doesn't scale nearly so well.

Re:Perhaps

[sammyF70]

Google is evil incarnated, Apple is style-over-function overpriced junk, and "The Year of The Linux Desktop" ain't coming soon.

Microsoft, on the other hand, has some really funny employees (and Reversi).

/em ducks

Re:Perhaps

[pe1rxq]

Benchmarking is exactly were the problem lies....

Having your code reviewed by a peer who can actually comment on it and understand what you are (or are not) doing is not bad. Either they understand it, or you have to defend your work with actual technical arguments. The end result is that you both have an opportunity to learn.
Having your code reviewed by a mindless idiot comparing it to the official procedure is bad.... Even worse when the idiot is replaced by a program. Forcing everybody to follow a single holy procedure simply reduces all code to be mediocre.

Re:Perhaps

[ScrewMaster]

Do programmers also loose karma for being fast and lose with their spelling?

/irony

They can be docked karma that way, but when they're not sure about something, they can cover their asses and submit anonymously. That way, if something totally whooshes over their heads, they're in the clear. They can later correct their own dumbass mistakes unanonymously and whore karma instead of losing it. What a perfect system!

Huh. Too bad Slashdot doesn't have a system like that.

Re:Perhaps

[ifdef]

We have a more informal system where I work. Whenever anybody checks in some code, whoever wants to automatically gets an email to notify them (for example, I am set up to receive notifications for any changes in a few directories where I do most of my work). Anybody who wants to then reviews the change. If there is nothing to comment on, it's completely transparent, and the person who checked in the changes is not even aware that they got reviewed. If there IS something to comment on, most likely someone will talk to the original coder (or send them an email), saying something to the effect of "by the way, did you consider such-and-such?", or maybe even "good idea!"

The system keeps track of who reviewed what change. As a check on the process, if there is any change that has been in the system for x days but has not been reviewed by at least m developers and n testers, the original coder gets an automatic email saying "please arrange for someone to review your code."

This is, of course, in addition to the automatic emails that get sent if a change actually BREAKS something. Those get dealt with right away.

So the review process does no harm to anybody's morale, except in the cases where that person really is producing bad code.

Strangely enough, it seems to work quite well.

As for your other point, about "A developer codes well and gets through a few audits. Now they're trusted, they can afford to let things slip for some time before anything is caught. There's less incentive to keep producing good code": I don't want anybody with that attitude in my group, period. I am not in any kind of supervisory position, but if there is someone on the team who only produces good code because of some "incentive", and the team lead doesn't do anything about that, then I don't want to work for that team lead, either. I will vote with my feet, and I don't think I would be the only one. And then this would become, no doubt, a much more "normal" software department, instead of an amazing one.

Re:Perhaps

[kbrasee]

/irony

/woosh

/irony

Error: Cyclic Dependency

Re:Perhaps

[ChrisA90278]

I don't think you even need "good code". I worked on a project that eventually failed and all the code we looked at in those reviews was "good". Either that or we made it good. The big problem was that it did the wrong things.

The problem is not with the people who write the code. Most are OK and reviews catch gross errors but in out case we had some basic "big picture" ideas wrong.

Microsoft Vista is a good example of this. Likely the code would pass a review and has few mistakes but the problem is the dumb ideas that got written into the specifications.

It's like the "bridge to nowhere" problem. Good, competent structural engineers build something no one wants or needs.

Re:Perhaps

[lgw]

Excellence can be determined in various ways, often through documentation, the great allergen of programmers. If you can't explain it, it isn't really done.

Suppose you have a programmer who's 10x as fast as your benchmark standard guy at producing tested, debugged code that meets requirements, but slower at writing docs. Do you make him write docs, or attach a junior programmer to him for that purpose?

Of course, somewhere up the ladder a software engineer's job becomes mostly writing docs (regardless of whether you go management or architect), but still - tying that anchor around the neck of your best dev is exactly the sort of thing Paul Graham was on about.

Re:Perhaps

[lgw]

I don't think you're disagreeing with me. Given whatever standard of quality you wish, there is at least a 10 to 1 productivity variance in the creation of software between software professionals. Identifying developers who are on the tail end of that bell curve and removing obstacles from their path is the key to both productivity and retaining your super-programmers.

The guys on the 10x end of the spectrum aren't just more productive - they can solve problems that the ordinary coder simply cannot solve at all. If you're part of the millionth re-invention of an inventory database, this doesn't much matter, but if your business sells software that soves new problems, these guys make a huge difference.

Of course, everyone thinks they're a super-coder when they're young, just like everyone thinks they're an above-average driver, but recognizing (and retaining) real talent is supposed to be a development manager's job.

Re:Perhaps

[lgw]

My fundamental point is not that documenation and paperwork should be avoided, but that in some cases it's just a stupid allocation of resoruces to have certain developers do that work. In the old days, IBM would pair a tech writer with *every* programer, so that really good docs could be produced with a minimum of distraction to the coding. It's hard to know whether that produced the best code (though there's a lot to be proud of in 70s mainframe code), but it produced lots of *really* good documentation.

Re:Perhaps

[networkBoy]

wow, a Beowulf cluster of /irony

Re:Perhaps

[russotto]

Documentation may be necessary; certainly documenting interfaces is, and some amount of internals documentation is prudent. But no amount of documentation will make poor code anything but poor. Nor will any lack of documentation make excellent code poor; it merely makes it undocumented.

Doing that requires submission to the processes needed by others and being both courageous and disciplined enough to do so.

Courageous submission... wasn't that one of the Ingsoc slogans?

A fine balance

[pwnies]

Obviously you can't go too much in the other direction either. The checks are there for people who write code seen on http://www.thedailywtf.com/ who actually click the flashing banner that says they've just won lots, and for those who open up the .exe they found in the email that contains 'instructions on how to get a bigger pen15 2day'.
It's like anything else in life. The sins of one hurt everyone.
There will always be people who get shampoo in their eyes, and because of that these checks will always exist.

It's official

[Anonymous Coward]

From TFA: "Programmers are unlike many types of workers in that the best ones actually prefer to work hard."

Thanks for telling me I suck, Paul. Now excuse me while I loaf.

More checks!

[I.M.O.G.]

From the article:

Whenever someone in an organization proposes to add a new check, they should have to explain not just the benefit but the cost. No matter how bad a job they did of analyzing it, this meta-check would at least remind everyone there had to be a cost, and send them looking for it.

So bureaucracy has a cost in that it places lots of checks on things, and the solution to that is adding more checks?

Sounds like solid bureaucracy to me!

Death March

[micromuncher]

I find the premise of the essay wrong. Go read "Death March" by Ed Yourdon (http://www.yourdonreport.com/). Most of the time problems aren't processes - they're people.

"Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway."

This is a red flag. Coders that just code are part of the problem of a death March. Who has worked with wunderkind that churns out 16 hours of useless bug ridden code? That refuses to write unit tests because they slow him down? And at the end of the project look back, the MetricsReloaded tells you "Yes, that developer wrote the most code, but it is also the most defective and has the least coverage?"

Good processes are adaptive. Good people are agile. You can't build skyscrapers on spec. I am so annoyed by people that push a methodology or ideology that cannot also cite the specific historical evolution of software processes.

Agreed, Too Much Oversight Kills

[cgifool]

My group is a prime example. We all worked for a startup that generally released a new version of our application about 3 times per year. Over a few years we had developed a nice lean development process that involved documenting our design, but only in enough detail to be able to fairly accurately estimate the development effort (in X days, X weeks, or X months).

Based on the estimates, the biz dev group would then pick and choose features to make up 3 months dev + test time.

This worked great, and we pretty much never had a late shipment and few bugs.

Then we got acquired by a giant 3-letter company with huge amounts of development process and tons and tons of "standards", and immediately were ordered to begin a 16 month release consisting of removing all open source and complying with standards. All their architects routinely veto our decisions and our design documents must be very very detailed and approved via heavyweight process before implementation can begin. 24 months later we're still in development, only recently the last design document was finally approved; at the moment it seems we'll be about 12 months late in total.

Now they're asking us why we have so many tests planned, and making us remove half of them. Supposedly quality is a major priority, but they have no testing group; only people to enforce standards. All tests and test cases are written and implemented by the developers themselves.

Dont even get me started about the outsourcing issues.

As a programmer I want to avoid mistakes...

[syousef]

What I don't want is:

- To be reprimanded for every little mistake I make, or worse be put in a position where a little mistake on my part can cause a huge, expensive and/or very visible problem

- To be forced to comply with procedures that do not in fact improve quality but do require 90% of my time leaving me with 10% time to program

- To have no creative input into my code.

There are good ways to achieve similar goals without the above antics. Continuous integration comes to mind. Well qualified specialist testers for User Acceptance is good too. Avoiding mistakes in a way that is programmer friendly will actually improve morale and an employer more desirable. The trouble is that too many employers try to equate software manufacture with mass production factory work in every way and treat their programmers accordingly. If you look at the kind of work a programmer enjoys vs the kind a factory worker is expected to do, no wonder they leave or won't join.

Nothing changes. Really.

[onescomplement]

Paul, as usual, backs into the key argument.

This keeps coming up in various shapes and forms but the fact of the matter is that brilliant, high producers aggregate in places; and so do idiots.

Tom DeMarco ran a study of this in the 80s wherein teams were asked to solve the same problem. He expected a scatter-plot. It was a 45 degree line between the people who knocked the problem off and those who were clueless.

What didn't matter:

Platform. Language (except assembler, those folks were _lost_.) Operating System.

What did matter:

Team coherence and capability.

Design and planning; raw ability to design and plan as a coherent team. And not just a bunch of losers following a Pythonesque "Book of Common Knowledge."

(I have been to many "Does the witch weigh less than wood" meetings...)

Look at the back cover of Boehm's "Software Engineering Economics." What he _measured_ was that team capability overarchs everything. Period.

I would also ask you to look at the surface exposure of development. Folks who develop on the shoulders of many giants can and should be trying lots of stuff, because that's why platforms are built.

Folks working closer to the core (the OS, drivers, fundamental code) don't change as quickly, nor should they.

I've worked as a hatmaker (sheer, unbridled creativity with fancy ribbons and flowers and such) for high-end ladies and I've sat, confounded by bad documentation for UARTS.

Two different regimes.

Been there, agree with that

[blake182]

This story hits very close to home for myself -- I've sold two companies to larger companies where they commercialized my software. When you're a scrappy startup, you ship instantly. When you get incorporated into a larger organization, you don't ship instantly, which hurts because your intrinsic motivation (for the A-listers and entrepreneurs anyway) is shipping.

One shocking thing in the article for me is just how much people would give up in order to ship faster -- startups that got acquired would give up some of the acquisition money in order to ship faster in the new company. It's probably a limited sample, but I know I've felt that way. This is a large portion of what I call "suck" -- things that slow down shipping. I'm not anti-QA, but after a particular point all you did was slow down shipping.

One satisfying aspect of the work I did at the last acquiring company is that every time I checked in code, I knew I could with a straight face recommend that we ship it. I mean, it wasn't a full QA pass, but it was code with a supporting set of automated unit tests incorporated into a system designed as an extensible framework. Any negative impact would be isolated to that specific functionality (high cohesion, low coupling). A small group of internal power users and my own server would take the daily builds and give feedback as to how it felt in production and report any major issues.

The message here seems to be "if you can optimize the process in some way, optimize it so you ship faster". And in the meantime, go ahead and pretend like you're shipping every day (a complete, ready-to-go, high quality build). You'll be surprised how much better you feel even with that.

Mistakes can mean breakthroughs

[mdrplg]

I've worked for both types of companies and I can say that the more 'checks' to prevent mistakes the more ossified the code becomes. Sometimes big mistakes lead to a breakthrough in understanding the problem. It often seems that slow release-to-server cycles inhibit the ability of programmers to learn from their mistakes.

Re: loose karma

[macraig]

Yes! Free Karma... right after Willie! I'm starting a grassroots movement to save the Karma from untimely demises.

Re:I have mixed feelings.

[russotto]

At the big company I worked at, someone added some features to code that I wrote. It broke my code. I wanted to go in and fix it. Why not? I knew how it worked. I couldn't without a defect written by a tester.

Annoying, but easy to handle. Testers love to find bugs, whether it be for the joy of crushing a young programmer's spirit, or for the look of fear in the eyes of the product manager as the release date approaches. Point a tester towards the bug, and he'll go right ahead and write a defect, possibly cackling evilly as he does so.