Slashdot Mirror
Avoiding Mistakes Can Be a Huge Mistake
theodp writes "No doubt many will nod knowingly as they read Paul Graham's The Other Half of 'Artists Ship', which delves into the downside of procedures developed by Big Companies to protect themselves against mistakes. Because every check you put on your programmers has a cost, Graham warns: 'And just as the greatest danger of being hard to sell to is not that you overpay but that the best suppliers won't even sell to you, the greatest danger of applying too many checks to your programmers is not that you'll make them unproductive, but that good programmers won't even want to work for you.' Sound familiar, anyone?"
Perhaps programmers that have consistently good code should have some value placed on them. We'll call it "Karma". Programmers with good Karma get audited less often than others. If they fail an audit, they loose some "Karma" and have to write a bunch of excellent code to get it back.
One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
Obviously you can't go too much in the other direction either. The checks are there for people who write code seen on http://www.thedailywtf.com/ who actually click the flashing banner that says they've just won lots, and for those who open up the .exe they found in the email that contains 'instructions on how to get a bigger pen15 2day'.
It's like anything else in life. The sins of one hurt everyone.
There will always be people who get shampoo in their eyes, and because of that these checks will always exist.
From TFA: "Programmers are unlike many types of workers in that the best ones actually prefer to work hard."
Thanks for telling me I suck, Paul. Now excuse me while I loaf.
From the article:
So bureaucracy has a cost in that it places lots of checks on things, and the solution to that is adding more checks?
Sounds like solid bureaucracy to me!
Overclockers
At big companies, software has to go through various approvals before it can be launched. And the cost of doing this can be enormousâ"in fact, discontinuous. I was talking recently to a group of three programmers whose startup had been acquired a few years before by a big company. When they'd been independent, they could release changes instantly. Now, they said, the absolute fastest they could get code released on the production servers was two weeks.
At the big company I worked at, someone added some features to code that I wrote. It broke my code. I wanted to go in and fix it. Why not? I knew how it worked. I couldn't without a defect written by a tester.
On the other hand, if it was reviewed, the feature wouldn't have gone in; at least not without my input - I would hope.
I find the premise of the essay wrong. Go read "Death March" by Ed Yourdon (http://www.yourdonreport.com/). Most of the time problems aren't processes - they're people.
"Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway."
This is a red flag. Coders that just code are part of the problem of a death March. Who has worked with wunderkind that churns out 16 hours of useless bug ridden code? That refuses to write unit tests because they slow him down? And at the end of the project look back, the MetricsReloaded tells you "Yes, that developer wrote the most code, but it is also the most defective and has the least coverage?"
Good processes are adaptive. Good people are agile. You can't build skyscrapers on spec. I am so annoyed by people that push a methodology or ideology that cannot also cite the specific historical evolution of software processes.
/\/\icro/\/\uncher
Anyone else pick up on this -
Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times. And when I used to mow lawns, I definitely didn't prefer it when the grass was long after a week of rain.
Just because YOU didn't like being busy at your other jobs doesn't mean everyone else in the world feels the same way. I worked in fast food and when I did, I liked it a lot better when it was busy. A 10 hour shift flies by when you're making orders non-stop, but once closing time came and it was time to clean up time seemed to stand still. I was still working but the work I was doing was so mundane and slow paced (mopping floors or washing the nights dishes) that it seemed to take forever.
There's also the possibility that you're a hobby programmer and not a career programmer. You might like programming outside of work, but there are people that wrote code just because it'll pay the bills. Those people generally like writing code as much as you liked mowing long grass.
My group is a prime example. We all worked for a startup that generally released a new version of our application about 3 times per year. Over a few years we had developed a nice lean development process that involved documenting our design, but only in enough detail to be able to fairly accurately estimate the development effort (in X days, X weeks, or X months).
Based on the estimates, the biz dev group would then pick and choose features to make up 3 months dev + test time.
This worked great, and we pretty much never had a late shipment and few bugs.
Then we got acquired by a giant 3-letter company with huge amounts of development process and tons and tons of "standards", and immediately were ordered to begin a 16 month release consisting of removing all open source and complying with standards. All their architects routinely veto our decisions and our design documents must be very very detailed and approved via heavyweight process before implementation can begin. 24 months later we're still in development, only recently the last design document was finally approved; at the moment it seems we'll be about 12 months late in total.
Now they're asking us why we have so many tests planned, and making us remove half of them. Supposedly quality is a major priority, but they have no testing group; only people to enforce standards. All tests and test cases are written and implemented by the developers themselves.
Dont even get me started about the outsourcing issues.
What I don't want is:
- To be reprimanded for every little mistake I make, or worse be put in a position where a little mistake on my part can cause a huge, expensive and/or very visible problem
- To be forced to comply with procedures that do not in fact improve quality but do require 90% of my time leaving me with 10% time to program
- To have no creative input into my code.
There are good ways to achieve similar goals without the above antics. Continuous integration comes to mind. Well qualified specialist testers for User Acceptance is good too. Avoiding mistakes in a way that is programmer friendly will actually improve morale and an employer more desirable. The trouble is that too many employers try to equate software manufacture with mass production factory work in every way and treat their programmers accordingly. If you look at the kind of work a programmer enjoys vs the kind a factory worker is expected to do, no wonder they leave or won't join.
These posts express my own personal views, not those of my employer
Now that I've I know, I'm going to dismantle all my QA procedures and hire the sloppiest programmers I can find!
How about I take every little thing you say and take it to the farthest possible extreme without any regard for the subtleties in what you originally said?
What you said sounds like a great idea! Why don't we KILL ALL THE PEOPLE WHO DO QA, like you are clearly suggesting, rape and torture their families and then hire pedophiles and trained gorillas to do the work?! Isn't that exactly what you were saying just now?! Isn't it?!!
This keeps coming up in various shapes and forms but the fact of the matter is that brilliant, high producers aggregate in places; and so do idiots.
Tom DeMarco ran a study of this in the 80s wherein teams were asked to solve the same problem. He expected a scatter-plot. It was a 45 degree line between the people who knocked the problem off and those who were clueless.
What didn't matter:
Platform. Language (except assembler, those folks were _lost_.) Operating System.
What did matter:
Team coherence and capability.
Design and planning; raw ability to design and plan as a coherent team. And not just a bunch of losers following a Pythonesque "Book of Common Knowledge."
(I have been to many "Does the witch weigh less than wood" meetings...)
Look at the back cover of Boehm's "Software Engineering Economics." What he _measured_ was that team capability overarchs everything. Period.
I would also ask you to look at the surface exposure of development. Folks who develop on the shoulders of many giants can and should be trying lots of stuff, because that's why platforms are built.
Folks working closer to the core (the OS, drivers, fundamental code) don't change as quickly, nor should they.
I've worked as a hatmaker (sheer, unbridled creativity with fancy ribbons and flowers and such) for high-end ladies and I've sat, confounded by bad documentation for UARTS.
Two different regimes.
This story hits very close to home for myself -- I've sold two companies to larger companies where they commercialized my software. When you're a scrappy startup, you ship instantly. When you get incorporated into a larger organization, you don't ship instantly, which hurts because your intrinsic motivation (for the A-listers and entrepreneurs anyway) is shipping.
One shocking thing in the article for me is just how much people would give up in order to ship faster -- startups that got acquired would give up some of the acquisition money in order to ship faster in the new company. It's probably a limited sample, but I know I've felt that way. This is a large portion of what I call "suck" -- things that slow down shipping. I'm not anti-QA, but after a particular point all you did was slow down shipping.
One satisfying aspect of the work I did at the last acquiring company is that every time I checked in code, I knew I could with a straight face recommend that we ship it. I mean, it wasn't a full QA pass, but it was code with a supporting set of automated unit tests incorporated into a system designed as an extensible framework. Any negative impact would be isolated to that specific functionality (high cohesion, low coupling). A small group of internal power users and my own server would take the daily builds and give feedback as to how it felt in production and report any major issues.
The message here seems to be "if you can optimize the process in some way, optimize it so you ship faster". And in the meantime, go ahead and pretend like you're shipping every day (a complete, ready-to-go, high quality build). You'll be surprised how much better you feel even with that.
I've worked for both types of companies and I can say that the more 'checks' to prevent mistakes the more ossified the code becomes. Sometimes big mistakes lead to a breakthrough in understanding the problem. It often seems that slow release-to-server cycles inhibit the ability of programmers to learn from their mistakes.
Today is an ephemeron, doomed to the crypt of yesterday.
This sort of thinking does not justify the new userpage.
Sometimes, life itself is sarcasm...
Yes! Free Karma... right after Willie! I'm starting a grassroots movement to save the Karma from untimely demises.
The time for that was what I missed on my last job. While there was not much oversight of how we actually wrote our code, deadline pressure always created a feeling of "you cannot afford taking that time just NOW".
The ultimate downer was when there was a gap between projects, one release had just been shipped and management hadn't made its mind up about what features to put into the next version. But even then there was no "business justification" for rewriting old crappy code. Instead we ended up writing some middleware for third party closed source software to make management's life easier. IMHO a complete waste of time.
In hindsight, that was a lot more motivation destroying than the development process as such.
C - the footgun of programming languages
The problem is that what you are describing is nothing at all like what the market actually wants. The market actually wants massive feature lists delivered as fast as possible, and their level of caring for whether or not quality is there is very low. This is very unlike, say, a bridge, where the primary public concern is going to be that the bridge not fail and kill them. Or an electrical product where the primary public concern is that it not short out and start a fire that kills them.
When things go wrong in other fields of engineering, it can frequently kill you. In most software engineering, not so much, and in the areas where it CAN kill you (think airplane fly by wire control software), you should be unsurprised to find out that many of the quality and testing standards are quite the same as in other fields.
Most software engineering is more like movie publishing. The audience quickly gets satiated, and is mostly looking to the next great thing with better FX, and not so much caring if you have a few flaws in your plot (see recent Batman movie if you are unclear on this).
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking
No, there really are programmers who are 10x as productive as "normal", and of course programmers who contribute negatively. You'll know either extreme when you work with one.
Boss: we've had a team of 5 on this problem for a couple months now, and they're way behind - could you take a look?
Dev: sure, give me a couple of weeks.
Two weeks later.
Boss: so, can how's that project going?
Dev: oh, I'm done.
Boss: Done? Great - what was your analysis; how can we speed things up?
Dev: No, no, I'm done with the project. The approach that team had was crap, so I just did it right. I'm running my unit test now, but it looks good so far.
Boss: -speechless-
You may not believe it until you work with someone like that, but they're out there (and large companies do a terrible job of retaining them - as Paul Graham suggests, the main thing devs like that want is to not have obstacles to their productivity).
Socialism: a lie told by totalitarians and believed by fools.
I know what "productive" means. Knowing that is part of my job. "Productive" is delivering features in a maintainable, supportable way, per unit time. Sizing features and defining "maintainable" and "supportable" in terms of concrete rules and best practices is a large part of the skillset of a very senior engineer. Finding the hyper-productive programmers and twisting management's arm to retain them at all costs is another.
Again, the 10x developer sounds like a myth until you work with him -- I'd say it's the top 1% -- but 3x developers are common. The "contributes negatively" programmer is someone you've met, though the hallmark of bad management is to encourage that guy to work more.
There's a reason that "real" engineering companies have fellowships and Distinguished Engineer titles to recognize, empower, and retain the top 1% engineers on other fields.
Socialism: a lie told by totalitarians and believed by fools.
"The Other Half of 'Artists Ship'" presumes that large organizations are motivated by efficiency and effectiveness. This presumption completely misunderstands large organizations. Large organizations come to exist because they have tapped into a large reliable long-term cash stream that can survive their collective incompetence. With their organizational funding assured, all that is left to do is to divy up the organizational spoils amongst the employees, management, and shareholders. And then the key determinant success in such an organization becomes not to be blamed for making a mistake. No employee will get blamed for inadvertently excluding a superior vendor from an acquisition process; most likely no one in his/her organization will even know it happened. They can get blamed for having too lax an acquisition process, however. No employee will be blamed for having too much testing or too much project management or too many software quality checks. They can get blamed for releasing defective software onto a production system without sufficient checks. So the internal goals of the organizational actors are perfectly reflected in the external behavior of the organization. More checks means less blame for mistakes; fewer checks means more blame. With the organizational cash stream assured, less blame means more promotions and greater compensation.
"You can engineer maintainability. You won't know how successful that effort was until after the maintenance is performed."
You can certainly try your best, but as you say, you only find out later how you did. But getting back to the main argument, programming "goodness" is a multidimensional quality that often can't be fully measured until some time has passed.
There is rarely any attempt to measure it in the real world beyond the misleading metric of LOC/time.
If 3x developers are common, then what is the baseline you are comparing them against ? The very worst programmers ?
So, since the baseline seems to be less than zero, and the best developers are three times that, are you trying to say that the most productive developers produce enormous piles of crap ?-)
But seriously, unless you can quantify productivity numerically in an objective fashion, saying someone is "three times" or "ten times" as productive as someone else is 0.961x as meaningless as "devolved systemic projection".
Yes. Specifically, real engineers and their companies get hauled to court if the product they designed blows up in the user's face. Software engineers don't; they and their company get paid more to help the users deal with the errors they made - this is known as "support".
Just a little bit cynical here ;(...
Forget magic. Any technology distinguishable from divine power is insufficiently advanced.
"Common" as in: "you've probably met one". When you can't measure something well, you can still measure it a little - a half order of magnitude isn't exactly overspecifying. However, when when you know you can give a team of 5 engineers N days, or your good programmer N days, and get the feature delivered on time, that's a pretty good basis for comparison!
There are plenty of software development jobs (even outside of life safety) where there's no room for serious errors. Sure, GUIs can be poorly designed and crash, and some CRUD programming has room for mistakes (or at least crashes), but any time you're handling customer data you don't really have any room for corrupting that - MySQL jokes aside.
Socialism: a lie told by totalitarians and believed by fools.
And I think the lesson is: The developer who cares is much, much more valuable than the developer that doesn't. So if your company is not inspiring your developers to care, you're going to hit a brick wall.
In my previous job, I worked for a huge manufacturing company. I developed, maintained and supported an add-on (plug-in, extension, module, whatever) for their 3D CAD/solid modeling application (Pro/Engineer).
When I first hired on, and started working with the application, I learned about the quality process we used: SPQA, or software process quality assurance. This was relatively straightforward, and, in my opinion, didn't add too much overhead. It was basically a checklist of things that a manager went over with the development group. Then we added AQA, application quality assurance. Not only did this add more overhead, but some of the requirements of the two processes conflicted!
The projects my group worked on were fun, cool, and cutting edge, and the majority of the staff was quite engaged in their work. So we worked through all this process nonsense, and basically got used to it. However, lurking in the background all the while was 6 Sigma. The company had adopted this methodology wholesale, and was forcing it to be used in every facet of operation. Now, not only did we have our previous processes, SPQA and AQA, but we had 6 Sigma to contend with.
Looking back, it's sad how much time and effort was wasted just trying to figure out how to integrate all these processes into our development. Literally, the whole section (about 12--15 people) spent many meetings and countless hours coming up with process maps, just to explain how our development would proceed, and comply with AQA, SPQA, 6 Sigma and anything else management felt like we needed to do. I used to eat lunch with the supervisor of another group---he had actually worked in the same group as me, years ago. I'd be moaning about how much of my effort was spent just trying to juggle all these processes, and he'd say, "Back in my day, we just thought up an idea and coded it." I'd talk about the growing queue of bugfixes and enhancements, and how none of them could ever be realized without eliminating all this process nonsense (or doubling the staff and budget) and he'd smile and say, "I remember when so-and-so called me up, asking if I could make the program do such-and-such, and I did it the same afternoon!" Sigh.
Maintainability damn well can be designed in from the beginning!
I agree, it most certainly can. Seriously, what is the point of a clean, well-engineered design, if you're not attempting to build a solid foundation for future work? Why not just crank it out any old way so long as the spec is met? Rhetorical question.
Learning how to code for maintainability up front is more a matter of experience (having been burned enough times by not having done it, and tried enough possible solutions until you've found those that work) that it is talent. It's also something that many programmers I've worked with over the years (almost three decades at last count) deem unnecessary, because they honestly don't know how. That, and the fact that thinking about maintainability requires actual forethought, as well as some extra bureaucratic/recordkeeping overhead. If you're not planning on being around for more than a few years in a given job, and don't give a flying fuck about the poor bastard down the line, then why waste a neuron on maintainability? That's how a lot of people look at it, I'm afraid.
I ran my own software business for, oh, fifteen years or so. I started out not concerning myself about maintainability because I was only worried about delivering the product on time. Then, after being in business for a few years and having to maintain my own code I changed my tune pretty quick. I mean, I got really tired of asking myself the same question over and over: "what the hell was I thinking?" And I mean that literally: when the code was written, what was I thinking? 'Cause it wasn't always obvious from the code itself, and I wasted a lot of time refiguring things out from scratch. Wasted even more time refactoring code because I hadn't bothered to think ahead, either. Now to be fair, I wasn't very experienced then and didn't really know what kinds of things for which to plan. That excuse doesn't wash anymore, I'm afraid.
Right now I'm responsible for several hundred thousand lines of code much of which is over a decade old. If you don't constantly work towards maintainability in an environment like that, you'll founder quickly. Matter of fact, several years ago I went through a major refactoring of that codebase (took a couple of months) just to make it into something that could be dealt with, long-term. Nobody had bothered until that time: the damn thing just kept growing like some amoeboid monstrosity. Reorganizing that project was a thunk of gruntwork that I don't want to ever have to do again, but I did it because not doing it was even more frustrating.
I used to talk to myself a lot more back then.
The higher the technology, the sharper that two-edged sword.