Slashdot Mirror
Is Process Killing the Software Industry?
blackbearnh writes "We all know by now that Test Driven Development is a best practice. And so is having 100% of your code reviewed. And 70% unit test coverage. And keeping your CCN complexity numbers below 20. And doing pre-sprint grooming of stories. And a hundred other industry 'best practices' that in isolation seem like a great idea. But at the end of the day, how much time does it leave for developers to be innovative and creative? A piece on O'Reilly Radar argues that excessive process in software development is sucking the life out of passionate developers, all in the name of making sure that 'good code' gets written. 'The underlying feedback loop making this progressively worse is that passionate programmers write great code, but process kills passion. Disaffected programmers write poor code, and poor code makes management add more process in an attempt to "make" their programmers write good code. That just makes morale worse, and so on.'"
If you want to go off and do your own thing, fine. Have at it.
But don't expect to write code that keeps a 777 safely in the air. That is the type of scenario that we need discipline, not creativity.
I'm all-too-aware of this issue and how quickly it sucks the life out of you and prevents anything from getting done, but at the same time obviously having no process doesn't lead to stellar code either. My question is, do any of you work in a place where you think you've struck the right balance? What are you doing?
Dislike the Electoral College? Lobby your state to join the National Popular Vote Interstate Compact.
70% unit coverage? CCN complexity 20? Pre-sprint grooming of stories? This article is apparently not meant for a general geek audience. Can someone translate these terms into something that non-professional programmers can comprehend?
Part of the hardcore faithful who believed in Apple long before it was cool again to do so
Yes. When you need to spend almost as much time learning how to and using the tools, processes, and configuration than actually producing code... then yes. And no it doesn't always help make better code. A lot of the time it takes time away from making code good.
-- I ignore anonymous replies to my comments and postings.
Get rid of management!
Karma be damned, this is relevant to TFA:
Come and work for my company, we have virtually no process whatsoever. We get given some vague sketches and told to go develop. We have no standup meetings, no pair programming (people on the same team never speak to each other and everyone develops to their own standards, you can get 3 different typoes of MVC framework in the same project), no coding standards, there isn't even a QA department (that's what the client is for innit), just code it up and crank it out. We even deploy to live servers when needed.
This is why I left development. After 5 years working at a software company I found that only 10% of my time or so was spent writing new code, which is the only thing I really liked about the job. The rest of the time was spent in meetings, wading through red tape, reviewing others' code, doing maintenance on the (junkpile) legacy system, doing remedial training for the front-line tech support staff, and the 5 million small tasks that have nothing to do with why I went into the career field.
On one hand, I can understand the creative process that goes into coding and the 'fun' you have in making it your thing.
However if you're likely to have millions of people depending on your code, which will alwso be modified by other people, then you had better have a good process as well.
I used to work at a company which required that pretty much all the important pieces of code have JUnit tests for them. Whenever someone else touches your code - which is bound to happen (I was modifying code which was written years ago - and the author wasn't employed anymore), it'll be a good thing if you know you haven't smashed anything.
So there's a time and a place for everything. If its very important code, yes please, strictness. If its something small and silly, then go creative on it.
I've worked on projects where they had procedures as thick as a phone book, and it was still possible to write crap code. In fact, due to the incompetence of the person who wrote them , they sometimes pushed you into writing bad code.
On the other hand, some programmers produce good code, whether they're specifically ordered to or not.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
That about sums it up. If they're so keen on doing it their way, then they can do it!
I think may be the guru-disciple system of education practiced (allegedly) in ancient India might bear better fruits. Guru lead teams with disciples learning from them might turn out better quality code, but the system would be expensive in the short run and takes a while to take root. The quarterly bottom line obsessed corporate world is as far away from the system of stoic ascetic guru living in a hut in a jungle accepting princes as students romanticized in Indian mythology.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Only about 10% of programmers are "artistes" who create new things of beauty...process stifles those.
80% of programmers come to work more-or-less on time, sit where they are assigned, work using the tools and processes given, and produce most of an organizations output. Process is beneficial to those, in that it makes their work useful in the overall "whole."
10% of programmers should be fired before they do any more damage. A better HR process would help with those.
The first version needs to written with passion. You then throw it away, and write the second version seriously. Also, code for your next php-driven social-thingy must be written with passion. Code for the satellite, not so much.
Religion is what happens when nature strikes and groupthink goes wrong.
"We all know by now that Test Driven Development is a best practice."
No, TDD is a painful waste of time that at best serves as a crutch for unskilled or insecure coders and at worst a smokescreen behind which serious bugs remain unfixed because they aren't picked up by any test cases.
The main issue is that measuring whether code is good or not is impossible. I promise you that I can write code that has the prescribed level of unit testing and complexity that also doesn't work. Software reliability/dependability was a problem in the 1960s. It's still a problem today. No silver bullet, and all that.
What really cripples things is when process is deemed a substitute for understanding the specifics of individual situations - where a one-size-fits-all-problems approach is adopted and imposed - usually by people who have no practical experience with the processes they espouse.
If software development could be successfully reduced to a process, I'd have automated it. Where there's a considerable burden of process, either the process is inappropriate - or developing the software itself is inappropriate as it amounts merely to re-inventing the wheel... an exercise in task creation that benefits no-one.
We should think of software development techniques and apply them judiciously - and the more techniques a developer masters, the wider their skill-set and the better they will adapt to new challenges. The critical question that needs to be asked is this: why is a technique being used and is it providing tangible benefits? If this question can't be adequately answered, everyone involved is wasting their time.
There is a balance between process and actually getting things done. Most companies never find it. The biggest problem I have is when this is used to audit progress on a task. For instance, I am a consultant for a company just starting to get out of startup mentality. There's been a push for more process and they've implemented their version of Agile. One of my first tasks was to write a script to dump data from Agile Zen so they could run reports on how fast developers are finishing stories.
Fail #1: All stores are not the same size! ...
Fail #2: Each team has different rules for stories and use agile zen differently. How do you compare them?
Fail #3: About half my time is spent in meetings, writing stories, fighting with QA to test things (not enough people). I'm supposed to do the same amount of work in half the time?
Fail #n: You're doing it wrong
The worst part is that when we had PRDs, at least there was a big picture loosely discussed. Now we get stories that don't integrate with each other and a big ugly mess.
MidnightBSD: The BSD for Everyone
How do we know that someone has written good code in the unit tests? The cycle goes on and on and in my opinion if you're not asking for help WHILE you are doing you work on an unknown system then you are deeply flawed as a programmer. You should be seeking advice before hand and during your coding, this is an even cheaper point to make changes than even a code review. Why are you having-a-go and then finding out you've done it wrong because you didn't ask the code owner how he thinks it should be done.
Passion isn't important. Cost and risk are important. The processes are put in place to (attempt) to minimize cost and risk associated with software development. Experience teaches us that cost and risk are very high when building software.
When it's your money paying for the development effort, feel free to structure it so that you can chase your passion.
I sympathize with the idea that this kind of bureaucracy can suck the life out of developers, but guys, this is work. If it were that fun, they wouldn't have to pay you to do it.
-- Truth goes out the door when rumor comes innuendo. -- Groucho Marx
As more and more DRM "solutions" are deployed, and the DRM systems themselves get larger, more complex, and more expensive, I think you'll find it not far from the truth that: "Process killing" IS the Software Industry.
And most software is not written by one individual.
As soon as you have an actual team writing software and as soon as there are others telling this team what they have to code, you need every bit of control you can get. There's no way around that and every anecdote "proving" you can get away with passion and good coders just proves that you can be lucky.
This is really the key insight of most Agile methodologies. Development should own the process and change it to suit their needs during regular retrospectives. The team (not the whining individual) should be able to say, "You know what, I think we're not getting bang for our buck out of this many unit tests, let's shift to 50% coverage." As long as that same team is taking ownership of the regression failures and making an informed trade-off their comfortable with, all is well.
If you get a good team together, you're going to get good code. You'll get better code if you empower them. Experienced and good teams will usually have a lot of these processes and tools in place because noticing things like high code complexity automagically alerts them to "bad smells" that can be examined and either accepted as necessary or invested in to address or test more thouroughly.
Generally, I think development is most fun when you're on a new project and don't give a damn about breaking things. Then it's pure creation. But once an app is older and there's some weird code you're staring at you have to decide, "is this probably a bug, or is this a bug fix for some weird situation or platform?" That's when you wish that the guy having fun three years ago had written some damn tests.
Yet another one-sided argument from a singular viewpoint which the author thinks would solve his or her problem in the real world. "Process makes me unhappy; find article, post Slashdot, ???, profit!" Ill-implemented process may cause undue headaches, but, unless I miss my guess, no work environment is perfect out the gate. Your willingness (and the willingness of the people around you) to adapt and change process to adapt to the varying people and personalities in your organization is a measure of quality in the workplace.
While I'm here, I may as well mention that coding is hard, debugging is hard, typing is hard, I wish my computer would just understand me the first time, I wish I could simply live in a tropical locale and "work" from the beach, and I wish I didn't have to really do anything "hard" to earn my salary. Now let me see if I can find some articles on those topics...
I’m so damn tired of people over analyzing the software development cycle. It goes against one of the basic tenants of computer science: Keep It Simple Stupid.
To write good software one does the following:
Write code
Test code
Fix code
Ship code
If you aren’t doing one or more of those things, then you’re a useless pile and you need another job in another industry.
Like in everything else, from economics, to love life (whatever it means for different people), there needs to be competition in software development. This means performance based compensation, this also means getting rid of those, who cannot achieve.
Developers should get goals in terms of business questions, then they should be let to do their own estimates and they should 'bid' on projects. There should be tracking of the success of different projects - from accuracy of estimate time and cost to number of features, to number of bugs per unit of time, to overall user satisfaction, to amount of time it takes to fix a bug or add a feature (this really would measure complexity and cleanliness of the code), to the average amount of time it takes the support team to deal with user issues (cleanness and usefulness of documentation.)
It is possible to do this, of-course large organizations rarely if ever achieve this level of 'enlightenment' even to start thinking about things this way, that's mostly because people do not understand economics, it's not about software.
You can't handle the truth.
Process isn't the problem, environment isn't the problem, language isn't the problem. The problem is that everyone looks to all of these things as the silver bullet that's going to "fix" software. Test driven development doesn't make good software any more than I-495 makes New York City. There is no silver bullet.
A large portion of code goes into devices that control extremely powerful technology that can easily kill a lot of people and/or do massive damage to other systems which can get very expensive. Testing and validation is the core of the engineering world and code is very much a part of that now (and becoming more so each day that more large systems are computer-controlled).
I recommend this book:
http://www.amazon.com/Inviting-Disaster-Lessons-Edge-Technology/dp/0066620821/ref=sr_1_1?ie=UTF8&qid=1305117862&sr=8-1
The sections about the Mars lander screw-ups (code-based) are very pertinent to what happens when the slightest piece of code does something wrong or unexpected. Bad code can easily result in a chain reaction that concludes in a major disaster.
In most engineering disciplines, the process of actually building something is long, hard, expensive, and persistent. If the project is build a bridge, you spend a lot of time making sure the design is right; why? because the process of actually building the bridge takes months or years, costs many millions of dollars, and once built is not easily replaced. There is no room for error, so process is taken very seriously as a central part of ensuring timely cost-effective correctness.
In software, the process of actually building something is instant, easy, free, and transient. Type "make all" and go get some coffee; find a bug? tweak a couple lines and do it again. This distorts the development process; "process" gets snubbed as a costly distracting annoyance instead of the means of assuring that what gets delivered is correct, because it's just so dang easy to fix and rebuild in seconds ... losing sight of the long-term cost of not doing it right the first time.
Can we get a "-1 Wrong" moderation option?
Processes are defined in the manufacturing industries to prevent slippage of 'achievable/defined' quality. Unfortunately the extension of processes from these industries (six sigma?) to software development doesnt reap the intended benefits.
Blame it on the guy who thought the role of a programmer and the line technician are the same!
TDD is coding not process. Grooming is requirements elaboration and design, essential work that engaged developers appreciate being part of (as opposed to being fed technical tasks as if they were automatons). CCN is just a measure of easy to maintain code and good design, not process. Doing some pairing on the hard parts solves reviews. While I agree that many orgs kill developers with process overhead, your examples are not the offensive parts, rather the essential parts.
i've seen supposedly passionate coding in practice. good people but it's almost impossible to work with them since they social outcasts.
a software release turns into a 4 hour nightmare because everything worked on their dev laptop but in the real world it doesn't work and they have to figure it out. but they don't give you all the information and you have to wait on the phone at night while they tinker and fix it in production. meanwhile employees can't take customer trouble tickets.
or the billing system can't make customer bills because it's now multi-threaded but there is no system to control which bills are processed when and the different threads lock each other out in the database because some selects lock down most of the table. yet the code is good and can't be changed
A wise elder at a defense contractor once told me that process was like cholesterol. You can't have none. But too much will kill you.
Except for ending slavery, the Nazis, communism, & securing American independence, war has never solved anything.
Creativity passion and, I add, freedom are dead. Not only in programming, have you seen a Hollywhood movie lately?
It's all about disaffection, process, workarounds and hacks to avoid the process, plus some people forcing some other people to make things every day more overkill, making them in turn more disaffected, unpassionate and happy to go home at 5:00 PM sharp.
Nonetheless which remains the most used OS in the world?
I'll give you a hint: its name begins with the contrary of "lose". And it's not exactly a "creative" product.
While all you process-laden geeks with your SCRUM teams and unit tests are getting the life sucked out of you in the Valley, we here in Automation Alley live by the seat of our pants, Montgomery-Scott-style, delivering barely-passable code on time, with the enhancements and 80/20 fixes to be pushed ahead of next year's model refresh, and a keen eye on the Bugzilla board, waiting for our user community to serve up the bug list. Thrilling, I tell ya.
Didn't yo mama ever tell you not to buy a first-year model?
Makin' money, makin' friends, makin' whoopee and wearin' Depends
Of course doing everything to an extreme isn't a cost effective way of working. If he has his way the other developers on his team would leave or kill themselves: "But, for example, maybe junior (or specialized) developers should be writing the unit tests, leaving the more seasoned developers free to concentrate on the actual implementation of the application.". Sounds a little arrogant to me.
Are you finding the majority of your code is actually made up of unit tests?
How can you be confident that your tested code is safe when delivered into the clients hands?
Introducing Meta-Test(tm) - with this breakthrough in software engineering methodology your drones can increase their coverage numbers without writing a single extra line of application code!!!!
Hurry! Don't miss your spot at our upcoming seminar where you will receive details on how to send your drones to our new training sessions that you can bill back to your clients and put your project even further behind schedule!!!
Andy Warhol got it right / Everybody gets the limelight
Andy Warhol got it wrong / Fifteen minutes is too long.
I've always been passionate about my code and attempt to test as much of it as possible, but I've found what saves on testing / process is discipline in coding style, commenting, consistent use of language idioms, good function / variable names, checking all function returns, lots of debug logging... Lots of little things that a programmer can do to help them read and diagnose their code during development and testing. It also helps to work with a "testing" partner (an element of Extreme Programming). I don't formally follow Extreme, but I do know from experience working in pairs or on occasion threes works really well: let the passionate go nuts with the support of a tester who should be equally passionate about that aspect of code (it is possible to find), and maybe the third is simply a mentor to overview and act as a sounding board for design ideas and problem solving. My partner and I will often discuss and compare ideas again and again; I'll code something, test it, then pass it to him to test it in depth. In many ways it becomes a game to write the code clean and for him to fault it. http://www.dailywav.com/0600/kaboom.wav
And i love programmers how manage to encapsulate good ideas in a way that they are not harmful.
If you want to see what happens when you keep layering more process on to try and fix perceived problems, just take a look at how any bureaucracy makes decisions. Government is well known for doing things that don't make any sense in the real world, and the reason why is that our (I work for a government) decision making isn't based on reality. Or even decision making, for that matter. It's all based on following a process and doing whatever comes out at the end. With enough process and a process-focused group of managers, the results don't even matter. Literally, this is the way people think: "Whatever happens has to be right because we followed the process, and if it goes wrong it's not our fault because we followed the process!"
There's always going to be some level of organization and control that is needed when several people are working on a common project, but you have to balance that with the need for people to actually think. When you pile up process on process on process you remove that and instead get a bunch of people who can't see what's actually going on because they're too busy complying with 14 different processes.
This particularly applies to software development. It's really important for a team to pick out what they specifically need for their project and what works for the team. Throwing in something like (for example) Test Driven Development just because it's currently a buzzword-compliant process won't get you anything except a lot of wasted time. Sadly, bad managers love to do exactly that because adding more process is itself a good CYA (Cover Your Ass) process.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
The real problem is that methodologies are followed, taught and practiced like religion instead of science. A new buzzword or buzzphrase and a new way of doing things will never substitute for thinking through whether what you're doing is actually working to attain the goal you've set out. Unit tests for example are brilliant when the developer gets control and is honest about using them. This almost never happens in practice. In practice unit test coverage becomes some bureaucratic check box to tick. What's worse is that there are dozens of industry expert consultancies wanting to sell you their own particular brand of excrement which may have some merit but the way it's sold - as a fix all for everything - will never be anything but a pile of poo.
These posts express my own personal views, not those of my employer
Depends on what the software is doing. If it's controlling my car or my mom's pacemaker I want the developers having a passion for making sure the code is 100% correct.
Too often "process" and tools are used as a substitute for competence. It's VERY easy to follow process and design patterns and use all the latest tools and end up with a brittle design & implementation. It's also easy to create a very similar design/implementation that's is robust and extensible and easy to maintain. And 90% of the people I've worked with over the last 10-12 years wouldn't be able to look at them and determine which is better (they'd most likely say the first is better if they were told the name of the methodology and design patterns used). After a year where the first fails repeatedly (crashes, can't meet new feature deadlines, etc) and the 2nd has been a huge success, they couldn't tell you why.
Excessive process may limit creativity. But lack of it may as well kill creativity in a moment. Process is like a contract. It doesn't only impose an obligation on a developer. It also protects developer.
You may get very tired very fast by rapidly changing requirements if you don't have sprints. In the middle of day your manager comes by you and says that you have to throw away 300 lines of code because your client want entirely different thing now. And if that happens three times a week by the end of month you just blankly stare at /. and youtube half a day because by that time you may know if new requirement are arriving or you can do that damn thing by the end of the day.
If you don't have any tests on a project and have more than one developer you may very fast get tired fixing same bugs over and over again.
I mean, all these best practices can be a good thing if they practiced wisely. Anything can do harm if used in excess. You can even poison yourself with pure water if you drink it a lot.
From the other hand some process can leave some time for devs to tackle their ideas in a branch. Say, implement some cool feature that was not in any user story. The prototype can be evaluated and scheduled for next sprint to get its user story, test, documentation and polish. Enthusiasm can not be planned. And most of the creative code I've seen was written without "test first" or deeply thought through user story. That's what 20% at Google (and all the similar programs) do. They codify creative fork. Also 20% is a process too.
How often does software crash? well, less today than in yester years... but... let's contrast this to the software written for NASA systems... IIRC, 1 crash in the last 10 years?... and you damn well bet they have process.
An article I'd read (can't find the link) indicated that almost every line of code is already in spec (as pseudo-code), and approved... if it's not in spec and approved, it doesn't get in... does that make the process boring? perhaps, but the result is unmatched by any other software.
I can not say more then, I agree... Processes are killing my mood to code, and that produced less code, and less quality of the code.
Test Driven Development is fine, so long as *you* (the coder) isn't the one writing and performing the tests. Testing against yourself is a useless waste of time.
Religion is what happens when nature strikes and groupthink goes wrong.
The creativity should come from Designers now. Developers just implement the design. It is the Designers that need the passion, Developers don't have that role anymore.
Its quite managers ideas to put any metrics in what you are doing, since software development is an industry already, not only some geeky stuff... That's not a bat practice actually... This will help establish estimations, count risks etc.
But sometimes gathering this statistics or assuring them to be gathered can be of significant overhead. For example to provide the required metrics I was forced to write junit tests for simple enums...
So I think this is a question of the correct approach to current situation. If it is 777 you really need it, but if it is a small webapp that's just waste of time an money...
Always talk to your manager.. or you client. Remember the rule: cost, speed, quality - choose two...
My problem with process has always been budget. Folks higher up budget on the basis of minimal process and expect full process. If you want 70% unit-test coverage, that is twice the time the code would have taken if you did not write unit test. Add 3 times the time if you want good integration tests to go with it.
Unfortunately, this makes a project costly. The problem occurs if the PM then demands full process when the time is not accordingly budgeted for it.
Release cycles also become long.
That's fine, but what will you do with all your spare time without an income?
I know that one particular development group at Apple found considerable gains in productivity from using unit tests consistently, where every time they fixed a bug, they wrote a test to confirm the fix. The upshot was that they spent far less time chasing down regressions than they did before they adopted the policy.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
If you hire only Rock Star level developers then you need very little process. Most process is about mediocre managers looking to minimize the impact from mediocre developers.
Seriously, there's a place for creativity in the development process but there has to be some level of validation. There have been many attempts at coming up with the best process including Agile however they all endeavor to separate the artistic part of coding from the brass tacks of getting the work done in an accountable fashion. More businesses today look at developers as "resources" rather than folks who have to literally create something out of thin air. With "Design Patterns" we had a set of templates now that architects could throw on a UML diagram and behold "that's what we use, don't divert from the path." With TDD we now had "code coverage" so we could think about what the testers should be doing rather than actually making the algorithms and business logic work. I remember, many years ago, sitting in a room with our VP of Software Development. He had just bought a new tool that analyzed code. It wasn't Coverty, it was something else. Anyway, we used to do peer reviews within the Architecture team and we didn't do it on check-ins either, we did it on random sections of code with developers who we thought could use the help. It was meant to be a learning experience, both ways. Well he plops down this tool which naturally looks at all the wrong shit "you only have 2% comment coverage, our standards say a minimum of 25%." To which I promptly retorted "who picked that number? Our standards are that we comment logic that isn't intuitive." To which the meeting then went on to focus why the developer didn't use exception based handling in the code vs. checking return codes after every system call.
That's the problem with a lot of these tools, you have some dipshit in authority setting the standards. Things like burndown rates are there to give the project managers something to do. Now, while I think that this article is a pile of shit, there is some truth to letting developers use their creative skills. Certainly, prototype. Create. Work till midnight on that new bubble sort algorithm and test. Of course you can write a better bubble sort, you know you can, and if you just did it there would be a 50% savings in CPU cycles for this splash screen in the first 1/2 second of elapsed time. Do all of this but also realize that there's deadlines, standards and testing that has to be done. Also realize that you won't be the only person who has to read that code and while you may think that it's perfect, in a year or so you'll think it's ugly because it hasn't been written in the latest language or you just learned that neat new template class mechanism and could eliminate 50% of what you wrote and just use templates! Sometimes you just have to leave it alone and call it done, which is the biggest problem in software development.
Have the bean counters gotten in the way? Yes, but remember they're the ones who have to show management that you're tracking on schedule and delivering something that will work; keeping you employed.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
You can use mediocre programmers if you have (and enforce) good development process and train them in best practices. Being able to build code that is easy for the programmer to read (and others to read), separation of sql/html,css,js/ etc into their respective parts of the MVC/ORM pattern, consistent inline documentation, consistent comments on code checkins, etc., following the same rules for writing code (development docs)
A good development process can alleviate many of the problems with having to do all that testing as you test because you have a bad development process wherein you are unable to easily have the developer look at his comments, his documentation and his clearly written code and go 'oh thats the issue'
Not saying this will ELIMINATE the need for testing but this reduces the need for at least 30-50% of it if everyone is properly trained in a good development process and it is enforced.
This is my sig. There are many like it but this one is mine.
Software professionals aren't deterred by software processes; some may B&M that writing test cases for a UI is a PITA and a waste of time, but a professional software engineer doesn't mind this extra work, because it's just part of the job. Architects, Doctors, and structural engineers have processes that they need to follow, and so should software engineers. Software engineering has only really been coming into its own in the last 30 years, so it's still a young engineering discipline, and the entire field is still struggling to come up with a guide-book. From the stakeholder's perspective: software is crushingly expensive. It takes many man-hours to develop quality software. Project management processes used by "metal-benders" who actually produce a tangible product didn't always map well to software development practices; but 30 years ago that's all anybody had. The last 15 years has seen a huge improvement in software design processes. Much of what was thought to be the last word on the subject (SEI/CMMI, and ISO-9000) turned out to be impractical, inappropriate and/or counterproductive in the real world. Newer mentalities like Agile, and XP are much better suited, but are usually snubbed by middle management tiers because the control gates require qualitative decisions not quantitative. Any idiot can sign off on a status report if X-many bugs per Y-lines of code is met. ---But when decisions need to be made solely on subjective opinion and experience, then suddenly nobody wants to sign off on anything; requests are denied, decisions are postponed, and the project suffers. That's not a problem with processes, that's a problem of ineffective leadership, and it's endemic in the software industry. --But that's the subject of an entire other essay...
Process is killing much more than the software industry. It is hacking its way through creativity everywhere. And personal initiative. And responsibility. When the process gets the responsibility, then you need corrective processes to cater for process failure. And "dynamic processes" to improve process improvement. Etc.
Although I am involved in several ITIL implementation projects, I have stopped believing in the CSI (Continual Service Improvement) and a few other processes along the way. Instead one should foster a culture that aims to improve.
It is unfortunate that Toyota's success has made processes the Universal Truth in business. It is not. There are a whole list of items much more important than processes in any company, such as
Is the business operating in the right market
Does the company deliver the right products or services?
Do they have the right personnel?'
Does the company have the right management?
I recently came home from Tanzania where I delivered a talk on ITIL to some 60 people that had never seen a PC or a white man before, and interestingly enough; Processes was not among their chief concerns in life. And I betcha that if we tried to enforce processes on that village, their spirit of play would not get nourishment.
Some food for thought:
http://isene.com/processability.pdf
http://isene.com/liquid.pdf
http://www.twitter.com/isene
an agreed upon way to communicate and it should be chosen by the team rather than imposed on them. People who think otherwise, especially people who think they can reduce software development to a process are really claiming that they can implement artificial intelligence using people pushing paper as their computing model. Any computer scientist will laugh at the idea, but it does not stop pure managers trying to do exactly that.
As the island of our knowledge grows, so does the shore of our ignorance.
So, in the article he does suggest some process is probably good. That's good. But then he suggests that "better" coders might need less. In practice, that is probably true, but his implication goes farther than "let's hold the hand of the junior guy a bit more."
So... how to architects, civil engineers, electrical engineers, graphic designers, technical writers, project managers, and, I dunno... just about every white (and blue, for that matter) collar employee stay "passionate" in the face of some level of process and bureaucracy? How would you like to go on a bridge (or a plane, as others have mentioned) built by folks who are so smart they don't need any process to validate their genius?
This sounds more like "I'm so good I don't need to do these dumb things that slow me down" and less "perhaps we should keep the process just at the point it is beneficial".
I just spent two years with a team of 6 refactoring a critical legacy application built and maintained by people who were "smarter" and "better" than code reviews, TDD, planning, documenting, or hell, breaking that 8K header file into header/class and maybe even into a couple of objects. And truth be told: they were ALL really, really smart.... and really, really willing to cut corners just to "get something done", resulting in an unmaintainable mess that cost us somewhere in the neighborhood of 1.8MM to clean up.
To all you guys who are WAAAAAY too smart to have processes: that's awesome. Please just go work somewhere else, preferably someplace I'll never work.
As someone that got started as a cowboy coder, I have to agree. Some process is useful - too much is deadly. The US government is a perfect example of how process can suck the life out of people. I was assigned to a task, and on day one we had to come up with a time-line. I figured what it should take and tripled it - 12 weeks for something that should have taken a month tops. Of course it was rejected - when we finished 13 weeks later, I pointed out that I was the closest to a guesstimate. They wanted to know what "I" was going to do to solve this - it was ideal timing since I handed in my resignation and told them good luck. That was my solution to a broken system that couldn't be fixed - too many chiefs and not enough indians.
Couple in licensing and you have a deadly one-two to innovation and development - that's why App writers are one or two men shops. If you want something done, the team for any given component shouldn't exceed two or three people - and you let them work out the interface with the others unless it's set in stone early.
I generally make my living jumping into a project that's behind on delivery and hopeless, the first thing I do is get rid of all of the meetings and every day I go around and talk to people to find out where we are and what needs to be done. Of course, once things are on track and working the bean-counters come in and screw it up again.
a lot of software is still developed in the same a way a master blacksmith might craft a piece of work
many companies think they can treat software development as a factory mass-producing generic products with ISO9000-alike paperwork
So, your code is a pile of s**t? It doesn't matter, you followed the company approved guidelines, created the necessary documentation and so on... Oh, yes... and you'll fix it in the QA phase ... well, may be.
Programming, motherfucker.
If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
Please, come work for me!
I always love it when the developers working in my group tell me to shove my directives up my ass!
Why are you letting these clowns ruin our country?
Bad code is worse for a company than developers who are non-creative.
I've worked in teams from 1 to 3 to 5 to 15 to 50 to 500 programmers. The fewer the number of programmers, the better they need to be to produce the same output quality. Why? When you have huge teams, you **need** process to get everything working. You need formal interface documents so classes and functions connect properly. you probably have dedicated QA people writing and running test cases. Those people LOVE to find bugs, since every bug they find proves the need for more QA people.
In smaller teams, you probably don't have QA or much time for others to review or even look at your code. YOU need to be better.
A single bad developer on a small team can cause so many bugs as to become a joke to the users. I had one of those on my team of 5. To him, closing bugs out of our bug tracking system was the goal, not actually verifying he didn't make 5 new bugs with every 1 closed. It became a real problem. Then we had the GUI developer who only used the mouse, never the keyboard. That sucked for me and our blind clients. The GUI underlined acceleration keys, but none of them worked.
Some of the best code that I've ever seen was in a small team of 5. In that team, even memory leaks were considered bugs. Our bug list was never more than single digits and each was fixed within 24 hours. It was a matter of pride in that team. We didn't have QA, we didn't do "test driven development", we just did our jobs. The program was pretty cool too. Adobe came to our lab, got a demo of our program and stole most of the features for Acrobat. BTW, we didn't have javascript in our tool, but we did have all the editing and authoring capabilities years before Adobe did. We showed them how to do search for PDFs and across multiple PDF files. Sadly, they invented a "library" for their searches.
Anyway, lack of innovative designs is much less important to coding than writing crap code. Good designs can come from Product Managers and others who don't have a clue about coding.
Bad processes imposed by incompetent non-developers, who do not understand what they are doing and why do.
Work for yourself.
The most productive teams I've worked with didn't take the entire process, they took what they needed to stay coordinated and discarded the rest. Actually several teams evolved a process that looked very much like agile long before the whole thing was defined. Short release cycles, short meetings to stay coordinated and small, focused teams. Most teams that adopt any form of testing at any point don't ever stick to it (Even though it'd really help.)
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
I've seen both sides of this coin, and I can sympathize with both viewpoints. The "Agile" nonsense is the absolute most buzzword-tastic pile of crap I've ever had to mess with. It's a bunch of people more focused on playing a word game than actually getting work done.
On the other hand, I was recently given the task of doing some housework on a PHP application that a former colleague of mine had written on his own (it was a system designed to track purchases at local pawn shops for review by law enforcement for stolen merchandise).
Lets just say I wasn't impressed. Single character variable names everywhere. Virtually no comments. Code that should have been put into a function was just cut and pasted each time an operation was done. Tons of stuff (like the actual list of organizations we were receiving data from) was HARD CODED into the code. The passwords for the users were stored in plain text in a database.
What's sad is that I wasn't allowed to actually correct most of this. Management explicitly dictated that I was only to correct a few outstanding issues - NOT really delve into the code. The reasoning was "how much work" that program is because based on the workload of that previous guy (who was maintaining this nearly full-time) put into it. They figured we'd need another full-time developer. In reality the function of the program was relatively simple - it's just that it was so fucking poorly written that any small change took a mountain of work.
Just IMHO, what I think companies need more than feel-good processes is a small set of very thoroughly enforced coding standards. Developers who don't follow them need to seek employment elsewhere. No amount of process is going to make a bad programmer good.
"People who think they know everything are very annoying to those of us who do."-Mark Twain
Why is that, that good code quality somehow is opposite of passion and creativity? I agree that pointless process is killing that, but good practice can be good.
I for myself like writing tests, like having easy code that have low complexity. That is why I can be creative in the first place. With my tests in place I can write my code and be sure I won't break anything and with low complexity I always have the overview over my code. Both are making sure that I can be creative.
The real question is why developers are not writing testable and low complexity code in the first place. Why is nobody teaching them how to write good code? Nobody is arguing that in the architecture or machine design, because if you not following good practice there the building will collapse or your machine will blow up.
Please define "passionate programmers" and "great code". If by that you mean people that just code away and leaving behind a mess of a code that only them can understand and maintain, then nobody wants that kind of programmers anyway.
True, there was a lot of stupid things in the past, like the Waterfall Model. But this is why the software industry is still a very young industry. Other industries have over 500 years of experience, like architecture or machine building. But the software industry is only about 50 or 60 years old.
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
No, but bad process is.
If you've got a small team of extremely competent programmers, is excessive process likely to do much more harm than good?
Sure.
If you've got a humungous team of barely-competent cut-and-paste code monkeys that their managers don't know how to evaluate, do you need astonishing amounts of process to get any work out of them?
Yeup.
Can you stop all business activity while you try to change from an organization of the second type to an organization of the first type?
Nope.
As someone with over 15 years of professional soft dev experience, my experiences have proven that unnecessary creativity destroys the reliability of software. Technical inaccuracies, misunderstood and misappropriated patterns and strategies, "tricks" and undisciplined programming are all the best ways to kill a piece of software before it even hits production. Engineering has plenty of room for creativity. Solving complex problems requires such creativity that it often translates to pure talent. But this work is based on a foundation of science and mathematics. Too many times have I seen developers do something tricky solely because they thought it was a nifty technique. These people scare the shit out of me and they permeate the industry. As web development has matured, I have seen the wheat separated from chaff. But for every 2-3 serious sw engineers, there's the 1-2 people that are obtuse, asinine and out of their depth. Most often these people have lingered for some reason or another, the Peter Principle usually. One thing that has proven itself to me: Only the fool suffers a fool. These fools are often the demise of well-intentioned projects. If you want to be "creative" in software development and ignore 40 years of research and science, go to fucking art school and please quit trying to convince the rest of the world to take you seriously.
I've been programming since the 80's and getting paid to do it since the mid 90's. It was the mid to late 90's when I got introduced to life cycle methodologies and at first I applauded them. I was even selected for a committee to help firm up our "best practices". I thought I would improving things by stopping bad practices by bad programmers. After a while it turned into a nightmare as those bad programmers started using the "best practice" rules as cover for the crap they produced. "But it passed unit testing and UAT, it must be awesome". Our project budgets went up and implementation time increased while our customers were visibly pissed at the final product.
The problem with SDLC process management has been sufficiently skewered by Joel Spolsky in a blog post several years ago. By putting a bureaucratic process in place you replace rock stars with compliance monkeys. Actually delivering a solid product is far less important than meeting checkpoints. And when your a programmer who's dedicated to a polished final product it's like murdering your soul to watch mid level managers (who know nothing of development) work hard at micro-managing you to ensure those check points are met on time. The actual outcome is irrelevant, the surgery was a success even though the patient died.
I have worked in both environments, the loosy goosy no rules and the highly rigid SDLC bureaucratic world. I understand the need for review and verification and the problems of wild west development. But development process methodologies are dangerous enablers of bad middle managers and "consultants" that sell you management processes that don't really help any body.
I'm an experienced developer, I don't need any of these new-fangled practices to make my code good.
I'm glad I don't have to maintain this guys crap.
Developers in general won't accept processes that amount to busy work. You can tell when you have this kind of process because your manager is constantly harping on it but eventually gives up. Good developers define the processes.
That's it. That's the reason why software projects under a rigorous process are far later delivered and more short on features.
We all know how many course corrections in the middle (or even the final) stages of a project are often needed to save the day.
Unfortunately, if this doesn't sound familiar to you, you have not been long enough in the Software Bussiness(TM).
Ok, so you wrote a gazilion unit tests for your huge java project. Every time you commit your changes to SVN, or worse, every time you run maven to compile, you're forced to run all these tests. And the do catch a lot of your errors. But it takes ages every time. You're drinking more coffey than you're coding. And what do you do about client-side testing of your javascript? Maybe you're modern and you're using selenium for that. All integrated into your build server.
Here's what I do. I don't spend all that time. And when something breaks, I use git bisect to find the patch that broke it. Then I do git log -u to get the diff of that patch, and since all my commits are fairly small, that's like, 20 lines of code, max, to figure out why it breaks.
Oh, and I'm using the network graph at github to do code review...
Every passionate developer I have ever know hates "process". At the same time, they all understand that at least some process is necessary. It all depends on two things: what kind of work you are doing, and what kind of management you have.
- What you are doing: (A) If you have a small team working on an internal application, you need a lot less process. (B) If you are a huge team working on safety critical code, you need a lot more process.
- Your management: do you have managers smart enough to realize that A != B, or do you have PHBs who want to fill in boxes on some checklist?
Essentially the "agile" manifesto: as much process as necessary, as little as possible. That said, speaking as a passionate type, I would never want to work on a huge project within a bureaucratic company, precisely because I want to do technical work, not paperwork. I imagine many other technically passionate people feel the same way...
Enjoy life! This is not a dress rehearsal.
If your car worked they way 99% of software does, you'd be furious. I'm done with paying for "creative" software. I just want software that actually works, and it's very hard to find. Most of the "creative" features on software I don't even want. They fail to follow to Law of Least Astonishment. What I want is my laptop, cell phone, GPS, etc. to not crash. Every feature I hate when I'm trying to reboot the device at driving 80 mph in a place I'm unfamiliar with. I long for the day when software engineering will actually be a form of engineering.
Let's explain the typical history of process.
A group of relatively skilled people come up with an idea oh how to improve software. Things like unit-testing, agile... They try it out with other skilled people and notice that the process does create an improvement.
This process becomes the latest and greatest craze and everyone in the software industry thinks it is going to result in better software... yet it doesn't.
The reason... you need skilled people to begin with.
I never turn against process. I just firmly think you need good people first before you start on this process binge.
For example, unit-testing is a great idea. I use it all the time. Yet, when half the people don't know what a 'unit' is, unit testing just flops. Unit-testing works great when you have your code well abstracted in clean units.
So for unit-testing to work... you need, abstracted, clean code. How many 'bad' developers write abstracted and clean code? You also need developers who can write proper tests and think of the cases to test for unit testing. Again, if you have a 'bad' developer who isn't checking for errors in actual code... why on earth do you think they're going to write useful unit-tests?
The same is true for every other process. Agile needs good managers, product people, developers, testers...
It's not even about time. Believe me, if all the code I worked on was clean, abstracted... written by good people... I'd have no problem writing unit tests for everything possible and following process 100%.
But typically, you have poorly written code with many mediocre/understaffed people and then some mandate to implement process which doesn't work well given the poor code and people... and just eats up time... and you get frustrated at the process.
In reality... the process isn't the problem perse. It is just exposing the poor code and mediocre/understaffed people.
N/T
Software Programming should be handled like all engineering disciplines. Imagine if all engineering firms started building bridges, apartment blocks or cars the way software developers write code. To put it simply, we'd all be dead. The next time you're restarting your computer, upgrading your drivers, installing a new firmware, just think to yourself, when was the last time something purely mechanical stopped working for no apparent reason.
I wish all software was written with the same engineering philosophy which NASA employs. They Write the Right Stuff. Sure we wouldn't have as much software as we do today and it wouldn't be able to do as much, but what it did do would work flawlessly and I wouldn't feel like throwing my keyboard through my monitor at least once a day while at work, only to arrive home to have to restart my Sky HD box because it's crashed or has incredible menu lag.
is not knowing when to use process and when to throw it away.
... if you read his article (the one linked in the story) you see he does not know anything about "agil processes" or Scrum (e.g. tracking hours for burn down charts, lol)
Read the comments on his web site to his "article" ... most are "insightfull".
angel'o'sphere
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
Years ago while working in a very large software house I asked the question, what would the environment and tools look like if we wanted to produce a highly creative environment, amazingly there few concrete responses that we could do anything with. I've always thought there is a blind-spot in the software industry when it comes to process. For example, if I ask a question like, what percentage of software has been written as part of your SDLC to address symptoms --- I get blank stares. Take some time to consider that, as the number is actually very high. Then I like to ask one from the flip side --- what percentage of software has been written that prepares for change so the team can respond to new business opportunities. This has nothing to do we anyones intelligence, this is the nature of how software is taught, learned and practiced by our community. Even the open source community with much freedom to develop their own policies can be every bit a restrictive as an enterprise. One last thought I have is that the SDLC and with its policies and processes is more an unbounded problem, whereas most software problems being solved are bounded problems and if we consider software as a literary work of art that is where one persons art is another persons trash
http://programming-motherfucker.com/
It's how software developers are compensated and managed. We're leaving the profession in droves, and soon there will be no one left to execute your process.
I have seen many passionate developers right crappy code. Code that cost money at the end of the day.
Code is engineering, use engineer practices.
The Kruger Dunning explains most post on
I've been managing a testing team for the last couple of years and we've been doing Agile. We had a core team of people attend an Agile Boot Camp for a week, and it was one of the best things I've done in the last 18 years in the industry. For the first year Agiles was great, then the "real world" of doing software in a very large company started to creep in. It is still functioning much better than Waterfall, but here are my thoughts:
1. The first rule of process improvement is that in order for a process to work, it has to be useful. Because we were trained on Agile, we made it useful. The trainers were able to quickly shut up those few project managers who thought they were already experts in Agile and wanted to "improve" it.
2. As a career tester, it was nice to hear testing portrayed as a positive thing instead of what always happens - people blame the existence of bugs on the testing team. It was really great to see developers, project managers, and product development people to think like testers. It was also great for me and my team to see into those worlds as well.
3. Communication! It was really great for the first year. We really collaborated with Product Development, Testing, and Development teams. We worked really well together. Everyone was focused on this project. Then people's attention started to get diverted and shared between the many things that come up outside our project. The general quality of things deteriorated.
4. We had to tweak things. Simply because of the environment we are in, and the project particulars where we aren't a stand-alone application and had to interface with other apps/groups who follow Waterfall, we had to modify how we do things. It hurt us, but was unavoidable. I think Agile would be awesome if you could do it in a bubble.
5. Transparency. This goes with communication, but initially everyone was open. If something didn't get done, people would just say that. There was no blame, let's just address why it didn't get done and move on. But I honestly think that developers have that "god complex" where at the same time they want to be told what is wanted (removing any creative control) and not letting anyone else know what is really going on (controlling the reins of things). I really liked not having to play any politics or that other BS that always happens - finger pointing. We're still doing OK, but it's crept in from time to time.
Agile as a process works - provided that it's applied to projects that can effectively use it. Agile isn't the right process for every project. And I think that is ultimately what "process" needs to be seen as, an effective means to an end. The process is just there in order to get things done, if it doesn't work then it needs to change. Too often, people don't want that (or they want it too much). People like to blame the process - and rightly so. For a process to work, it needs to be usable, and there is no panacea when it comes to process. You just have to find the right one for what you're trying to do.
My beliefs do not require that you agree with them.
Whoever wrote this crap didnt have to clean up someones elses spaghetti code. Much less worry about the actual productivity of their end product.
Software development is both an art and science. It is creative but exacting. The processes that are put in place around development are there to make sure the developers generate a bug free artifact that delivers on the requirements.
It is not process that removes creativity or productivity from developers. Metrics and processes can be used to level up developers from mediocre to good to great. Developers need to be able to see in the metrics and review what is “wrong” with their code and what they can do to improve it. This post is really about Management and Leadership not software development processes. Any process from just go do it to Agile Extreme Programming can be used to generate good code or bad.
The real culprit is a lack of understanding on both sides of the aisle. A certain type of developer think they are above the process. Their “creativity” is so important that they lose sight of the fact that they are being paid to deliver a product that meets a set of requirements and quality specs. On the other side of the coin are project managers, business analysts and corporate managers that don’t understand how software development works or even what a good set of requirements are. Their ignorance leads them to over process the development cycle. The development becomes about process instead of the process being about development.
When diva developers and poor management come together you can bet that the project will fail.
No sigs in BETA. Beta SUCKS.
Is shitty code from poor practice developers killing the businesses of the software industries customers? Is poorly tested, poorly documented spaghetti code just another way of saying massively expensive clusterfuck with a time delay? And does anyone except software developers give a fuck about creative and innovative software design if it happens to be the cost of actually being able to do business?
If you want to experiment with beautiful code - fund it yourself, or do it as part of an academic career. If you want someone else to pay you for your time, write boring code that works. Artists starve, engineers build stable products.
The quality of the code produced will be a direct reflection of the quality of the developers that produced the code. The problem is that many companies want to hire developers on the cheap (H1B's, off-shore teams etc) and then find some miracle of process or management whereby they can get high quality code from low quality developers. Unfortunately there is now magic wand of process management than can get high quality code from poor developers, the solution is the hire high quality developers who take pride in their work and who have a genuine concern about the quality of the product they produce. There is no management trickery that can circumvent the basic fact that code quality will always reflect the quality of the developers who produce it.
Notice how I used hyperbole there, in my subject line? That's because the title of this article is ridiculous. Nothing is going to kill the software industry. Why play Fox News here? The real issue is efficiency, not existence.
I do not respond to cowards. Especially anonymous ones.
You can afford more playfulness (perhaps I exaggerate) when you are accessing the system through layers that prevent disasters. Database triggers and the Permissive Action Links on nuclear weapons come to mind.
Simulators help greatly. The console prints out "Bang" instead of launching a real ICBM. Or run the new transaction software against test versions of the database rather than "Production".
Has anybody tried using an Agile Process to create formalized specs?
Process is fine, as long as it's always aimed at producing better software. When the goal becomes the process itself, that's when you have a problem.
If you are having lots of trouble with changes introducing bugs, it's time to extend test coverage, because that will help the goal of producing reliable software on time. If the goal is just "90% test coverage" without any relationship to how that improves the software product, you've got a process problem.
I've been in situations where everything is going fine, we are producing good code sprint to sprint, but the scrum master gets focused on getting a smooth burn down chart. The burn down chart is NOT the product. If the product is being produced successfully & on time, focusing on the burn down chart is a distraction that is hurting, not helping.
It sounds like you are more of a code cowboy rather than a software designer. You enjoy the challenge of designing/writing code for the joy of the play. This is great for prototyping software and implementation of throwaway tools, and you should switch jobs until you can find a place where you only do this. BUT this is very rare.
When it comes to actually having software that is going to have a life cycle of say 10+ years, with the original writers/developers no long around, you really do need process. It is the cost of your software causing the company servers to crash, or a game to break so horribly nobody buys it, or even worse 1,000,000 smart phones to lock up that is the reason managers want process and testing and design documents. THEY are paying your salary, so don't complain.
AHA I have the perfect job for you... quit your curren job and start writing some iphone Apps and see if anybody buys them... hopefully you can live on the income.
I see the vision of unit tests, but unless all the developers do, you can't just mandate a percentage of code coverage and expect to magically get better quality. My organization was at about 35% coverage when management freaked out and mandated 80% across the board. The developers were not all on board and we ended up with a bunch of fake unit tests that cover the code but make no meaningful assertions, not to mention three months of lost time. These tests now only serve to slow us down and make us less agile, yet they provide zero quality assurance. Furthermore, the 35% of good, solid, thorough tests we used to have has been lost amongst this sea of fake tests.
Another mistake management often makes it thinking that unit test code coverage will increase quality now. It doesn't. It only helps you maintain the quality that you already have, over time.
Do you think Mark Russinovich used development processes when he created the SysInternal suite of tools for Windows (perhaps the best set of developer tools ever made)? For great developers, process just gets in the way. For the other 95%, process is necessary.
"What creativity is needed to code a crystal clear requirement or specification? Sorry I don't get it."
I sure as hell wouldn't hire you then - you sound like a standard issue glue coder doing lego brick style programming. Algorithms and solutions to tricky problems don't just invent themselves, someone with creative flair and intelligence needs to dream them up. Thats obviously not you.
sure you can over do lot of these practices, but I find that I spend less time fixing bugs when I do them. There is certainly a balance and diminishing returns though.
Too much software is written without any thought as to how it fits within a process. Management tends to just throw technology at a problem, vs. first analyzing what they are trying to achieve and molding the software to that process to increase efficiency. Too often, software is chosen, and the business then molds their process around it, making things a pain in the ass for everybody except your "passionate" developers, who likely don't know a whole lot about the existing (or lack thereof) process. Retarded.
Stifle creativity?? Give me a break. Yes thats right, programmers creativity should be stifled. What programmers don't understand is that no one is interested in their dumb ideas. Don't be creative, code jockeys, just do what you're told. But of course thats not enough for the massive egos of most programmers. They need to have something more important to do than just writing code they've been handsomely paid for. The fact that you think programmers need to be "creative" says a lot about your misunderstanding of software development. I would like to spend more time on this but I actually have real work to do, developing software like a professional, following a process. Grow up.
Rock Star programmers are a myth.
The Kruger Dunning explains most post on
I have read some of the happy medium replies here, but I have seen much more harm in the development process come from either a lack of process or a lack of adherence to process. And I have not yet worked for a company that ordered everything off of the process menu.
Also, allegedly liberating processes such as Agile that ultimately have turned into jobs programs for engineers who fancy themselves middle managers are unhelpful. I recently left a company that had former highly-paid engineers wasted on a full time basis as "Scrum Masters" (nonsense roles fit for interns and Bob Schatz groupies) who burned through the time of actual contributors on a daily basis in order to have data to present to real managers. By the time I left, six people in my group who had been at senior positions as developers and SQAEs had jumped on the Scrum Master bandwagon. They spent their days holding lengthy scrums and scrum-of-scrums and then bothered developers so that they could acquire that obligatory hour-by-hour data to present to other non-contributors. It was terribly frustrating since the rest of us had work to complete.
Management wants shippable code, predictable results, and the ability to swap developers in and out of programming roles as needed. As the life force gets drained from drone A, they want to be able to replace it with drone B without downtime or a learning curve.
A strict process tends to help this.
"But don't expect to write code that keeps a 777 safely in the air."
I'm sorry, but thinking up code that keeps a 777 flying safely requires someone with creative thinking as well as good process. You won't always find the answers in he Dummies Guide to Writing Avionics Software.
A lot of so-called standard software development processes treat its workers like the machines they use. This sentiment has permeated not just programming, but all industry for the last two hundred years.
Yet attempting to automate programming misses the whole point of what it is; it's akin to trying to reduce second-order logic to first-order logic.
All these process-related failures we see in software development are misguided attempts to recreate a traditional computer with people.
Agile for the win!
Not to throw stones, but I think a major point that is missing here is that software developers are not the same as software engineers. Software developers do not need extensive process and procedures to function, software engineers do. Someone spends 80% of their day writing code is a software developer, while someone who spends 80% of their day writing requirements, verifying against requirements, capturing design, etc is a software engineer.
I'm neither, but have been labeled as both.
No, you cannot use mediocre programmers. Process is not a substitute for competance. In my view, much of the problem with process is that managers have a fixed budget. They then hire a non-coding software architect for 30% of the budget. Often, the initial software design does not survive first contact with reality. Then, they require process which consumes 50% of the budget leaving 20% of the budget to get the code implemented. The life is sucked out of the programmers since the budget is not generally increased with increased process consumption of the budget. If a programmer is being set up for failure, they should quit before the management can blame them for a failure not of their making. True Agile methods recommend only committing to getting done what you have already got done. The customer can cut off funding whenever the program is sufficient, progress is too slow, or they have spent more than they want.
To the question about killing the software industry: No way. Seriously, do you see software companies becoming any less prolific today than ever? I don't. It's incredible to me how much (quality) software is churned out on a daily basis, processes in place.
That said, the question about whether heavy process can kill ones creativity - yeah, absolutely. Some programmers become frustrated more quickly when there are barriers between their ideas and their code hitting the download sites. That's normal, and there isn't anything wrong with that. I'm one of those programmers, and that's why I always pursue my own side projects. That's our outlet, and once in a blue moon one of us makes something phenomenal, without the processes. Once in a while.
But to imply that such programmers are so great in number that it would kill the software industry is just not realistic. There are simply a huge number of creative developers who can work within the constraints given to them.
I don't mean that as an insult to Cowboys or Butchers. I'm a Cowboy and I'm not ashamed of it. I can bang stuff out. I can get things done. I welcome and recognize the other guys--the Butchers. They're the maintenance coders. They're the guys who don't mind writing design documents post-facto, knowing that nobody will read them. They don't mind meating with managers. Of course, if you want to disparage the Butchers you could say that Butchers, well... butcher. You didn't hear that from me though. In order to put meat on the table, you need Cowboys and you need Butchers.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
... the company I used to work for made us follow the same processes for COTS rollouts as developers had to follow for new code. There's nothing quite as pointless as loading MS Word into CVS.
They don't mind meating with managers.
Freudian slip: I think you are jonesing for a nice juicy burger, dude.
Have gnu, will travel.
We all know by now that Test Driven Development is a best practice.
I certainly don't know that, although I do know that some people think it is.
It's kind of the stereotypical mistake of process: a technique that makes a lot of sense in some cases and produces great results in those cases is generalized to a lot of situations in which it's either useless or counterproductive.
My team is agile. We have a burn down chart. We have a weekly iteration planning session. We pair program, we test first, we write our cucumber tests, javascript tests, junit tests, and we code 35 or more hours a week. We're also in one of the strictest change management environments I've ever worked in, and we're still churning out high quality, highly tested code in high volume. In our stand up every morning we talk about the stories that were completed the day before and we mark them on our burn down chart at that time. Our Kanban board helps drive us through the process and it keeps everything visible to all the team members. Documentation for our semi-monthly releases is a chore - it takes us about 8 man hours a month, but were working to automate a lot of it.
Agile means adapting, and it sounds like you're not.
Your sig(k) has been stolen. There is a puff of smoke!
Comment removed based on user account deletion
You are suffering the same confusion I'm trying to point out.
Software engineers (and their management) confuse the blueprint with the finished product. Your source code is not the final product, it is the complete detailed blueprint of how to build the final product - from which your compiler chain then builds the final executable (in a manner nigh unto instant and free).
This confusion leads to misapplication of processes, which in turn lead to silly assertions such as TFA. As build time approaches zero, so does the imperative to do correct design.
Can we get a "-1 Wrong" moderation option?
I'm an old school developer (1975, anyone?) who still views code as an art rather than a bean-countable process and has no interest whatever in management (job security through non-productivity.) The endless processes I have to deal with nowadays do suck most of the joy out, and despite all the typos I've seen caught by code reviews, I'm not convinced they make better software. What they do is:
- Make managers, who have never known what we do anyway, happy by giving them numbers to look at. Their ultimate (and hopefully futile) goal is to systematize development to the point that monkeys can do it. Then programmers will be screwed just like teachers and firefighters are now.
- Make old programmers who saw the way things were going and (unlike me) converted into process inventors while their colleagues were being outsourced, secure and/or tenured.
I'm behind the power curve (none of my startups made me rich) but I still enjoy writing code; my hope is that, by the time I get to stop pumping out crap for corporations (e.g., my dogwalking business takes off), that I still like writing code.
Where I work we got a new mangler in.
He was hot for us to use the SCUM process....we called it scrotum for obvious reasons...
Long story short, we did it for a week, bitched to the folks who pay us, the silly stuff ended.
The mangler did not last long...
My code is perfect.. its your test routine that is broken...
It's been my experience, at least in a bank, when you shove excessive process down everyone's throats, the developers are the least of your worries. We'll happily spend half the day doing the necessary paperwork to update an address on the web site, chasing down people for approvals, etc etc.
The real issue is once it takes 2 weeks to change a phone number on a fucking web site, because of excessive process, change control meetings, qa sign off etc etc etc. your business customers (the departments you write your software for) fire your IT organization out of frustration and go outside the company to a contractor that's not hampered by such processes, procedures, and bullshit. When you ask them to change a phone number, it gets done before the phone is hung up, just like we used to do it.
Then your entire IT development unit gets fired, largely because of some busybody manager that had some great ideas about shoving their stupidity up everyone's ass.
True story...
When I worked at this bank, there was one change control meeting a week. You had to first get the idea approved at a CC meeting. Then you had to go code it, get it signed off by QA, who had to schedule you at least a week out, because they were busy too, and present the qa results at a second meeting. So it quite literally took a minimum of 2 full weeks to change a fucking phone number on a web site. They tried to move me to a different state. I found another job and quit because I didn't want to move. 6 months later the entire team got laid off because the business units went to an outside contractor that could get stuff done in minutes instead of weeks.
Process is GREAT!
Don't kid yourself. It's the size of the regexp AND how you use it that counts.
... of succeeding as a QA functionary (With bonus metrics for more quality!):
1. Come up with a metric which can be tenuously tied to quality. Ten bonus points for metric being ill-defined.
2. Make developers (and your QA team) jump through hoops to measure the metric and make it go up/down (as appropriate). One bonus point for each percent in increased time used to calculate/track the metric.
3. Point to increase/decrease (as appropriate) in metric to show your "effectiveness". Five bonus points for the delta being statistically insignificant.
3.5 (optional) If metric does not change, speak out forcefully about how developers are "resistant to change" or "don't care about quality". Twenty-five bonus points for each employee sacked during this time.
4. Never measure customer satisfaction. Minus 200 bonus points for doing this.
Repeat as needed and celebrate your success in improving quality!
"Look at how much I'm improving things"(LAHMIIT) metrics are the bane of my existence.
That is all.
The problem IMHO is when processes get out of hand - exceeding the complexity of the original project. True, there are projects that require near-perfect code as e.g. people's lives might depend on the quality. But just as programs suffer from excessive feature-growth (of stuff hardly needed, see M$ Office products e.g.), the same way process definitions tend to grow over time. Plus the compexity of a program if also often not easy to judge. For experienced (or "genius") programmers, some problem might be trivial, and quickly coded, while management (not understanding the problem) might see it as a difficult problem, endangering the company, which then require complex planning and QC. All the while the programmer could have finished the project less than the time required to even get the plans in order.
This reminds me of a Dilbert comic, where the PHB would come up and require some plan on how to impllement something. Dilbert would reply, he'd already solved the problem. To which the PHB replied, he'd still have to make the plan ...
Personally, having programmed for something like the last 30 years (starting with ZX Spectrum and C64, via Amiga through Unix/Linux), I can pretty much code pretty decent code up to a certain complexity without doing much of planning ahead of time. Anyway, I do "fall back" to doing some more or less intricate planning at times, when I "feel" like the problem might have some tough points that could mess the whole thing up. Do date, I barely ever had major problems with this way of working. But then, I'm also just about the only person in the company doing any serious programming in that area ... so no team to keep in line ...
"fleshed out", please.
flushed out is what happens to the project just before it is completed.
Intron: the portion of DNA which expresses nothing useful.
http://www.stevemcconnell.com/ieeesoftware/eic10.htm
The author draws a distinction between "process-oriented" vs. "commitment-oriented" development. Either can work or fail. If you don't like what's going on you can talk with coworkers and/or managers, and either try to make it better or look for new employment.
I've worked at 3 (including fortune 500) shops in 8 years and all three before I started work claimed to be "Agile". Total bunk. Worse. Code. Ever.
I would love to work somewhere that has a team and not individual cowboys working in isolation 3 feet from eachother.
Process? where exactly? I would love to be hired somewhere that has process.
But judging from what I read on the net, I've just had bad luck finding places to work.
The processes are here because, frankly speaking, the vast majority of those employed as software developers are bad programmers. Creativity of the few good programmers that are left is collateral damage.
Most good programmers have since moved on, seeing as those that 'manage' them have short work days, earn few times more and get to spend time filling in excel sheets and inventing 'processes', 'intervention plans', 'initiatives' and the like. Folks that started programming when they were in the university, cheap labor from countries where individualism and creativity are discouraged, those types filled the ranks of corporate software development teams for the past decade. They are also, for most part, working for external customers, being treated as outsourced, expendable labor, usually not under too much pressure since their management is long accustomed to mediocricity and often needs plausible excuses and some external vendor to blame, not results. In such environment processes are needed to get anything done.
There are only a few companies that openly show lack of tolerance for incompetence and hire smart, talented programmers. Where I live a top notch programmer can earn about 120K USD a year, that means about 60K net. Any manager, mediocre product of business administration that will never invent anything - will earn 50% more easily, and work 60% less. So noone expects the programmers to be too smart anymore.
I'm a pointy hair manager who as a programmer used to be pretty good with the awards and recognitions to prove it. The big thing I've noticed since moving into management is the dirty little secrets of bugs. I've witnessed engineers who claim to be rockstars who don't need a process consistently generating two days of bug fixing for every one day of work. Before you can claim you don't need a process - be a engineer and scientist and prove your methodology works. If it does, god bless you and lets move on, if it doesn't lets have a talk and figure out how we can improve on this.
"There is no other way of guarding oneself against flattery than by letting men understand that they will not offend you by speaking the truth; but when everyone can tell you the truth, you lose their respect."
Niccolo Machiavelli, The Prince
Hey buddy, can i bum a karma? ~}CinderellaManson{~
It's impossible to cover every possible scenario in unit tests. Also, unit tests don't cover the system as a whole.
I have a mostly academic perspective of this debate.
Basically, the goal of process is to achieve high quality code at a low cost, or to put it another way the goal is to make development efficient. How this objective is achieved isn't really that important. Anyway, due to the massive complexity of a development team (member personality and interpersonal chemistry are complex affairs) and the plethora of different software objectives/metrics (nuclear plant safety software is measured mostly by lack of failures while social networking software is mostly measured in adoption rates), there's not really one right process.
However, just leaving it at that isn't very informative. It turns out there is one key element to process that everything good about process flows from...which is collecting data on what the developers are doing and taking the time to analyze that data to find areas where the process can be improved, leading to constant improvement in the efficiency of the process. Collect (Quantitative) Data -> Analyze Data -> Adjust Process -> Goto Collect Data.
Keep in mind that the data collection and improvement analysis are also part of the process. Collecting too much data (or the wrong kind of data) can be very detrimental as it takes time away from the developers doing real work to handle paperwork and constant navel gazing sucks up otherwise useful time. On the flip side of the coin, too little data collection will not allow you to construct an accurate picture of what's going on and good analysis of that data requires real time and effort...but this effort will pay off in the long run by making the real work more efficient for many years.
This process also takes some experimentation. "Let's try adding/dropping/modifying this practice and see what happens." "Oh, the data looks good after we implemented this, so let's continue to do it." "Hmm, this caused a slight decrease in performance due to time usage, I don't think it's worth it so lets drop it." It also takes some deeper thought. "Well, there was a slight negative impact in the short run, but this should have a positive long term impact, so lets hold out on judging it just yet." "This software is needed for a one-shot event, so we could drop all our long-term processes this one time to save effort." etc.
Existing processes and best practices are great places to look for ways to improve your process. "Code reviews have well known positive benefits across a wide range of situations, so we should incorporate that into our process." This really cuts down on the amount of experimentation needed to find a good process. However, following these suggestions blindly will lead to all sorts of trouble because as several other posters have mentioned it's a form of "cargo-cult engineering". One might adopt a practice that's good for high-quality high-cost systems when your goal is to make an acceptable-quality low-cost system quickly. One might adopt a practice that has a subtle negative effect, like driving away a good programmer or wasting small amounts of developer time for no benefit. If the boss is blind enough, they might even adopt a practice that completely destroys the team and project.
Constant analysis and review also helps reduce problems like valuable people leaving due to process. If they know the new practice they hate is only experimental and they'll be able to make a case against it at review time it makes it easier for them to tough it out for a while.
Once the process of continual improvement gets started, the team will constantly get better and better. Realistically there will be ups and downs, but the team will recover from the downs (such as a valuable person leaving), putting them at a nearly constant state of high efficiency...and the extra work put into maintaining and improving the process will be worth it.
Now of course there's more to good process than just striving to be in a state of constant improvement, but that would take far too long to talk about in depth. It includes things lik
Blaming the tools is sidestepping the issue.
It's appallingly poor management that causes these problems, not the toolset.
Yes, process intensive tool make it easier to hide mismanagement - but it's the
symptom, not the cause.
From my experience the optimum process just depends from company culture and team culture and how clear the requirements are.
The _only_ constant observation I made is that every project that did not test on every level, passionately, very early and continuously was doomed. If you don't define and prove quality you will not deliver quality. If you test you will define what you build. The more complete your tests are, and the higher levels they reach, the more you and your customer know where you are heading to.
"Just build your idea" may feel good at first, but your team will suffer and burn at the most inconvenient point in time.
DO TEST. A LOT. AND THEN MORE. AUTOMATE.
EMBRACE YOUR TESTERS. OR ELSE.
It's way easier to be creative if when can be sure that your project will not die a horrible death.
The code is really the nuts and bolts, concrete and other raw material that the people use to assemble the product; just like in a bridge building project. If I have already poured half the concrete and the piers are already standing, it's a little difficult to turn what was a suspension bridge into a truss style or move one of the ends
It doesn't end there. Where I see a lot of confusion, is assuming that an executable is the end product. It's an intermediate component in a much larger system. Example: Why am I building the bridge? To join two roads togother? To join two cities? To link two countries? It's always as part of something else. It all depends on scope, but it's never because I saw a river and said..."Hey I think I'll build a bridge". The same with most complex systems. The 777 code modules fit into some part of the final system. You can point at the avionics software in isolation and then plug it into the entertainment system.
We all have a personal quality quotient (says I). If your PQQ is low you'll do a crappy job regardless of process.
Your real process is what you do under pressure (not what your documents say). People with high personal quality quotients don't take short cuts just because of schedule pressure.
Thoughts on process from
Agile Project Leaders Network (APLN) Tim Lister
"Process is what you don't do naturally." If you did it naturally, you wouldn't need a process. You need a process when... "your instincts to do some thing naturally are not optimal."
Example - Swimming. Picture what someone that can't swim does when they fall in water.
Developers can create solutions without process.
Process only creates problems, like someone auditing your code when not needed. Or noone auditing your code when you do need it. Same goes for testing, unimportant (old) parts get rigorously tested, but the latest development may be ignored entirely.
Process is something you need in a kindergarden, not when dealing with grown ups who are supposed to be trustworthy.
How can you tell the difference between a good, experienced developer and a mediocre, inexperienced developer? The good developer already has "processes" (habits and self-discipline) that she has learned and honed through hard experience, knows they work, and resents when someone tries to force her to change for the sake of change or waste time on things that she has already tried and found wanting. Processes will only hamper good developers. The bad developer also resents "processes", but that's because they don't have any self-discipline. Processes probably won't help bad developers either. The trick is to hire good developers that get along; the only way to tell a good developer from a bad is to look at their code, which also requires . . . good developers. A good place to start, though, is to make sure they can actually program. Another good sign, though, is a good developer will occasionally approach you and suggest possibly trying a new "process" to see if it will make your shop more productive or reduce bugs.
Nathan's blog
Selling software != Selling consulting
IT Consulting companies business is to sell process to their clients.
Slashdot = Sarcasm
... that the floggings will continue until morale improves.
Ok, there's now a few hundred comments here and no one has linked to http://www.processimpact.com/articles/qualreqs.html or similar. Try the PDF version - http://www.processimpact.com/articles/qualreqs.pdf
Have a google around. There's a full document of which is the summary.
Read the Book of 5 Rings.
Read the Art of War (alternatively, listen to it..) ...
and start applying this to your job.
Requirements management is something everyone tries to avoid because it is 'too hard'.
---
On the darker side, I work with people to whom writing a short (2 to 3 pages) 'technical specification' is a 'waste of time' as the 'code documents itself' and 'it is obvious what the code does from reading it'.
Funny part about this comes when 'simple utilities' need to be updated.. and only the original coder knows how they are meant to function.
Yes, it reads from that file.
Yes, it writes to that file..
Yes, it does that processing..
But, what is the actually business requirement?
No idea, because that coder is gone. The system has changed.. and now we don't know if it should be doing x, y or z. What do we do now? No technical documentation, no business spec, and over 400 people use this (currently broken) utility on a daily basis.
Wait! I know! Let's do some requirements work and determine what it *should* be doing now.
No, we can't do that.
Too much work.
Let's just 'fix' the code.
Hang on. You have just made a code change.. slapped the code back into Prod... how do you know that your change 1) is what users need and 2) actually works?
So, they brought the code down again, and made another change. This went on for several weeks, annoyed users to no end to the point of end users not using the utility... and our reputation goes further down the drain.
All because no one wants to do requirements analysis, change impact analysis, business specifications, technical specifications or actually properly document their code.
That rage you feel when someone says "the code documents itself!". It means it is time to fix the processes, or leave.
Meanwhile, yes, we have a review process. Test plans are required. Testing is 'required'. Why don't these processes work? Well, it's called "rubber stamping".
If you can't beat them, join them or leave.
You have a sick, twisted mind. Please subscribe me to your newsletter.
Guru as team lead probably is NOT a good idea. Again, team lead is at least partially a social skill. I've known several guru class coders, and I wouldn't like to have any of them as bosses.
I can see merit in having a master/apprentice system too.
An experienced coder has a young apprentice. The apprentice does the scut work: Making the code pretty to whatever the flavour of the week is, reading the inline documentation, and adding to it as needed. (Or writing it from scratch...) Writing test suites. Assisting in the debug process.
Apprentices should be moved from journeyman to journeyman, say every 6 months. In this way they become familiar with various parts of the code base. They see a bunch of different work styles. They get practice at coming up to speed with other people's code. In addition they act as pollenators. "Mike showed me a neat trick with...."
After several years of this, they become journeymen. But now journeymen are paired to journeymen. The PAIR has to sign off on a milestone. For each segment, one codes, one documents/tests, both debug.
After a few years of these, if there is merit, they become masters, and are paired with an apprentice.
Some journeymen may not be master quality. At this point they are diverted into sales, and HR, and support....
Third Career: Tree Farmer Second Career: Computer Geek First Career: Teacher, Outdoor Instructor, Photographer.
Ha, and you thought this was new: Theory of alienation
I found it very difficult, as a permanent team member, to not throw my whole shoulder at whatever we were doing. I resented the people that got in the way of our success; I wanted to do nothing more than make the best fucking in the world.
When I watched the 2000-2001 tech meltdown, and saw lots of colleagues who were equally passionate and dedicated ejected like horses in a still sea, I got pretty jaded about it all. These people were wildly committed to their teams and projects, and the company they worked for (at the time three letters starts with S) had zero reciprocal commitment. The horses were drowning at sea; and the survivors were ordering $300 bottles of port at pointless team dinners.
The best lesson from that company was that reciprocal commitment is fair game - I have something a company wants, and they have something I want ($$$). The relationship ends there; they get what they pay for.
No lessor quality - often in fact better, because repeat business is sweet - but complete emotional detachment. That is the way it should be; its not a marriage, it is a hook-up. If businesses moved back to giving a rats ass about the people that make them a success; well that would be different. Try and find two.
The problem is not Process (either the lack of or too much of), the problem is People.
No Process + Bad People = bad results (see GIGO)
No Process + Good People = good results
Processes + Bad People = bad results
Processes + Good People = good results
See the pattern? Now, in theory, good Process + Good People should > No Process + Good People
I've tried implementing various processes and the problem is always the people. Some people get it, some don't. Some refuse to use it, some dont understand it. Some people's internal workings just arent compatible with the process. Experience is important.. more experience usually means more acceptance of process.
You have to match the process to the people or vice versa.
-- Senior Software Engineer, Attorney appearance services, locallawyerapp.com.