What Makes Software Development So Hard?

lizzyben writes to mention that CIO Insight is running a short piece that takes a look at why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises. From the article: "I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem."

Comments

good question

[macadamia_harold]

What Makes Software Development So Hard?

Why is conducting an orchestra so hard?

Re:good question

[jonnyelectronic]

It isn't. Haven't you seen them, all they do it wave their hands around and look silly. That's the point. Everyone thinks it's a breeze when it isn't, so everyone underestimates everything.

Re:good question

[xs650]

Why is herding cats so hard?

Re:good question

[Kumiorava]

I think you are close to the truth with this analogy. Conducting an orchestra isn't that hard when all musicians actually are professionals and thoroughly trained to work under your supervision. Big problem is that majority of the so called software industry participants are not skilled enough to actually perform the task they need to. I bet conducting an orchestra is difficult if the musicians play badly to begin with. Finding project full of skilled people is a major problem and there is no way you can see right away if someone is doing their job well.

On smaller projects my experience is that there are projects that I can count on going well, because I assign certain people to them. Other projects with double staffing does not do to job nearly as well, even when the task is half the size. This does not change (significantly) over time and is not related to education.

To answer the original question why software projects are so hard is that similarly to orchestra one bad player can ruin the show. Either that being the boss who has team meeting every hour, designer that doesn't know what is needed, customer that cannot give direction or software engineer that hides bugs in the code and causes several weeks of debugging for the team. Once the orchestra is filled with professionals I believe there is nothing inherently difficult with software engineering. Those projects happen, but it's rare.

Re:good question

[__aaclcg7560]

They're like programmers who can't agree on where to go out for lunch without doing a probability analysis of getting food poison.

Re:good question

[Ambush Commander]

Well, if you wanted to get really technical about it, the conductor's work is mostly done before the performance, so that any reasonably proficient orchestra would be able to do a reasonably good run sans the conductor.

Conducting, however, is a lot harder than it looks, especially during rehearsal. It requires the simultaneous processing of the score (possibly twenty plus lines going on simultaneously), time (not so easy once you get into funky metric changes), expression (precisely what the conductor is all about) while keeping track of mistakes that happened during the piece even though not stopping immediately. It's extremely easy to see when a conductor is inexperienced, boring, or hasn't looked over their music properly.

For conductors of student groups, they also have to keep the members of the ensemble engaged through tasteful storytelling while not completely going off tangent, they must be extremely creative in figuring out how to get their point across, they must be careful not to over-rehearse a section, etc etc etc.

Ask any marching band student turned drum major. Being a good musician does not mean you'll be a good conductor, and a more generalized notion is the core meaning of most of the other analogies offered by Slashdotters around here.

Re:good question

[larkost]

You are close, but the real piece you are missing is that most of a conductors work goes on long before you see them up on stage conducting. The real work is in music selection (you have to know your orchestra and what its strengths and weaknesses are), properly managing rehearsal schedules (what groups should meet, and what they should work on), choosing the right players and the right section leaders (the latter is both a delegation and a political art), and the actual concert conducting is much more show than anything else (unless things start to go wrong).

I know a player in the Philadelphia Orchestra and he tells me that they largely ignore the conducting that happens during a performance (granted that conductor's stick work is impossible to precisely read). They know how he wants the piece played because of what they did in rehearsal.

Oh... and I do have the experience to comment on this, I was in orchestras for 12 years.

Re:good question

[jdray]

The thing that orchestras have that most programmers don't is a program design (the score) that they're working with their peers to develop (play) from. The composer knew what he wanted. Most users don't.

Re:good question

[JoGlo]

Wow,sounds a lot like running a project!

Simultaneous management of multiple requirements (possibly 5,000 competing business requirements competing simultaneously); time (not so easy when the scope of the project is creaping, but the deadline isn't, or the estimating process was off, or the requirements have changed.....) while keeping track of 1,000 action items, issues, defects and delivery dates at the same time. It's extremely easy to see when a Project Manager is inexperienced, or undertrained, or bored, or being overworked into an early grave!

For project Managers there is the ongoing need to maintain team cohesion, and team spirit, and still get the code out the door - tested and working perfectly, on time and under budget.

Like musicians, being a good programmer doesn't mean that you'll make a good Project Manager. Some of the best programmers out there would be dead in the water if put in charge of a real project, for no fault of their own, other than being totally unequipped to run a project.

As for delivering on time and on budget, it is a well known truism that the first 90% of the job takes 90% of the budget, and unless you are REALLY lucky, it's likely that the other 10% of the job will also take 90% of the budget....but I digress.

Re:good question

[nigelo]

>As for delivering on time and on budget, it is a well known truism that the first 90% of the job takes 90% of the budget, and unless you are REALLY lucky, it's likely that the other 10% of the job will also take 90% of the budget....but I digress.

This is why you should do the hard parts first.

Re:good question

[JoGlo]

Agreed, but not always possible, as the hard parts often fall into view AFTER you've done a lot of the work

"Who'd have thought that little bitty piece there would have caused so much trouble?"

Or, as a cartoon I have on the wall of my cubicle says: "We decided to save the total screw up part for the end".

I also once had an entirely approrpiate wall mounted cartoon that showed a really complex diagram of how the application was to work, with a gap in the middle, and the PM explaining the gap as

"Here, we're expecting a great leap of faith to fill in the missing detail."

It never does, of course.

Has anyone here not read Yourdan's "Death March"? If you have any interests in projects and why they fail, then get hold of this book for a good read.

Re:good question

[nigelo]

It is certainly true that if you don't know what it is supposed to do, it'll be probably be very hard to figure out how to do it.

But don't overlook that it is human nature to say 'I don't really know how this works, but how hard can it be?' (to be an orchestra conductor...) 'No matter, let's do the parts we know, and get them behind us' thus nicely taking up most of the time allowed, leaving little or no time for the little-understood-but-hope-it's-easy 'then-a-miracle-occurs' piece that never got sized or included in the estimating/planning, etc. This (rightly) leads to the claim that 'the hard parts didn't fall into view until after most of the work was done'.

Again, do the hard parts first - if you don't know what it does or how it does it, consider it hard. If you go over budget on the early stuff, at least it was on the hard parts, and maybe your team can subsequently pull off the implementation of the easy parts in less time than you originally thought, especially now that they've learned how to do the hard stuff.

Either way, it'll be easier to deliver the latter parts under the pressure at the end of the project, particularly if you've overrun budget/schedule.

Re:good question

[xiphoris]

The thing that orchestras have that most programmers don't is a program design (the score) that they're working with their peers to develop (play) from. The composer knew what he wanted. Most users don't.

True, but a better analogy in this situation is the customers of the orchestra -- ie, the listeners. This might be a concert hall, or the person who commissioned the piece to be played (perhaps more often in older history).

The conductor is still accountable if the piece doesn't sound like what the customer (audience) expects.

Re:good question

[Matje]

The reason you attack the hard problems first is that you have less money invested if the hard problems turn out to be unsolvable (within your time/money/skill scope).

The big difference between a school test and a software project is that you only need to answer as many questions correctly as necessary to get a passing grade (you can thus skip the hardest questions), whereas in a software project you must solve the hard problems if you want to complete the project.

Re:good question

[swiftstream]

Not always, by any means--things often change at the last minute. Just an example: Yo-Yo Ma played with some students at Patrick Deval's inaugral gala last week (http://news.bostonherald.com/localPolitics/view.b g?articleid=174458). One song they played he finished arranging at 2:00PM the afternoon of the performance, faxed it over to where the students were waiting, and rehearsed with them at 3:30 PM. They performed it a few hours later. Oh, they knew what they were playing in a vague sense before then--they knew the song--but they didn't know the details of what notes they were expected to play (the arrangement). It's not so different from software development after all--in fact, that was the third arrangement of the same song they had been given. Changing requirements, anyone?

It's complicated

[fittekuk]

Software is complicated, and obviously the larger the project, the more peices need to fit together, etc. I think a lot of it though has to do with the team involved - both the developers and management. It seems to me, there are lots of "developers" around who really have no business writing software. And even more self-proclaimed architects who really have no clue.

Re:It's complicated

[Thangodin]

There are a number of problems:

1. Clients rarely know what they want. Most software projects are designed and written for someone else's requirements. These requirements often come in the form of "I don't like that, but I have no idea what I actually want."

2. Coders are systematizers, who tend towards the asocial end of the spectrum, which is why silicon valley produces so many autistic and Auspergers's kids. Large projects require communication, and there are a lot of coders out there who may be good with machines but are lousy with people.

3. As a result, lead programmers are often tantrum throwing prima-donnas, intellectual bullies who use their position to undercut those under them and make themselves look superior. You may have to promote and fire half a dozen programmers before you fill the lead programmer position with someone who is actually good at the job. In the meanwhile, you've just lost 5 good programmers. And if you actually manage to find a good one, you'll have to pay him in blood, because he's probably worth more than your house.

3. Salesmen, who deal with the client, are professional bullshitters, who are generally wafer thin on the technical details. That's actually not a slam--this is literally their job--to sing whatever lullaby the client wants to hear. Programmers will generally learn their lesson after being burned a couple of times and will not promise the impossible. The sales people never get burned--it was the programmers who screwed up, right? Calls for heroics on the part of programmers invariably begin with someone in marketing.

So, a few tips:

1. Get someone to hammer out the details of the specifications before signing the contract.

2. Get someone with some technical knowledge to study those specs, and the deadlines, before signing the product.

3. Get a lead programmer who fights for his team, not for the himself, and who is more interested in getting things done the right way than in getting them done his way.

4. Get a project manager (preferrably female--tends to calm the more socially inept coders, less testosterone poisoning) who handles communication between the coding team and the client. Do not, repeat DO NOT allow the coder to meet the client. They will eat the client! You will have a hard time getting paid, and hiding the bones is a bitch.

It's design not development

[gilesjuk]

Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

Re:It's design not development

[MindStalker]

Well no one really teaches software design do they.

You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge. Software design isn't really taught in this manor, sure your taught how all the bridge building tools work, and even a lot of the engineering specifications. But I have yet to see the software design school that covered more than a class or two into truly how to design software. Then again, we've been building bridges for thousands of years, and designing software systems for a few decades. It does take time for these things to really get figured out.

Re:It's design not development

[kevin_conaway]

Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

The problem is:

• There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change
• The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

Re:It's design not development

[truthsearch]

The last project I was on which had requirements that didn't change at all during development was 7 years ago. And it was built right on schedule. For me it's been the ever-changing requirements which delay projects the most. The other half of the problem is usually bad estimates, but that's almost irrelevant after the requirements change.

Re:It's design not development

[nuzak]

We've had a couple of thousand years of practice building bridges, and bridges solve a known problem: connect points A and B when there's a gap or a river inbetween.

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

Re:It's design not development

[MeanMF]

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

That was yesterday....We'll be sending you the updated specs soon.

Re:It's design not development

[xero314]

I think you are getting at the point but in a bit of a backward way. The problem is that somewhere along the line the idea of Software Engineering was thrown out the window and replaced with ( or reverted too, depending on how you look at it) software development. If you apply the principles of Engineering to software you end up with far more stable and durable products. If we built buildings like we build software there would be millions of people dying from "unforeseen bugs" or rooms that could not be accessed because the "interface" was not clearly defined (of course you would go back and put the door in by severing an important support beam causing massive loss of structural integrity, but I think you get the point.)

Excuses like;

There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change

and;

The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

are the exact reason software sucks. Not because these things are true, because they certainly are not, but because we are not applying logical engineering principles to the problem.

Re:It's design not development

[HappySqurriel]

That and software is more like art than engineering, but no one wants to admit that.
Of all the Renaissance painters that ever painted, only a few became forever famous.

Same thing with coders.

I personally would use the term "Craft" rather than art because I have always believed that art is about "saying" something through your craft which I don't think is really the goal of software development. The truth is that anyone can be taught to be a reasonably good muscian, artist, or blacksmith and with proper motivation anyone can become great. I would say that there are two problems currently, schools focus on the wrong things (how often did you gather requirements, properly design, implement to given specifications, effectively test, or maintain a program in your school years?) and the industry seems to focus more on technology rather than skillset (does it really make that big of a difference to program in C# as compared to Java?); the result is you have someone who is trained to use a hammer designing your house, and you ask someone if they know how to drive a "Ford" when you are interviewing a truck driver.

Re:It's design not development

[Anonymous Coward]

I would have laughed, but this hit a bit too close to work.

Re:It's design not development

[CastrTroy]

But people aren't willing to wait or pay for engineered software. Windows already costs $199, and a real OS that followed engineering principles wouldn't cost $199. It would cost probably around $10,000 per copy, if it were being sold on every desktop. And it wouldn't look as nice, and wouldn't do all the whiz-bang things that windows does. You can get engineered software, just ask NASA, but don't expect that grandma will be running engineered software on her $299 Dell box any time soon.

Re:It's design not development

[RingDev]

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

Close, but not quite. software engineering is "I want it to do blassie. Make it do blassie by yesterday." Where 'blassie' is a loose concept and the developer has to determine and define the underlying logic.

-Rick

Re:It's design not development

[StikyPad]

For the love of God! If you want us to troubleshoot your code, you're going to have to use some meaningful variable names and stop putting multiple operations on a single line!

Re:It's design not development

[0xdeadbeef]

Yeah, you're right. Kind of like how writing a novel is easy. It's creating an engaging plot with realistic and interesting characters, that's where it gets hard.

This non-insight brought to you by the discredited waterfall model, the ideology driving the outsourcing fad, and incompetent god-architects with no real programming experience.

Re:It's design not development

[Richard Steiner]

That and software is more like art than engineering, but no one wants to admit that.

I suspect a lot of programmers are fully aware of that (though I would agree with another respondant that "craft" might be somewhat more accurate), but that a large number of managers (mainly those who don't have a software development background themselves) don't fully understand the truth behind the statement.

Software development is still a subjective process in many respects, and I'm not completely convinced that it's a bad thing. Sometimes the effective expression of a radical idea requires a solo programmer who just happens to have a brilliant insight or even just a vision of something.

Writing software isn't like building a bridge -- there are often a fair number of incompatible but equally elegant solutions to a give problem, and sometimes the elegance is readability and supportability rather than cute tricks in the code (which might legitmately save clocks or bytes but make also make life miserable for some unsuspecting support guy five years down the road).

Re:It's design not development

[noidentity]

Fortunately, they do still teach English (at least in some parts).

Re:It's design not development

[I Like Pudding]

That was yesterday....We'll be sending you the updated specs soon.

There are lies, damned lies, and software specs.

Re:It's design not development

[nuzak]

> you're making it sound easy and simple

The execution isn't simple, but the problem is pretty well defined. You don't generally get people asking for exit and entry lanes in the middle of a bridge, or asking it to accomodate aeroplanes, or asking you to build them out of styrofoam just because some magazine said styrofoam was the hottest new material. Or deciding that each piece of the bridge should have a special connector that requires new tools, or finding that your welders suddenly don't work when you choose a different brand of paint.

Software engineering generally sucks because it's surrounded by suckage all around. Programmers are trained and treated as wiz-kid hackers who can do everything ad hoc, projects and tools are designed that way, and it's no surprise that the products come out that way too.

When it's all said and done, it's probably harder to build a good large bridge than a good payroll program. The thing is, most bridges are eventually finished someday.

Re:It's design not development

[smellsofbikes]

>we've been building bridges for thousands of years

And despite that, they still fall down.

Read Henry Petrowski's book, "To Engineer Is Human" some time. The point of the book is: an engineer, whether bridge or software, is building a compromise between deadlines, money, user requirements, and laws of nature. As time marches on, that compromise will *always* tilt towards deadlines and money, and eventually it will break, at which point we've learned something for next time.

Re:It's design not development

[radtea]

But people aren't willing to wait or pay for engineered software.

I think you've mistaken "overdesigned software" for "engineered software". In my experience engineered software is cheaper than the usual mess, because there are fewer bugs and they are found earlier in the development process. Modularity is strictly enforced so the scope of bugs is smaller and unit testing is easier.

It is certainly possible to expend huge amounts on designing a thing to death, but given the howling screw-ups that NASA has produced over the years (was that metric or Imperial?) I wouldn't hold them up as anything like an exemplar of good software engineering practice.

Remember, engineering of all kinds is a matter of finding adequate approximations and compromises to meet specified performance criteria. It is not and never has been about making everything "perfect" by some impossible standard. Perfectionism is expensive. Engineering is cheap.

The real problem with software engineering is that the vast majority of people writing software today simply do not have the intellectual discipline to undertake it.

What makes software development so hard......

[Chineseyes]

Viagara, Cialis, red bull, two brazilian hookers and a swiss midget.

One word: Skill, Talent and Knowledge

[rudy_wayne]

Anyone can write a book. But, writing a book that's *GOOD* and that people will want to read is an entirely different matter.
Computer software is no different.

People expect too much too soon.

[Dex5791]

Good software takes time to develop. There is sometimes a tendancy to set unrealistic goals and when they aren't met the people in charge feel let down.

What large projects DON'T have problems?

[panaceaa]

In related news, humans still can't seem to bridges with any reliable schedule or budget. Despite the fact that bridges have been built probably since the dawn of man, and we've been building suspension bridges for at least 500 years.

Two things make software "hard"

[mandelbr0t]

First, the people who make the technology decisions often don't have the required technical know-how, and have a terrible tendency not to listen to people who do. OTOH, the people who have the technical know-how have absolutely no idea how to write a business case. Thus, there's usually a disconnect between the people who understand the business requirements, and the people who understand the technical requirements. Vendor loyalty has been known to be a sticky issue as a result.

Second, there's always a problem getting a bunch of talented, egotistical (ok, so not all software developers have ego problems...) quirky, eccentric and generally difficult people to work toward a common goal. The common analogy to being a successful director/manager of a software project is to that of "herding cats". My experience has been that business types don't react well to the often-emotional developer types, hearing the emotional outburst, but ignoring the content of it. Developers would do well to learn some more social skills, and director/manager types would do well to listen better.

mandelbr0t

Re:Two things make software "hard"

[k12linux]

In a prior job I was asked (in front of everyone in the boardroom) to estimate how long it would take to do a project which had just come up and only been discussed for 15 minutes or so. When I said that I would need some time to spec it out I was told that would not be possible.

After thinking about it for a minute or two I decided that 90 days would be reasonable based on what I knew of the specs at the time and my current work load but that I could manage 60 if I could put this one first on the list of priorities.

I was then told that 60 was too long, how about two weeks? I finally compromised at 30 days with overtime and absolutely no other work on anything else in the company.

I finished the programming on the evening of the 30th day, went home and wrote up some instructions I could use for training the people who needed to use it. First thing the next morning I was taken to a conference room by my boss and literally *yelled* at because nobody was using the program yet. (Thisw as the 31st day.)

I explained that the program was done but I had to train the users that morning so they could be operational in the afternoon. (Only about 10 people needed to use it right away.) This was dismissed with a *shout* of "Nobody is using it, therefore it is not finished!" He considered the program a failure.

Do you see anywhere that management could have been at fault for the project "failing" here? It's all too common to have management fall into the Capt. Kirk role. (Scotty: "I have the engines at 100%", Kirk: "Damnit Scotty, I need more power!") Maybe it's the techies who do screw around that make them think the rest of us just need to be pushed harder to get better performance out of us. I don't know but I've seen it more than you might think.

BTW, when I say "yelling" I mean it. Face red, blood pressure skyrocketing, the works. BTW #2, the program worked just fine starting late that afternoon after staff were shown where it was and how to use it. BTW #3, that was the day I decided to find a new job and stop blowing off offers from headhunters. (I was somewhere else with a 40% pay raise three months later.)

Theory is great

[hsmith]

practical application isn't so. Requirements change, clients see something and want something different. You work a month on something and find out it isn't working anywhere close to the way it should. Tasks are more complex then people thought at first, the list goes on. It is a complex thing with infinite ways to do things. It isn't any easy process.

PHB?

[Jumper99]

I find most of the issues that arise on projects are not so much that the developers are incompetent, but that no one sets realistic, or are not allowed to set realistic expectations. How many times does the business change the specs? How many times are the developers given more time to allow for the changes?

The last project I worked on had a hard deadline. We were in the UAT phase when the business decided it wanted a change. A big one. Despite the fact that we were mere days from production, we were expected to make the changes, test them, and still meet the schedule. IT managements take was as usual. The business pays the bills so the business gets what it wants.

Until IT management can stand up to the business side, software development will always suffer.

Hard is better than impossible

[composer777]

The reason it's hard is because we are doing the impossible, at least if you had asked someone 50, 20, or in some cases even 10 years ago. Solving previously impossible problems requires quite a bit of innovation and novel thinking. But, it's better than it was decades ago, even if it is difficult. What we need to remind people is that even with fast computers, the problems are still difficult to solve.

Also, automating tasks that we used to perform ourselves is also difficult. It's one thing to walk and chew gum, but another thing entirely to precisely describe the mechanics of doing so. (One is very easy, the other requires creating precise models of how things work, a very hard problem).

Ideas come faster than code

[nharmon]

What Makes Software Development So Hard?

Simple. The current state of technology in regards to human imagination makes it many times easier to think up of good ideas or improvements for computer software than it takes to create or implement them. Thus the requirements for developers are often made by the ones with the ideas with no idea of the costs (in terms of time or money) involved in making those ideas a reality.

Joe Sacks says "Gee, it would be a great idea of our flight tracking program also tracked fuel usage and recommended fuel loads to minimize weight on subsequent flights". See? It only took Joe all of about three minutes to think that up. He figures that a reasonable amount of time to implement this idea is one or two days. After all, those "computer guys" are pretty smart and can "do anything".

Two days later Joe receives a report that the "project" is over budget (needed to check with the FAA on this one, that cost money) and overdue.

Sure, you could just say that these problems need good project management. But seriously the problem is that these "problems" are not considered worthy of project management. It only took Joe three minutes to think the idea up...why should he hire a project manager for *THAT*. It's not like he's building a new hangar.

One wag of my finger, one tip of my hat

[iPaul]

First the wag: The idea that the interveiwee states early on, that software development is not introspective, is horse-hockey. We think about it all the time. We invent new, clever ways to diagram software, capture requirements, interview users, validate functionality and come with all sorts of certifications. More than, say accounting, we are process focused people. Maybe our processes suck, but we spend a lot of time, energy, certification exams, etc, on those processes.

Tip of my hat: He does mention starting small and iterating. I think that's the best way to build software.

Re:It's not hard

[daVinci1980]

Spoken like someone who hasn't ever been involved in the development of a large software project.

Large projects bring problems with them that aren't noticeable on small projects. The working set of my project is around 10 gigs, most of which is code and text files. The tree changes quite frequently, and syncing to that tree is painful.

What makes it more painful is when the tree is broken. So we had to develop tools to help ensure that the tree isn't broken, and that we have a way to tell what the last known good submission was.

There's performance issues related to the source repository, because no matter what repository you're using, they all have issues when you have 200 people working in the same place at the same time. (This is true of virtually any database application).

For Me

[KermodeBear]

What makes software development so difficult for me in my particular case:

1. Requirements are not clearly defined
2. Requirements that are defined change constantly
3. No existing documentation on the system I work with
4. Outdated technology (Not a big issue, but many things are just easier to do with newer software)
5. The sales department sells features that do not exist, promise a date, and do not consult IT at all about the feasibility or time estimates

4 out of the 5 things I have listed above have to do with bad information / lack of communication.

Complexity & Culture

[denoir]

There are two major reasons why software engineering is today not comparable to more traditional types of engineering.

The first reason is the complexity of the systems. There are essentially extremely many interacting parts that cause all sorts of emergent phenomena. This is a field that we don't really know how to handle very well - good quantitative models of complex systems simply don't exist. We need more study of systems theory and chaos theory to build some form of predictive models on which in turn can base planning.

The second problem is a sort of 'freedom of thought' culture that sees coding as the vaguely mathematical expression of ideas. As there are a huge number of degrees of freedom most attempts to regulate it in practice have been abandoned. So instead of just the problem of describing a mathematical problem, you throw individual human preferences, thinking and biases into the equation.

Of course it can't continue that way forever and we need to move to a meta level of coding. We do have well-constructed systems, such as the software on-board an airplane - but they are laughably primitive (because you have to account for all possible states the software can get in to). And we have advanced software, but it is laughably unreliable.

Still, there is reason to be hopeful. We're not inventing the wheel every time any more (just the horse carriage and so on). Modern programming systems have extensive class libraries that are on average a more stable foundation than making a custom implementation from scratch every time.

Ultimately however we need to know how to handle complex systems and how to enforce convergence/stability to systems where the huge number of possible combination of states can be analyzed. And sad as it is, if we want it to be reliable, the liberal individual approach to coding has to be abandoned in favour of a much more strict industrial way of thinking. Right now we have handcraft and we need industrial precision, standardization and quality.

If it was easy

[dantal]

I would be out of a job ;)

It's simple

[ozzee]

When you ask a carpenter "are you done ?", almost anyone can verify the answer just by looking at it.

When you ask a programmer "are you done ?", it is difficult to verify the answer.

There are so many ways to accomplish a goal in software that you need a number of very expensive controls in place to validate the result. Very few managers know how to do it and very few C*O's are willing to spend the money to put in place even the minimal control measures even though I can say in every case where ther were good controls put in place the benefits far outweighed the costs.

This leads to a plethora of problems that I won't repeat here but I know too well.

Eventually everything breaks down when the C*O's start making technical decisions that they are not qualified to make and it's all downhill from there.

Mod parent up!

[EmbeddedJanitor]

While the comment might just sound like a silly grab for a funny, I think this is a very valid point. We really have far too many developers who were not intended by the Intelligent Designer to be so. Software quality and robustness etc (which impacts on delivery schedules etc) are dictated mainly by the lowest performer in the group rather than the highest performer.

There's a whole industry of dev tools out there trying to convince management that SilverBullet will make your software come out on time, or make your crappy programmers more productive. I don't think so.

What we really need is to promote the concept of the CodeSmith like a balcksmish, silversmith or whatever, coding is a skilled artisan occupation. Instead of trying to keep good coders coding, most organisations try promoting them and making them managers. Eventually you're left with just the dregs coding.

Re:Mod parent up!

[Beryllium Sphere(tm)]

>Instead of trying to keep good coders coding, most organisations try promoting them and making them managers.

Whereupon it turns out that they have exactly the wrong skill set to be managers and things get even worse.

I once saw a deep-thinking and productive coder promoted to management. The result was a catastrophe of Biblical proportions. In terms of morale alone, when employees got together to compare notes they found that they'd independently been thinking of where to take cover if the guy went postal.

Re:Mod parent up!

[FecesFlingingRhesus]

Actually,
Though the comment was party in jest, though I do believe that it is a prevailing issue. I believe there are far too many developers in the industry that are not competent in their ability to write quality software. This is evident in many developers adhering to patterns for the sake of using pattern when the patterns do not fit their needed problem set. Web programming is saturated with this kind of mentality; this is evident in new frameworks coming out weekly that rehash the same old problems and poorly I might add. Your comment about development being an artisan type endeavor might be more true than anyone wants to admit. The business side of the industry is scared of the idea that software may not be able to be manufactured. Crafting conjures the realization that each peace is unique and unlike an assembly line, carries unpredictability and risk as it cannot be quantified until it is completed. This takes away businesses ability to look like they are in control and masters of the software domain and makes people who otherwise feel as thought they are all controlling realize that in fact they have no control nor can they even begin to comprehend what needs to be done to control software development. Yet everyone goes on quietly knowing this and not saying anything because no one wants to look like a jackass. So the business goes on acting as though they are in control while not treating the process as a creative one because no one wants to admit that software cannot be manufactured. The sooner companies realize this, the better off the industry as a whole will be. Developers will be looked at as skilled contributors and not an cog that are replicable by the next guy with the same skills on the resume, Timelines will be more realistic and business personnel will realize that in fact they have no freaking clue how to manage a development environment and that they had better get someone who does know.

Re:Mod parent up!

[mrchaotica]

What we really need is to promote the concept of the CodeSmith like a balcksmish, silversmith or whatever, coding is a skilled artisan occupation.

Or promote the concept of the title "Software Engineer" requiring a professional certification, the way real engineering does. Any slob off the street can't suddenly proclaim himself a "Professional Civil Engineer" and start building skyscrapers; why should he be allowed to proclaim himself a "Professional Software Engineer" and start coding medical databases?

Re:Mod parent up!

[jc42]

Or promote the concept of the title "Software Engineer" requiring a professional certification, the way real engineering does.

Actually, this has been attempted repeatedly in the software business. It has always failed, for pretty much the same two reasons.

First, the only thing that the organizers (invariably software managers) can agree on is testing for knowledge of the mundane introductory-level details of IBM/Microsoft software. This means that in the more technical areas, where other (higher-quality) systems are used, there is no certification.

Second, even if the customer wants software that runs on IBM/Microsoft systems, they quickly realize that that "mundane introductory-level details" part can be rephrased as "It's a totally Mickey-Mouse certification", and the people who get certified tend to be the novices who think that a certification will get them the big bucks that their knowledge doesn't qualify them for.

So in software, "professional certification" always turns out to be a big joke. Everyone knows this, but pretends that a good certification program would help.

And no, I don't know how to make a non-joke certification program, either, despite several decades of building software. If I did, I certainly wouldn't be hired to take part in the committee that manages the certification program. (But I'm disqualified anyway, because I've worked too much on non-IBM, non-Microsoft systems to be considered for a position in a certification committee. ;-)

Re:Mod parent up!

[Jerry Coffin]

...promote the concept of the title "Software Engineer" requiring a professional certification, the way real engineering does.

For this to do any good, you'd need reasonable criteria upon which to base the certification. IMO, TFA was mostly correct in pointing out that right now, we don't know what those criteria should be. The Mythical Man-Month remains relevant today largely because relatively little progress has been made in solving the problems it explored. As Brooks pointed out in No Silver Bullet, we've cured a lot of the accidental problems of software engineering, but done relatively little about the essential problems.

In terms of the certification itself, we're just about as lost. The correlation between formal schooling and success as a software engineer is weak at best, and quite possibly negative. Experience indicates more, but still very little. Certification based on such criteria would do more harm than good.

There are other problems as well. With buildings, it's pretty easy for most people to realize that "skyscraper" and "dog house" have different requirements as to the level of expertise involved in their design. In software those differences often aren't nearly so obvious. The managerial people who'd need to enforce the requirement for certification don't know the difference between a skyscraper and a scale model of one for a train set (and sometimes the developers don't either). "Web developers" (in particular) seem prone to jumping (metaphorically) from doll houses to 18-lane suspension bridges in high-wind, earthquake-prone areas.

Where software developers sell themselves short

[Digital_Quartz]

When you've contracted an engineer to design a bridge, and you want to make several large-scale changes to that bridge, the engineer will come back and say "If you want these changes done, this is how long it will take. If I don't have that much time, I can't make the changes".

In fact, I'd go a step further; software developers tend to say "This is how long it will take to make the change, and this is how long it will take for me to hack something together." Bridge engineers don't say things like that. They don't put that "hack something unsafe together" option out there on the table, and neither should we.

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

Re:Where software developers sell themselves short

[kthejoker]

The chief reason for this, though, is the lack of clear requirements on the initital project manager / architect's part, though.

When you're building a bridge, there's a base requirement: something can cross it. Now what can cross it, how fast, structural integrity, etc, all play a factor in the specifics, but at the end of the day, either something can cross the bridge (safely / quickly / profitably) or it cannot.

If you were told to write a program to add two numbers together, and that was all, you'd have a pretty easy time rejecting "hacky" options, too - because you know all of the requirements.

But most of the time you don't, or you misunderstand them, or (in the classic case) you offer up something and the project suddenly takes a dramatic turn because you did one little neat thing in it and they want the whole system programmed around that tiny widget (at the expense of all others).

Comparing software to a bridge is useless. A bridge is like one specific program - a calculator, or a no-frills text editor. Building software is more like urban planning. Now go ask engineers how many times they've screwed the pooch on that one.

Re:Where software developers sell themselves short

[Beryllium Sphere(tm)]

>If you were told to write a program to add two numbers together, and that was all, you'd have a pretty easy time rejecting "hacky" options, too - because you know all of the requirements.

Can the sum of the numbers exceed MAXINT? MAXINT on what platform? Should overflows set a flag, throw an exception, or be reported in-band?

Can the numbers be floating point? If one is much smaller than the other, how large a fraction of the smaller number's precision are you willing to lose?

Can the numbers be multiple-precision? Is the multiple-precision library you're using compatible with the rest of your system? Does it fit into your memory requirements?

Do the numbers come from some real-world source that need to be checked for sanity, for example the output of a sensor?

In what form does the output need to be? Encoded in ASN.1? BCD? IBM "unpacked decimal"? String? If output is variable length, is there a maximum size that must be enforced to prevent buffer overflows? If the maximum size is exceeded, should the result be truncated, or should the function throw an exception, or ...

The sickening thing being that I've probably left out some important issues. Like thread safety.

Look up the bug history of "IEFBR14", originally a single return instruction that required four or five revisions to meet shifting requirements.

Re:Where software developers sell themselves short

[Grishnakh]

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

You seem to be missing something: software engineers are not "professionals" in the sense that civil engineers are, and they're not licensed. There's no liability to a "software engineer" if anything happens, only to his employer.

You can't say "no" to your boss and expect to keep your job long. Bridge-building engineers don't have that problem because they ARE the boss, just like doctors don't work for bosses as they themselves are the boss.

You want to make software engineers accountable for their work? You'll need to mandate government licensing, and then you'll need to pay those developers 2x-3x as much because they'll have to have malpractice insurance. Somehow I don't think any software companies want this.

Re:Where software developers sell themselves short

[Digital_Quartz]

There's no liability to the software designer's employer, either, or next to none.

As the old joke goes, Microsoft made IIS susceptible to worms like Melissa and Code Red, costing billions of dollars to companies around the world, and didn't have to spend a dime in restitution, while Intel made the Pentium with a floating point error that affected a handful of people doing extremely precise simulation work, and had to spend a small fortune recalling chips.

Even if we just had software liability, we'd see a much greater focus on quality in our industry. Although we tread a thin line trying to introduce that concept without having horrendous effect on open source.

It's not that complicated...

[Gunark]

The answer to this question isn't complicated. Software development is hard because it requires that the developers impose an artificial abstraction on the real world -- a real world that could be conceptualized in an infinite number of ways.

Once you decide on some particular conceptualization, you make certain assumptions about what is and isn't possible. This is the root cause of most if not all of developers' headaches. Despite our best intentions, we find that our carefully engineered abstraction does not capture everything about the real world relevant to the given task (either because we got it wrong the first time, or because the task itself has changed).

This, in fact, is a deep problem in computation in general -- a huge obstacle in artificial intelligence, for example. We need an algorithm that can pick out the salient aspects of a problem and ignore the irrelevant ones. Until the day that we have this (and that day may never come) the hard part of software development will remain an art form based on intuition and creativity.

...because planning isn't doing and we want to do!

[jofny]

I see more projects go bad because of poor communication and lack of architectural efforts than anything else.
A sufficiently detailed question needs no answer. Likewise, with software development (and really most computer systems development), everyone wants to rush out and start coding as fast as possible...but if they'd made sure there were no assumptions, defined their problem specifically, and spent way more time than they wanted to developing an architecture, the coding afterwards would just be data entry for the most part.
But...people tend to be averse to taking the up-front costs of not only gathering their requirements, but also -understanding them- and making them -fit together-. Frameworks and Architectures tend to be poorly developed and understood ahead of time. Features are developed in isolation from each other. Test cases need to be developed at every step to keep people in agreement and to make find issues early. If your project isn't working in a manner where every new addition to it can't be tested as it's written, you're doing something wrong.
Requirements need to be reviewed and revised and reviewed and revised constantly with all staekholders against those test cases.
But...maybe I'm off base.

An old-timey programmer looks at this...

[troll]

We promise according to our hopes; we perform according to our fears.

Managers -- down to mid-level -- need to have dates, so the programmer gives the earliest possible date. The software goes out buggy, untested. More pressure on the programmer and management doesn't trust programmer estimates anymore.

Solution? None, with the current business model.

Short Answer

[955301]

Software is still a Science and not an Engineering practice.

As long as the design can also be the implementor and estimates based on actual analysis are optional, Software will NEVER be an engineering study.

These aren't changing quickly for what I believe are a few reasons:

IEEE has not created a Professional Engineer - Software and noone really wants them to.
Companies don't like to be told they have to hire something that sounds expensive to build something they cannot see.
Companies will NEVER open their software to outside inspection the way construction companies must open their buildings because of the concept (flawed, I think) of Intellectual Property.

If a company had to have their software inspected by a Software P.E. before using it in production or selling it to end users - If Software P.E.'s had to adhere to standards which included concrete estimates and testing - If companies were not allowed to use anyone they could find that has seen a computer to write their software... commercial software development would be much further ahead.

Do I believe any of this should happen? No. Why? Because I want it to continue to fail. I do not believe software development should be put on a pedestal or only performed by "experts". I believe shoot from the hip software projects allow open source software projects to exist and to succeed in the market.

Open source works without accurate estimates because the contributors can flock to good projects and don't have to adhere to a labour budget. Company employees can't get wind of the cool software project and leave the crappy one's - corporate structure and budget's won't let them.

I don't believe companies with more than 120-150 people are stable - once they breach this range empire building occurs and massive uncontrollable monsters result. If a company truly needs more than 150 people it should split into two and partner on the project at hand. I believe this is a human condition - humans work best in tribes where they can personally know all of the members.

All of this might be completely and utterly wrong. But it's my hypothesis.

Because software design isn't construction

[Todd Knarr]

Software design and development is more akin to an architect designing a building rather than the more common analogy of building the building once it's designed. Except that software developers are often burdened with requirements that any architect who valued his license would reject. For example, management often dictates which parts are to be used (ie. "We're going to use MS SQL Server as the database engine."). What architect would design a skyscraper after having been ordered by the client to use pine 2x4s instead of steel beams, or design a 1-story residential house under the requirement that he use titanium box beams instead of 2x4s? And then there's what the article notes at the top of page 3: software requirements change right up to the day of release. When an architect goes to design a building, the requirements are fixed before he starts. Square footage, height, number of floors, number of people each floor has to accomodate, number of elevators, number of toilets needed on each floor, how high the ceiling of the lobby floor will be, all that's fixed in stone before any design work begins. Sometimes that requires back-and-forth between the client and the architect to get everything clear, but it all happens before major design work begins. No architect's going to design the foundations of a building until he knows exactly (or at least very very closely) how much mass is going to have to sit on those foundations and how much horizontal shear they're going to have to handle. If the client can't decide what he wants, the architect just goes "Fine, call me when you decide what you want.". And even when this kind of analysis is done, software requirements change constantly after design's started. See above, no architect's going to accept the client changing the number of floors or the square footage of floors without also agreeing to a complete redesign to accomodate the changes with all the delays and additional costs that requires. If the client didn't like it, the architect would just hand the client the work to date and tell him to have a nice day. And no there wouldn't be any refund of money, the architect's held up his part of the deal as best the client will let him.

And then there's another parallel: architecture is only half engineering, the other half is art. Every building is different, and it's accepted that the architect's going to have to have time to come up with the parts that aren't off-the-shelf standard. There isn't a single standard floor-plan for a single-family 3-bedroom house, there aren't any rules that can be mechanically applied to create a floor-plan, and it's accepted that the process of creating a floor-plan is more creative than anything else and people without the knack for it just aren't going to be very good at it. And that on the next single-family 3-bedroom house much of that work's going to have to be done over again because the new client doesn't want the same thing the previous client wanted.

Software development is hard because....

[Anonymous Coward]

... of all the hot babes running around the offices in miniskirts giving massages that make it tricky to type. That, and the martinis that keep spilling in the keyboard. Combine that with the constant parties, sailing trips, ski trips, etc and it's a wonder anything gets done.

What's that you say? It's not the .com bubble anymore? Glad I left software.

Gap between computer science and person problems

[lawpoop]

I believe that these days there is a gap between the kinds of concepts that you learn in computer science, which are mathematical things, and the everyday social problems that people are trying to solve with computers. It was summed up nicely a few years ago in a criticism of open-source software someone on the internet -- someone was complaining that the developers of GNUcash were dealing with memory leaks. I think they were using c or c++, and the complainer was saying they should migrate to java or python or something. If you're trying to make a computer program to solve problems, it's a waste of time to be solving computer problems. You should be solving the pre-existing human problem.

Every tool has its own problems. One problem is accounting, or keeping track of money. With pen and paper, you can run out of ink or paper. Humans aren't good at adding numbers in their heads. With computers, you can completely erase all of your work with a few clicks of the mouse or keyboard. So there are ways that problems inherent in the tool can emerge, which take energy and attention away from solving the pre-existing problem.

Currently with computers we are trying to abstract problems such as banking and business into simple logic puzzles. I think that's too much of a simplification. I think we need to create a virtual world full of basic human-percieved concepts, such as time, weather, humans, animals, etc. and create programs by manipulating those basic ideas and objects.

An example is an ontological system like OpenCyc. An ontological system holds hundreds of thousands of logical assertions like "Animals eat Food" and "Paris is the Capital of France". Basically, an ontology system has some basic common sense. From all of these assertions, it can make logical conclusions. So, if well tell Cyc that Duke is a Dog, it can conclude that Duke has a tail and eats food. If we tell it that Duke lives in Paris, it knows that Duke lives in France.

Now imagine, instead of dealing with animals and where they live, it has a bunch of assertions about generally accepting accounting principles. One day, you might be able to just sit down and talk with an ontological system via email or IM, and say, "We got a check from client A for $575, another check for $440." and then the computer balances the books with all the other accounting principles it 'knows'.

Current programs seems to be exclusively a digital re-creation of paper-based forms and filing cabinets. It's a sheet of paper with a bunch of field:value pairs, and reports are the resulting logical operations you can do with all of that data. This is *basically* the relational database. I think we are hitting the limit of the field:value model of reality. I think there are other models, such as virtual realities like online worlds, knowledge systems like opencyc, etc.

Programmers are working exclusively at too low of a level. Yes, of course, we will always need to teach and understand basic boolean logic and computer science terms, but we need to start working at higher-level, human friendly concepts.

Why it's so difficult

[BluedemonX]

BLOAT - Look at J2EE and COM.

EGO - Noone will admit they don't know what they want, noone will admit that what they wrote thet were learning while they were coding it, etc.

GENERAL CLUELESSNESS: Figure out what you want and build it. In that order. "If I don't write down clear specifications I can't be held accountable if they're wrong" is the manager's mantra.

MAGIC BULLET THEORY: Technology X will solve all our problems! And our neighbour's problems (see first point)

HR ISSUES: Must have 10 years experience in .NET (posted a year after .NET was released), Must have experience in Java, J2EE, J2ME, J2SE, JAX, JNI, JWD, JQV, WWJD, SOA, SOL, SQL, SPL, APL, OBE, CBE, ... (see point one) and AN EXPERT IN ALL THE ABOVE.

Corollary to the above: don't hire someone with a clue who architects well and can translate the design into whatever language you want (or tell you why "object oriented" and "EJB" don't co-exist) find someone who either guesses the exact answer to your clever pointer question you thought was a cool hack, or find someone with tons of certifications and ultrageekery in the language du jour (the GOLDEN HAMMER antipattern)

Lack of a specification language!

[vinn01]

Home builders have architectural plans. Machinist have blueprints. Electronic equipment builders have schematics.

Software engineers are stuck trying to figure out the incoherent ramblings of marketing/sales/business analysts/corporate executives/users and a host of others who have no means to specify what they are asking for.

Software specifications are uniformly deplorable.

Re:It's not hard

[CastrTroy]

Exactly. Anybody who thinks it's easy to complete a large software project without problems has obviously never worked on real software. Secondly, anybody who asks why it's so hard to develop good software has never worked on a large software project. You can say a lot of stuff about following processes, and doing testing, but until you're working on a large project, with lots of different developers, with time and money constraints, you don't know what you're talking about.

Software has HYSTERICAL complexity.

[Ungrounded Lightning]

What makes software so hard? The enormous complexity of the software constructions.

Why is it so complex? Because it's so EASY to build it.

Example: Pre-computers, the moment-to-moment computations necessary to run an automobile engine were performed by mechanical devices, mainly the distributor and carburator. Every term in every computation was manufactured as a number of physical components, several of which are moving parts.

For instance: The RPM input to the spark advance was computed by two weights on pivots, with springs and stops, rotating a sleeve on a shaft. The shaft was driven by the camshft through a gear and the sleeve carried the cam driving the contact points or (in an electronic ignition) the starwheel that passed the sensor coil. Adding this computation (compared to no RMP spark advance) added five moving and four stationary parts, to be assembeled, and a test stand the volume of an office cube to test the result, and allow a worker to adjust the constants by bending the spring supports with a screwdiver.

In software this computation can be done by PART of ONE line of code. (In real engine controls it's actually done by more - mainly so the computation can be more complex and thus better approximate ideal running conditions.)

Software changed the game completely: When adding a piece of a compuation requires a moment of thought, a minute with a text editor, and issuing a compile command (plus whatever testing is necessary to convince yourself you got it right), rather than months of an engineer's and draftsman's time, manufacuture of dies, lab work to check the result, repeat through three layers of departments (to prove it can be done, to prove it can be done reliably, and to prove it can be manufactured affordablly), the amount of work and time to implement one bit of complexity reduced drastically.

The result was that the amount of complexity that can be afforded rose in proportion. Given that the proportion was hysterically large, the amount of complexity handled by each person became enormous.

Unfortunately, programming is NOT formulaic. Portions are - and as they are identified they are rendered into algorithms and software is written to perform them. The result is that the part people work on is ALWAYS the part that is "fuzzy" and difficult to formalize.

Programming consists of rendering a set of requirements into a correct specification for meeting the requirements. (The reset is automated.) This is not an easy task - and it gets more difficult with increasing complexity of function. Unfortunately, methodologies for performing it have remained in a catch-up game: The better the tools, the more complexity a worker can handle. The more he can handle, the more he is assigned.

To quote McClary's Third Law of Computer Technology: Software complexity expands to exceed the capability of any software development methodology.

Bay Bridge

[whoever57]

The Bay Bridge is an interesing parallel -- the delays have been caused by unclear and shifting requirements (which are focussed on the aesthetics). These changing requirements have led to delays and cost overruns.

It does not have to

[trydk]

I find a general tendency today towards KICS (Keep It Complex, Stupid) programming instead of KISS. This means that people have modules with tens (or, shudder, hundreds) of thousands of lines of code, which are hard to maintain (an average programmer can probably handle a few thousands lines of code in her head, if that much) and even harder to test/debug. I personally try to make my programs very modular, where the modules are simple, single-function and testable. I do put a lot of thought into planning ahead and designing the module interfaces so I rarely have to change the interface specifications during the project. And I put comments into the programs, often way above the norm, in fact, some of my code could be read and understood completely just by reading the comments. Waste of time, some would say, but I like it. To be slightly more specific about the modularity:

• Simple means small and relatively linear, i.e. as little convoluted code as possible.

• Single-function means, well, that it performs only a single function. In the program I may have a general formatting module that can format dates, strings and numbers, depending a bit on the complexity of each type of formatting, I might choose to have further modules that each handle one type, i.e. a date-formatting module, a string-formatting module and a number-formatting module.

• Testable means that each and every module can be tested independently, which normally implies no side-effects (not an absolute demand, though). I always write a test section for each module and make the tests cover as many parts of the code as possible (hopefully all sections of the code, but for slightly complex modules this may be infeasible or even impossible).

By sticking to these rules and by documenting the code thoroughly, coding does not have to be too difficult. These rules/principles, I know, are part of some of the major design methodologies, but you do not have to make it more fancy than this to make it work. The above principles are very hard to retrofit into existing (legacy?) code that was not built this way, but extensions could still be made in a modular way that, if not to the letter, then in principle, adheres to the rules.

Re:It's not hard

[joto]

In my experience the only time it becomes difficult is when people are NOT doing their jobs.

In that case, you have essentially proven that you have no experience with large software projects. Even if everybody is qualified for the job they are hired to do, and are enthusiastic about the project, and doing the things they are supposed to do, things just don't work out right at first try. Or second try. Or third try. Or... you get the picture. And then, when that problem's finally done with, you have 7 new ones, where at least 3 of them are total show stoppers.

It's of course easy to start pointing fingers at people. That guy is an idiot. The management has no idea what we are doing. Writing documentation is killing my productivity. I had to rewrite this assholes code since he didn't follow my favourite bracing style. And so on... The point is, the guy you call an idiot probably knows three hundred times more than you about fluid dynamics, which is what all the project is about. Management isn't supposed to know what you are doing, they are supposed to handle budgets, equipment, hiring, firing, etc. And the last two are kind of obvious...

Software development is hard, because it's not a solved problem. You don't build software the same way you build buildings. There are no rigid rules to follow, no best practices that can be universally agreed upon. The purpose of each new software project is to solve (a) problem(s) that has never been solved before. And because of that, there are great uncertainties involved. You can guesstimate a lot of parameters, but eventually, some of the unknowns are going to bite you in the ass. (As Rumsfeld said: There are known knowns...)

The only other problem I've encountered with software development are when people have unrealistic expectations, despite me telling them exactly what to expect and delivering on exactly that.

Not exactly an easy person to work with, are you?

Development fails because developers are ignored

[ph1ll]

Well said.

But while stereotypes persist that programmers have no people skills, you forget that many business people don't either.

Just ask yourself: how many effective, people-oriented bosses have I ever had? If your answer is "not many", you're not alone.

I've been software engineering for over a decade. These are a few observations I'd like to share with managers:

1. Hire good people. If you don't, you're hosed before you start. If you can't tell the good people from the bad, you're in the wrong business.
2. Software engineering often fails because software engineers are ignored. In a badly run company, nobody even asks developers which is the best approach. Imagine if civil engineering were like this...
3. "Software is too expensive to do cheaply". Sure, you can get 20 engineers with 2 years experience apiece. But it's more effective to get 4 engineers with 10 years experience.
4. Let the team (largely) manage themselves. Open source projects work just fine without management - or, at least they do no worse than managed software...
5. Don't do things to keep morale high; instead, cease doing things that keep morale low. We engineers just want to do good work in a nice environment. It goes back to hiring good people and then leaving them well alone.
6. Give the team a hand in decision making. The team will be more incentivised to hit targets they set than the ones you set. Afterall, the only incentive they have for hitting your impossible deadlines is furthering your career.
7. Automated regression tests! There are still managers who don't know what these are/how important they are. Along with all other good processes, automated regression tests reduce stress.

Today's applications are like 1850s houses.

[Kadin2048]

Of all the Renaissance painters that ever painted, only a few became forever famous.

I think this might be taking the art side a little far.

With a painting, the surface appearance is the end product. If it looks good, it is good.

This is distinctly not the case with software. Not only does the application (or whatever) have to look good on casual inspection, but have to be built to work well, including under conditions that might not have been thought of in the beginning.

I think architecture as a metaphor for software development is actually pretty good; architects have to combine artistic judgment and technical skill in order to produce a building that's both aesthetically appealing -- true works of art, in some cases -- but also structurally sound, designed according to accepted standards and principles. It's not enough, in most cases, to just have either one. Nobody wants an ugly building (well, most people don't), and nobody wants a building that's just a facade without substance.

Software "architecture" is where house-building was a few hundred years ago. When you wanted a house built, you just went and got a few people that you thought would be good at building houses, or had good reputations for building them, and told them what you wanted. They might or might not have any formal training, and the only 'standards' were what they knew from past experience would probably work.

I suspect that in a century or two, software will be not unlike house-building is today. While every house today is different, depending on the owner's requirements, there are a set of well-understood and basically accepted standards for putting them together. That doesn't mean that some builders aren't better than others, or that some don't cut corners, or that there's not room for artistry and physical beauty in the final product, but just that in theory, they all meet minimum standards for structural integrity and other key factors. In time, I think software design will probably follow. Every piece of custom software will be different, because they each solve different problems, but getting to that solution will involve a combination of creativity and the application of existing standards, built up from the accepted body of knowledge of what works and what doesn't.

Re:Today's applications are like 1850s houses.

[Grishnakh]

I suspect that in a century or two, software will be not unlike house-building is today.

So you're saying that in 100-200 years, almost all software will be developed by large companies who charge high prices, but hire illegal immigrants with little or no training to do the building? Worse, the software won't be written correctly, and won't be checked because 1) the company doesn't care and 2) the government software inspector is paid off by the company to look the other way instead of doing a proper inspection to make sure the software is "to code"? And while the software may look cool at first, it'll stop working in a few years after numerous parts have broken? And if all that wasn't bad enough, the software developers will come back to work after-hours and steal parts of the application and resell them on the black market?

No thanks.

Re:Today's applications are like 1850s houses.

[Glonoinha]

Shit man you don't have to wait a century for that scenario - that's what we have TODAY.

Re:Lesson:

[BSAtHome]

> Wish somebody had told that to the first guy who coded ls. Three screens worth of switches is NOT doing "one thing well".

You are wrong. ls does _one_ thing very well: it lists files; and it does a very good job at that IMO. The switches are not to increase the information stream, but are meant to decrease it (your milage may vary). The formatting is an important part (for compatibility) but the function is still to list files.

Although the options act visually as formatting, they are, in fact, filters. They filter the information in a very well defined way (well, most do ;-). The real use of ls is in scripts in combination with other `small' programs.

Looks like he doesn't know Knuth

[plopez]

FTA:
The idea is that if we're going to turn the creation of software into a true science, we need to first have principles. We need to know the fundamentals and formulas by which software behaves. What are the laws and principles we can count on in creating it?

I think Knuth has been working on that.

But in general, the reason software is hard is that the problem domain is hard. You can't solve anything without understanding the problem. This is why hiring code monkeys causes so many problems. THey may make nice GUIs and know how to build a web based wizard 'out of the box', but end up knowing nothing about the problem at hand.

Other observations:
1) He gets it write when he talks about things always changing. Part of the reason projects like Vista get delayed is that they take so long is that the requirements change. If you can't do it in 6-9 months, don't do it.

2) Nice to see him reference Brooks.

3) Another nice qoute:
The only real reason that a team gets together and writes a new piece of software is that they want to do something new, something another product doesn't do, whether it's to add a feature, or to be compatible in a different way with another system.

For years now I have said: "Software development is R&D". If the software has been built then use it. If you must do scratch development, then understand that it will be open ended; and all attempts to schedule, budget and predict will fail. It is *not* an industrial process, though we tend to treat it as such.

4) My observation is that much of 'Software Development' is actually social. You are working with users, managers, investors and programmers and trying to get everyone to agree on something. It gets political *fast*. So a good SD team must not only bee good at learning new problems domains and coding up solutions, but also must be good at human relationsships. And the closer the development team is to the end user, the easier this is.

The real point is...

[Anonymous Coward]

...that software development needs to have a team of programmers who are talented and competent, and a project leader who knows his people and can effectively lead and manage both the team and the project... and a corporate culture which doesn't treat its people distrustfully like a bunch of indentured servants, then they can write good software and get it done in reasonable amount of time. Getting and retaining a team of very good talented and experienced programmers is very difficult and expensive however since most development firms are cheapskates and not only do not pay enough, but treat their people like dirt, binding them under all kinds of bullshit NDA and non-compete contracts, poor work environment with noisy cubicles or "bullpen" office areas, not empowering them with latest technology workstations and tools, etc. The software development company that teats its programmers golden is the rare exception... the exact opposite "sweatshop" like I described above is generally the norm, as I've seen it in my 20+ years in the biz, hence you usually get kids right out of school as entry-level positions doing the bulk of the development. The best cream of the crop programmers have mostly all quit programming years ago and now are all sitting in management positions and programming no longer.

A good orchestra conductor who is in front of a bunch of rank beginner inexperienced musicians will not be able to make very good sounding music. You get what you pay for.

Re:The real point is...

[Metasquares]

The week before a concert is absolutely brutal, especially if you're rehearsing with a band/orchestra (but even if you're playing solo piano and rehearsing on your own). You're treated fairly well at other times (most importantly, after the performance), but you can sure feel like dirt during all of the rehearsals.

I guess it's like being treated with respect as a programmer, except that you still have deathmarches.

Your analogy is good, though - hire amateur musicians, and you're not going to get a good outcome. If the conductor knows next to nothing about music, you're not going to get a good outcome. If the instruments the orchestra is using are not tuned well, you're not going to get a good outcome. If you rehearse a piano part for weeks and the conductor suddenly asks you to play the oboe instead five minutes before the concert, you're not going to get a good outcome...

Captain obvious

[plopez]

Herding cats is hard because you are using the wrong management technique. You herd cattle (and sheep and goats and pigs etc.), you do *not* herd cats. Cats, you put them in the general area of the mice and let them do what they are good at. Micromanagement of cats is a losing proposition.

I dont understand the question......

[MrBandersnatch]

Software development is now incredibly *EASY*. I mean we have tools such as C#, VB, .Net, Java, Perl, PHP, Python, COM, CORBA, SQL, J2EE, IIS, APACHE, XML, XSLT, HTML, XHTML, SOAP, XML-RPC, JBOSS, ZOPE, CSS, AJAX, Javascript, XQuery, XPath, UML, Patterns, SCRUMM, WMF, CardSpace, Passport, Windows, Linux, OS-X, WME, Direct-X, OpenGL, SDL, Eclipse, SVG....I mean, all this stuff software development still cant be "hard" now can it?

Too much creativity left to the peons

[CPE1704TKS]

I've been thinking about this for a while, and my answer will probably rustle a few feathers will all the developers in the crowd. I know I don't like the answer since I am a developer as well, but I believe it is correct.

The simple fact is that too much of the software development is left to the peons, ie. the developers. The skillset of the developers are totally random, as is the style, their expertise, etc. It introduces too much variability to the software development process.

Look at civil engineering projects. They are able to create buildings, bridges, roads, etc, and for the most part are much better understood than software development projects. But the peons, ie. the construction workers, do not dictate anything. There is one way of bolting steel together, one way of mixing concrete, etc. The only person who has any say is the chief architect.

However, in software, a lot of the design decisions are left to the developer.

In order to introduce predictability to the software development cycle, unfortunately, all variability must be removed. Developers must not have the freedom to code whichever way they want to, but have a simple, standardized way of accomplishing similar tasks throughout the codebase. This completely removes creativity and enhances predictability and leave all creatvity to the chief architect for the project.

Obviously, I don't like this answer because it's the creativity portion of programing that I enjoy the most, but I think it is for the most part correct. With the people I work with, their skill levels vary from good to horrible. Software development requires a certain level of skill sets that most people don't have and too many people who shouldn't be coders are coders. These are the people that introduce bugs, etc.

Re:Too much creativity left to the peons

[toganet]

I tend to agree with you on this, although a bit reluctantly. I also think a large reason why this part of the problem persists is the general laziness of senior developers. Speaking from personal experience interacting with senior developers and so-called 'software architects', coders suck at communication and leadership skills. Whether it's ego or shyness, they don't seem to be able to do much beyond just coding.

Where I work the end result of this is that the 8 or 10 senior developers all have discrete areas of expertise, rarely interact even with each other outside of a rowdy lunch or two, and cannot effectively back each other up when one is out of town. In addition, junior developers struggle to get up to speed on our crazy chewing gum-and-duct-tape "platform", and our attrition rate is about 50%.

I'd love to work someplace where the senior developers actually design the application, provide detailed instructions to their subordinates, and consistent coding practices are followed -- but it sounds like those places are few and far between.

Re:Cats and penguins...

[thrillseeker]

It makes one wonder why the Linux mascot is a Penguin?

Indeed - it should have been this - it so appropos in so many ways.

Root Causes

[kaoshin]

The Worse is Better design philosophy, idiocy, selfishness and greed are factors leading to the difficulty of software design.

Training & under estimating the difficulty

[aGuyNamedJoe]

I think part of the problem is training. Too often programmers are taught "How to use a computer to do X" (and often " ... in language Y") instead of "How to do X". The focus is on the tool(s). If we taught carpentry similarly, and the Saw was the primary tool, the training would be "How do cut siding with a Saw", "Building Frameworks with a saw", "driving nails with a saw", "Roofing with saws!", etc.

Computer science seems to focus on the computer. Engineering more often focuses on problem solving and/or design, with a emphasis on the type of engineering the department specializes in. I think it would be interesting to see if there's a difference in projects where the primary degree is Computer Engineering vs Computer Science vs Electrical Engineering. Also, many of CompSci departments seem to have cut way back on math and other subjects that focus on abstract thinking. I don't want to condemn all CompSci departments of course, but having taught and helped in curriculum development in Comp Sci, I'm not completely blowing smoke here -- one has to put up a fight to keep broader courses in the curriculum

However, even if that's completely off base, and I'll allow it could be, there's also the fact that people tend to severly underestimate the difficulty of designing a system. We'll happily decide to do something in software we'd never consider doing in hardware. Why? Does the system design get easier when the construction material is completely insubstantial and there's essentially no intermediate level at which parts of the design can be simulated or tested? If a system is to be implemented in electronics, there's a lot of science that can be applied at various levels (differential equations, boolean algebra, component simulators, etc). If it's going to be in software, we don't need that, the programmer can do it all in her head!

We know hardware design is difficult. For some reason, we think software design is easy.

(I'm sorry, perhaps I'd best take my medication...)

Re:It's not hard

[Brummund]

I have no idea, but according to his blog, he can't adjust his alarm clock. So, my guess is he's in management somehow. ;-)

( http://www.makesitgood.net/ )