Why Software is Hard

GoCanes writes "Salon's Scott Rosenberg explains why even small-scale programming projects can take years to complete, one programmer is often better than two, and the meaning of 'Rosenberg's Law.' After almost 50 years, the state of the art is still pretty darn bad. His point is that as long as you're trying to do something that has already been done, then you have an adequate frame of reference to estimate how long it will take/cost. But if software is at all interesting, it's because no one else has done it before."

Comments

Not to take potshots, but

[Mateo_LeFou]

...can anyone explain Vista's schedule in light of this discovery?

Re:Not to take potshots, but

[edwardpickman]

...can anyone explain Vista's schedule in light of this discovery?

Another law explains it, Entropy.

Re:Not to take potshots, but

[__aaclcg7560]

All the programmers got better jobs at Google?

Re:Not to take potshots, but

[MindStalker]

Its esentially the problem with any large company. Any project coming out of a large company spends 90% of its time in committee and debates, and generally gets watered down to the few things those committees can agree on. Personally I disagree on the idea that two coders arn't as good as one. One coder rarely has the motivation as well as the insight to see clearly his way through a project (getting stuck on a problem and having another viewpoint is ALWAYS helpful). But there is definatly a deminishing return as project teams get larger.

Re:Not to take potshots, but

[Almost-Retired]

I disagree on the idea that two coders arn't as good as one

This is a point I've come to also after 72 years, the last 35 or so of it coding this and that although not much recently as I seem to be fadeing into the dim sunset of SS mentally.

When working in assembly, then it may be that one person is the optimum number of coders as I've done some never before been done stuff in assembly several times in the early years, sometimes with hand assembly (on an 1802 board) where you look up the nemonic and enter the hex equ in a hex monitor. It took me about 6 months to fine tune about 3k of code, but it was still running 12 years later when I last checked in at that station. And still saving the station 2-3 man hours a day and giving them a better air product at the same time.

But for a higher level language, I think 2 can be more productive, particularly when one knows what he wants to do, and the other knows how to do it once its properly outlined. Many times the coder himself is simply too close to the code to see the job it has to do, but the partner in turn has a good idea of what its got to do. The genesis of at least 2 fairly well known amiga programs were from the mind of a younger man in another dept at the tv station, and he would hack up what he thought might work but didn't, but once I knew the requirements, the final code more than likely came from my keyboard. He had the imagination that I lacked, possibly due to my advanceing age, and was in turn concentrating on his job's duties which I wasn't always aware (I had other responsibilities too) were being done at less than optimal methods. We sure made a good combo crew though.

I have NDI how many man-years in in vista right now, but I dare say it is a substantial investment in both time and programmer salaries. I'd also wager that at least 75% of any one programmers day was spent conferring with other programmers as to the best way to do it, and get it done within the generally immutable confines of the .h header files. This is NOT to me, best use of the programmers time, so the what does it do, and how it does that, really ought to be seperated in any large project.

As for re-using known good code ideas, or a 150 line snippet here and there, it is to be encouraged at every staff meeting, re-inventing the wheel is not good use of his time and as others have said, only serve to intro new bugs that then have to be run down and fixed. Programmers really should get over the attitude that I can write it quicker, and spend more time reviewing older code to see if it can be recycled. There is much knowledge in 10 year old code thats still in use everyday.

Will my little treatise make any difference at the end of the week? Donbesilly, This is after all, /. :-)

--
Cheers, Gene

Programmers

[bendodge]

One programmer is better than two for the same reason that one woman in the kitchen is better than 2. You have to get on a pretty large scale before you need multiple cooks/programmers.

Software programming in general is hard for 2 reasons:
1. Computers aren't built for interfacing with humans, thus UI us terribly time-consuming.
2. The environments people like to drop an app into can be so bizarre, that rock-solid stability is very difficult to achieve.

Re:Programmers

[cyborg_zx]

Indeed.

It is well known that men are superior in the kitchen.

Re:Programmers

[Aphrika]

...and also that 2 women are superior in the bedroom...

Nine women cannot have one baby in one month

[euice]

and of course, we are the better programmers, so better fire those other 8.

It needs more professionalism

[petes_PoV]

Mostly programmers are trained in the technical details of languages and the libraries/APIs associated with them. They don't gain skills in knowing what users really want and are hurried into producing barely-working stuff, fast.

Whatever testing is done often only tests that the product produces the correct answers when feed the proper input - no account is taken for how the program reacts to incorrect or incomplete data.
Changes are requested faster than they can be implemented and often are not communicated very well.

In short there are systemic failures throughout the whole process, from inception through to delivery. There is no single answer to why software is hard and there won't be until the industry matures and people start to get thrown out of the business for acting unprofessionally

Re:It needs more professionalism

[DarkOx]

Yea, specifications *are* the biggest problem. Unless I happen to be the end user, rarely does the end user know what he wants me to build until I show him/her the first prototype. At that point they might start to clarify what they want, If I am lucky their ideas are heavily colored by what they saw in the prototype and we can go for there, if not its back to square one, and usually several more trips there after that.

I get requests like we need a program that users can run from the web that shows them if batch scans are in balance. I am then left to resolve on my own things like:
*What is a batch scan
*Where can one find a batch scan
*What does it mean for it to balance
*Can the thing just print a big Y or N in size 48font on the screen or does it need to detail something
--Then comes the anticipatory stuff
*Is the user going to expect to be able to correct an in blance
*Now that I understand what a batch scan is *maybe* I see all this other stuff should I also report on those things
*etc .. etc

People like to expect programming to go like engineering or architecture. Its not the coders don't want to apply discipline to their craft its that they can't. Nobody would dream of approaching an engineer or an architect without a pretty good idea of what they want to do. That is not to say the professional is not going to have to help them work out most of the details but the basics are going to be pretty clear. Imagine if I sent a request to an Architect asking for a "Structure that will be used by people." and gave no other information. I really doubt I would get a call back.

You can argue the PAs are not as good as they should be about requirements gathering, but I think there could be much more professionalism on the part of requesters as well. Its a waste of my time and theirs when they engage me as a developer before they have put even the most basic thought into what it is that they want. Some of this stuff is really esoteric and I understand that I will have to help them figure out how to solve the problem. I do feel they should know what the problem is.

Programming without cookies

[Allicorn]

Programming websites that let you actually view a page without requiring a cookie is obviously hard for the folks at Salon.

Becuase People don't know what they want!

[Herkum01]

I would say the reason a lot of projects, even small ones take so much time is that requirements cannot be defined.

Compare building a house to software. Before you build a house

1. Plans are drawn up
2. A step-by-step schedule to created for the construction.
3. Contractors are brought it to complete the work as needed

Schedule times can slip but you still know where you are in terms of progression.

If we built this house the way we do software development

1. Hire all the construction workers
2. Tell them to build something.
3. At any point during construction tell them they are not doing it right.
4. After missing all the deadlines (which were made up by wants/desires of the customer) hire more workers.
5. Wonder why they cannot get the job done
6. Cancel the project after everyone realizes they don't want it anymore.

Re:Becuase People don't know what they want!

[cowscows]

I design buildings for a living, and I've dabbled in programming, and I think architecture and software development have a whole lot in common.

Your step one in "building a house" can go through all 6 of the steps that you have listed for software development. We get hired by clients, sometimes they have a good idea what they want, sometimes they don't. Sometimes what they want is feasible, sometimes it isn't. It's not unusual for even smaller projects to drag on for years, because the client keeps changing his/her mind. Many projects that cross our desks will never be built.

Many projects are not the traditional design phase ->building phase. They often overlap, and it's pretty messy.

I could go on for paragraphs with the similarities that I see between software design and architecture, but I'll save that for another post.

Software is hard AND there's lots of incompetence

[mrjb]

Yes, writing software is hard, especially writing good software. The hardest part is to make things simple, even harder is to make things simple AND flexible. The need for a thorough analysis is greatly underappreciated.

Incompetent developers tend to make things more complex than necessary. From that point on, under economic pressure, workarounds are needed to get things done. This in turn makes things even more complex than necessary. THAT is what makes writing software hard. The problem is, it is difficult to be aware of the skills that we lack. As such, a lot of programmers with a huge ego don't deserve one.

I'm not into Extreme Programming per se, but I've noticed that if multiple people look at a piece of software, chances of problems going undetected get smaller and smaller. Yes, even if you, a master programmer, show your code to a rookie, the chance of bugs going undetected will reduce. In fact, it will inevitably result in more bugs being detected before rolling them out to customers.

First Post!

[Harmonious Botch]

Two of us typed this. We thought it might be faster.

Re:Ah! The great unknown...

[alshithead]

"Ideas are not the product of labor."

Respectfully, I have to disagree. Some of my best ideas have come from pondering over a problem. Pondering can be effort. It's not like daydreaming. To think about a problem and apply logic to try and come up with a resolution requires effort in many, if not most, cases.

Software is neither "hard" or "easy"

[Bright Apollo]

Implementing a good design is usually half the battle. Creating a good design is usually the other half, but in practice, a solid design is almost always the part that gets skipped. Let me bore you with a brief anecdote.

I have a large, global project underway. User requirements are done and have been done, and we're turning those requirements into things we can code or deliver ("View a workorder", "Print asset detail", "Group revisions into single document"). Of that, we have 150 odd deliverable items, not to mention all the fit/ finish work we may have to do, and all of this barely touches on reports, security roles for users, etc.

The reason we're going to make our date, despite the 1280 discrete requirements we need to test, is that we've taken the time to look at the requirements from a few different angles and come up with a solid design plan, before even thinking about implementation. Each piece will build on another, really hard parts are identified early, blockers and such are flagged ASAP. We know things will emerge that we didn't expect, but we've got the biggest chunks identified and working together on paper. We have the flows mapped out, exceptions and variations listed, and a user group that has to sign off on every iteration of the incremental build (we're spiraling out functions and features).

The only thing "hard" about all of this is the incessant thinking about the details, and discipline required to focus on the un-fun part of software construction, i.e. the planning and design walkthroughs. The itch to code something already is growing, but delayed gratification means that when the time comes to actually write something, the design will almost certainly lead to a working, if not optimal, solution. We can refactor as we go, but it needs to work completely before it can work efficiently.

I've been following Chandler off and on, somewhat through Spolsky's references to it and some stray links around the web, and sounds like design didn't go deep enough into what it'll really take to build some of the pieces.

-BA

Once again...

[etnu]

Why doesn't anyone complain about how hard brain surgery is? Why doesn't anyone complain about how hard building space exploration vehicles is? Why doesn't anyone complain about how hard creating a successful marketing campaign is? Software engineering is difficult because it's a complex subject that takes a combination of intelligent people and training to produce good results. Just because businesses are too stupid to realize this doesn't make the problem go away. You can't throw complex projects at untrained, stupid, incompetent people and expect them to produce quality software. You can't just invent some magic formula for software development that will work 100% of the time to maximize efficiency. Software engineering is NOT manufacturing. Accept it and move on for fuck's sake.

darn customer wants and needs

[mach777]

Let me tell you why it is hard.

A person needs to cross a river, and so believes a bridge needs to be built.

What he should be doing is ask: "I need to cross this river, can you help me?"

Instead, he asks someone to build a bridge, a bridge of this and this dimension so his particular vehicle can pass, and a bridge with this and this feature because "it would be nice if..". He also wants a bridge that looks in this and that way, because since he is paying for the bridge he wants it to look the way he wants it to.

First he was crossing a river, he is now building a bridge. Will the bridge help him cross the river? Who knows? Maybe a bridge like he imagines can't be built, or only be built poorly.

Its the same thing with software. People want the strangest things. In fact, what they WANT is the only thing they know. They don't know what they NEED.

So you get an organisation, a business, with an office that has a problem, any problem. The problem will ALWAYS occur because some OTHER process isn't working somewhere else in the organisation. You get bad data from here, and the clerk is expected to output good data out there. In the 50's I bet they wanted more filing capability or better typewriters to solve these problems, in this day and age they want IT. They want a system!

So instead of using excel, notepad or even a piece of paper like any other sane human being would to keep track of the information, a yell echoes down the corridors: "We need software to support our business."

So some programmers are hired. They get a description of the problem, which isn't logical in the first place, and they are expected to solve it. Their tool, software, is built using a logical language. It is used to describe the data, and solve the problem by adding a few flows for that data during certain conditions.

So the programmer (or bridge builder) sits himself down. The first 10% of code/thought he outputs is usually all that should be done about the percieved problem. That is, a description of the Need.

The next 90% of coding is about the programmer trying to coerce a logical language around non-logical flows and non-optimal solutions, hammering a square button into a round whole, with GUI's, buttons and special extra functions for special extra cases, and those extra wants on top that really describe other problems.

So we will end up with a mishmash of buggy code that describes the wants of the customer. The Want to solve a problem that should have been solved with organizational changes, or changes in the work processes. But hey now everyone is happy again, software is supposed to be a bit buggy, the organisation is obviously still working non-optimally (software can't fix that), but at least the clerks now have webpages to input the bad data in as long as the servers are up.

Optimally, the programmer (yes the programmer) should look at the problem, trace it down the whole organisation, yea trace the customer Want to the REAL problem and the REAL need, proceed to make the organizational changes and be done without an IT system at all.

We all know that won't happen, but that is what should be done. Don't expect a logical function (software) that describes a non-optimal situation, to function in an optimal way.

Software doesn't work that way, thats why its hard.