Slashdot Mirror
What Makes Software Development So Hard?
lizzyben writes to mention that CIO Insight is running a short piece that takes a look at why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises. From the article: "I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem."
Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.
Anyone can write a book. But, writing a book that's *GOOD* and that people will want to read is an entirely different matter.
Computer software is no different.
Good software takes time to develop. There is sometimes a tendancy to set unrealistic goals and when they aren't met the people in charge feel let down.
First, the people who make the technology decisions often don't have the required technical know-how, and have a terrible tendency not to listen to people who do. OTOH, the people who have the technical know-how have absolutely no idea how to write a business case. Thus, there's usually a disconnect between the people who understand the business requirements, and the people who understand the technical requirements. Vendor loyalty has been known to be a sticky issue as a result.
Second, there's always a problem getting a bunch of talented, egotistical (ok, so not all software developers have ego problems...) quirky, eccentric and generally difficult people to work toward a common goal. The common analogy to being a successful director/manager of a software project is to that of "herding cats". My experience has been that business types don't react well to the often-emotional developer types, hearing the emotional outburst, but ignoring the content of it. Developers would do well to learn some more social skills, and director/manager types would do well to listen better.
mandelbr0t
"Please describe the scientific nature of the 'whammy'" - Agent Scully
It isn't. Haven't you seen them, all they do it wave their hands around and look silly. That's the point. Everyone thinks it's a breeze when it isn't, so everyone underestimates everything.
practical application isn't so. Requirements change, clients see something and want something different. You work a month on something and find out it isn't working anywhere close to the way it should. Tasks are more complex then people thought at first, the list goes on. It is a complex thing with infinite ways to do things. It isn't any easy process.
The reason it's hard is because we are doing the impossible, at least if you had asked someone 50, 20, or in some cases even 10 years ago. Solving previously impossible problems requires quite a bit of innovation and novel thinking. But, it's better than it was decades ago, even if it is difficult. What we need to remind people is that even with fast computers, the problems are still difficult to solve.
Also, automating tasks that we used to perform ourselves is also difficult. It's one thing to walk and chew gum, but another thing entirely to precisely describe the mechanics of doing so. (One is very easy, the other requires creating precise models of how things work, a very hard problem).
What Makes Software Development So Hard?
Simple. The current state of technology in regards to human imagination makes it many times easier to think up of good ideas or improvements for computer software than it takes to create or implement them. Thus the requirements for developers are often made by the ones with the ideas with no idea of the costs (in terms of time or money) involved in making those ideas a reality.
Joe Sacks says "Gee, it would be a great idea of our flight tracking program also tracked fuel usage and recommended fuel loads to minimize weight on subsequent flights". See? It only took Joe all of about three minutes to think that up. He figures that a reasonable amount of time to implement this idea is one or two days. After all, those "computer guys" are pretty smart and can "do anything".
Two days later Joe receives a report that the "project" is over budget (needed to check with the FAA on this one, that cost money) and overdue.
Sure, you could just say that these problems need good project management. But seriously the problem is that these "problems" are not considered worthy of project management. It only took Joe three minutes to think the idea up...why should he hire a project manager for *THAT*. It's not like he's building a new hangar.
Spoken like someone who hasn't ever been involved in the development of a large software project.
Large projects bring problems with them that aren't noticeable on small projects. The working set of my project is around 10 gigs, most of which is code and text files. The tree changes quite frequently, and syncing to that tree is painful.
What makes it more painful is when the tree is broken. So we had to develop tools to help ensure that the tree isn't broken, and that we have a way to tell what the last known good submission was.
There's performance issues related to the source repository, because no matter what repository you're using, they all have issues when you have 200 people working in the same place at the same time. (This is true of virtually any database application).
I currently have no clever signature witicism to add here.
When you ask a carpenter "are you done ?", almost anyone can verify the answer just by looking at it.
When you ask a programmer "are you done ?", it is difficult to verify the answer.
There are so many ways to accomplish a goal in software that you need a number of very expensive controls in place to validate the result. Very few managers know how to do it and very few C*O's are willing to spend the money to put in place even the minimal control measures even though I can say in every case where ther were good controls put in place the benefits far outweighed the costs.
This leads to a plethora of problems that I won't repeat here but I know too well.
Eventually everything breaks down when the C*O's start making technical decisions that they are not qualified to make and it's all downhill from there.
There's a whole industry of dev tools out there trying to convince management that SilverBullet will make your software come out on time, or make your crappy programmers more productive. I don't think so.
What we really need is to promote the concept of the CodeSmith like a balcksmish, silversmith or whatever, coding is a skilled artisan occupation. Instead of trying to keep good coders coding, most organisations try promoting them and making them managers. Eventually you're left with just the dregs coding.
Engineering is the art of compromise.
When you've contracted an engineer to design a bridge, and you want to make several large-scale changes to that bridge, the engineer will come back and say "If you want these changes done, this is how long it will take. If I don't have that much time, I can't make the changes".
In fact, I'd go a step further; software developers tend to say "This is how long it will take to make the change, and this is how long it will take for me to hack something together." Bridge engineers don't say things like that. They don't put that "hack something unsafe together" option out there on the table, and neither should we.
I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.
Software design and development is more akin to an architect designing a building rather than the more common analogy of building the building once it's designed. Except that software developers are often burdened with requirements that any architect who valued his license would reject. For example, management often dictates which parts are to be used (ie. "We're going to use MS SQL Server as the database engine."). What architect would design a skyscraper after having been ordered by the client to use pine 2x4s instead of steel beams, or design a 1-story residential house under the requirement that he use titanium box beams instead of 2x4s? And then there's what the article notes at the top of page 3: software requirements change right up to the day of release. When an architect goes to design a building, the requirements are fixed before he starts. Square footage, height, number of floors, number of people each floor has to accomodate, number of elevators, number of toilets needed on each floor, how high the ceiling of the lobby floor will be, all that's fixed in stone before any design work begins. Sometimes that requires back-and-forth between the client and the architect to get everything clear, but it all happens before major design work begins. No architect's going to design the foundations of a building until he knows exactly (or at least very very closely) how much mass is going to have to sit on those foundations and how much horizontal shear they're going to have to handle. If the client can't decide what he wants, the architect just goes "Fine, call me when you decide what you want.". And even when this kind of analysis is done, software requirements change constantly after design's started. See above, no architect's going to accept the client changing the number of floors or the square footage of floors without also agreeing to a complete redesign to accomodate the changes with all the delays and additional costs that requires. If the client didn't like it, the architect would just hand the client the work to date and tell him to have a nice day. And no there wouldn't be any refund of money, the architect's held up his part of the deal as best the client will let him.
And then there's another parallel: architecture is only half engineering, the other half is art. Every building is different, and it's accepted that the architect's going to have to have time to come up with the parts that aren't off-the-shelf standard. There isn't a single standard floor-plan for a single-family 3-bedroom house, there aren't any rules that can be mechanically applied to create a floor-plan, and it's accepted that the process of creating a floor-plan is more creative than anything else and people without the knack for it just aren't going to be very good at it. And that on the next single-family 3-bedroom house much of that work's going to have to be done over again because the new client doesn't want the same thing the previous client wanted.
Exactly. Anybody who thinks it's easy to complete a large software project without problems has obviously never worked on real software. Secondly, anybody who asks why it's so hard to develop good software has never worked on a large software project. You can say a lot of stuff about following processes, and doing testing, but until you're working on a large project, with lots of different developers, with time and money constraints, you don't know what you're talking about.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
What makes software so hard? The enormous complexity of the software constructions.
Why is it so complex? Because it's so EASY to build it.
Example: Pre-computers, the moment-to-moment computations necessary to run an automobile engine were performed by mechanical devices, mainly the distributor and carburator. Every term in every computation was manufactured as a number of physical components, several of which are moving parts.
For instance: The RPM input to the spark advance was computed by two weights on pivots, with springs and stops, rotating a sleeve on a shaft. The shaft was driven by the camshft through a gear and the sleeve carried the cam driving the contact points or (in an electronic ignition) the starwheel that passed the sensor coil. Adding this computation (compared to no RMP spark advance) added five moving and four stationary parts, to be assembeled, and a test stand the volume of an office cube to test the result, and allow a worker to adjust the constants by bending the spring supports with a screwdiver.
In software this computation can be done by PART of ONE line of code. (In real engine controls it's actually done by more - mainly so the computation can be more complex and thus better approximate ideal running conditions.)
Software changed the game completely: When adding a piece of a compuation requires a moment of thought, a minute with a text editor, and issuing a compile command (plus whatever testing is necessary to convince yourself you got it right), rather than months of an engineer's and draftsman's time, manufacuture of dies, lab work to check the result, repeat through three layers of departments (to prove it can be done, to prove it can be done reliably, and to prove it can be manufactured affordablly), the amount of work and time to implement one bit of complexity reduced drastically.
The result was that the amount of complexity that can be afforded rose in proportion. Given that the proportion was hysterically large, the amount of complexity handled by each person became enormous.
Unfortunately, programming is NOT formulaic. Portions are - and as they are identified they are rendered into algorithms and software is written to perform them. The result is that the part people work on is ALWAYS the part that is "fuzzy" and difficult to formalize.
Programming consists of rendering a set of requirements into a correct specification for meeting the requirements. (The reset is automated.) This is not an easy task - and it gets more difficult with increasing complexity of function. Unfortunately, methodologies for performing it have remained in a catch-up game: The better the tools, the more complexity a worker can handle. The more he can handle, the more he is assigned.
To quote McClary's Third Law of Computer Technology: Software complexity expands to exceed the capability of any software development methodology.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Well, if you wanted to get really technical about it, the conductor's work is mostly done before the performance, so that any reasonably proficient orchestra would be able to do a reasonably good run sans the conductor.
Conducting, however, is a lot harder than it looks, especially during rehearsal. It requires the simultaneous processing of the score (possibly twenty plus lines going on simultaneously), time (not so easy once you get into funky metric changes), expression (precisely what the conductor is all about) while keeping track of mistakes that happened during the piece even though not stopping immediately. It's extremely easy to see when a conductor is inexperienced, boring, or hasn't looked over their music properly.
For conductors of student groups, they also have to keep the members of the ensemble engaged through tasteful storytelling while not completely going off tangent, they must be extremely creative in figuring out how to get their point across, they must be careful not to over-rehearse a section, etc etc etc.
Ask any marching band student turned drum major. Being a good musician does not mean you'll be a good conductor, and a more generalized notion is the core meaning of most of the other analogies offered by Slashdotters around here.
In that case, you have essentially proven that you have no experience with large software projects. Even if everybody is qualified for the job they are hired to do, and are enthusiastic about the project, and doing the things they are supposed to do, things just don't work out right at first try. Or second try. Or third try. Or... you get the picture. And then, when that problem's finally done with, you have 7 new ones, where at least 3 of them are total show stoppers.
It's of course easy to start pointing fingers at people. That guy is an idiot. The management has no idea what we are doing. Writing documentation is killing my productivity. I had to rewrite this assholes code since he didn't follow my favourite bracing style. And so on... The point is, the guy you call an idiot probably knows three hundred times more than you about fluid dynamics, which is what all the project is about. Management isn't supposed to know what you are doing, they are supposed to handle budgets, equipment, hiring, firing, etc. And the last two are kind of obvious...
Software development is hard, because it's not a solved problem. You don't build software the same way you build buildings. There are no rigid rules to follow, no best practices that can be universally agreed upon. The purpose of each new software project is to solve (a) problem(s) that has never been solved before. And because of that, there are great uncertainties involved. You can guesstimate a lot of parameters, but eventually, some of the unknowns are going to bite you in the ass. (As Rumsfeld said: There are known knowns...)
Not exactly an easy person to work with, are you?
But while stereotypes persist that programmers have no people skills, you forget that many business people don't either.
Just ask yourself: how many effective, people-oriented bosses have I ever had? If your answer is "not many", you're not alone.
I've been software engineering for over a decade. These are a few observations I'd like to share with managers:
--- "We've always been at war with Eastasia."
...that software development needs to have a team of programmers who are talented and competent, and a project leader who knows his people and can effectively lead and manage both the team and the project... and a corporate culture which doesn't treat its people distrustfully like a bunch of indentured servants, then they can write good software and get it done in reasonable amount of time. Getting and retaining a team of very good talented and experienced programmers is very difficult and expensive however since most development firms are cheapskates and not only do not pay enough, but treat their people like dirt, binding them under all kinds of bullshit NDA and non-compete contracts, poor work environment with noisy cubicles or "bullpen" office areas, not empowering them with latest technology workstations and tools, etc. The software development company that teats its programmers golden is the rare exception... the exact opposite "sweatshop" like I described above is generally the norm, as I've seen it in my 20+ years in the biz, hence you usually get kids right out of school as entry-level positions doing the bulk of the development. The best cream of the crop programmers have mostly all quit programming years ago and now are all sitting in management positions and programming no longer.
A good orchestra conductor who is in front of a bunch of rank beginner inexperienced musicians will not be able to make very good sounding music. You get what you pay for.
The thing that orchestras have that most programmers don't is a program design (the score) that they're working with their peers to develop (play) from. The composer knew what he wanted. Most users don't.
The Spoon
Updated 6/28/2011
Simultaneous management of multiple requirements (possibly 5,000 competing business requirements competing simultaneously); time (not so easy when the scope of the project is creaping, but the deadline isn't, or the estimating process was off, or the requirements have changed.....) while keeping track of 1,000 action items, issues, defects and delivery dates at the same time. It's extremely easy to see when a Project Manager is inexperienced, or undertrained, or bored, or being overworked into an early grave!
For project Managers there is the ongoing need to maintain team cohesion, and team spirit, and still get the code out the door - tested and working perfectly, on time and under budget.
Like musicians, being a good programmer doesn't mean that you'll make a good Project Manager. Some of the best programmers out there would be dead in the water if put in charge of a real project, for no fault of their own, other than being totally unequipped to run a project.
As for delivering on time and on budget, it is a well known truism that the first 90% of the job takes 90% of the budget, and unless you are REALLY lucky, it's likely that the other 10% of the job will also take 90% of the budget....but I digress.
Will those of you who think that you know what you are doing, get out of the way of those of us who know what we are doi
Herding cats is hard because you are using the wrong management technique. You herd cattle (and sheep and goats and pigs etc.), you do *not* herd cats. Cats, you put them in the general area of the mice and let them do what they are good at. Micromanagement of cats is a losing proposition.
putting the 'B' in LGBTQ+