What Makes Software Development So Hard?

lizzyben writes to mention that CIO Insight is running a short piece that takes a look at why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises. From the article: "I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem."

Comments

good question

[macadamia_harold]

What Makes Software Development So Hard?

Why is conducting an orchestra so hard?

Re:good question

[jonnyelectronic]

It isn't. Haven't you seen them, all they do it wave their hands around and look silly. That's the point. Everyone thinks it's a breeze when it isn't, so everyone underestimates everything.

Re:good question

[Anonymous Coward]

Why is conducting an orchestra so hard? According to Kurt Masur it's becasue some musicians behave like a spoiled five year old brat. Of course that's only one opinion...

Re:good question

[xs650]

Why is herding cats so hard?

Re:good question

[Ambush Commander]

Whoever modded this flamebait doesn't have their sarcasm-detector tuned properly today.

Re:good question

[jZnat]

And whoever modded it insightful clearly knows shit about music...

Re:good question

[Kumiorava]

I think you are close to the truth with this analogy. Conducting an orchestra isn't that hard when all musicians actually are professionals and thoroughly trained to work under your supervision. Big problem is that majority of the so called software industry participants are not skilled enough to actually perform the task they need to. I bet conducting an orchestra is difficult if the musicians play badly to begin with. Finding project full of skilled people is a major problem and there is no way you can see right away if someone is doing their job well.

On smaller projects my experience is that there are projects that I can count on going well, because I assign certain people to them. Other projects with double staffing does not do to job nearly as well, even when the task is half the size. This does not change (significantly) over time and is not related to education.

To answer the original question why software projects are so hard is that similarly to orchestra one bad player can ruin the show. Either that being the boss who has team meeting every hour, designer that doesn't know what is needed, customer that cannot give direction or software engineer that hides bugs in the code and causes several weeks of debugging for the team. Once the orchestra is filled with professionals I believe there is nothing inherently difficult with software engineering. Those projects happen, but it's rare.

Re:good question

[__aaclcg7560]

They're like programmers who can't agree on where to go out for lunch without doing a probability analysis of getting food poison.

Re:good question

[Ambush Commander]

Well, if you wanted to get really technical about it, the conductor's work is mostly done before the performance, so that any reasonably proficient orchestra would be able to do a reasonably good run sans the conductor.

Conducting, however, is a lot harder than it looks, especially during rehearsal. It requires the simultaneous processing of the score (possibly twenty plus lines going on simultaneously), time (not so easy once you get into funky metric changes), expression (precisely what the conductor is all about) while keeping track of mistakes that happened during the piece even though not stopping immediately. It's extremely easy to see when a conductor is inexperienced, boring, or hasn't looked over their music properly.

For conductors of student groups, they also have to keep the members of the ensemble engaged through tasteful storytelling while not completely going off tangent, they must be extremely creative in figuring out how to get their point across, they must be careful not to over-rehearse a section, etc etc etc.

Ask any marching band student turned drum major. Being a good musician does not mean you'll be a good conductor, and a more generalized notion is the core meaning of most of the other analogies offered by Slashdotters around here.

Re:good question

[larkost]

You are close, but the real piece you are missing is that most of a conductors work goes on long before you see them up on stage conducting. The real work is in music selection (you have to know your orchestra and what its strengths and weaknesses are), properly managing rehearsal schedules (what groups should meet, and what they should work on), choosing the right players and the right section leaders (the latter is both a delegation and a political art), and the actual concert conducting is much more show than anything else (unless things start to go wrong).

I know a player in the Philadelphia Orchestra and he tells me that they largely ignore the conducting that happens during a performance (granted that conductor's stick work is impossible to precisely read). They know how he wants the piece played because of what they did in rehearsal.

Oh... and I do have the experience to comment on this, I was in orchestras for 12 years.

Re:good question

[jdray]

The thing that orchestras have that most programmers don't is a program design (the score) that they're working with their peers to develop (play) from. The composer knew what he wanted. Most users don't.

Re:good question

[JoGlo]

Wow,sounds a lot like running a project!

Simultaneous management of multiple requirements (possibly 5,000 competing business requirements competing simultaneously); time (not so easy when the scope of the project is creaping, but the deadline isn't, or the estimating process was off, or the requirements have changed.....) while keeping track of 1,000 action items, issues, defects and delivery dates at the same time. It's extremely easy to see when a Project Manager is inexperienced, or undertrained, or bored, or being overworked into an early grave!

For project Managers there is the ongoing need to maintain team cohesion, and team spirit, and still get the code out the door - tested and working perfectly, on time and under budget.

Like musicians, being a good programmer doesn't mean that you'll make a good Project Manager. Some of the best programmers out there would be dead in the water if put in charge of a real project, for no fault of their own, other than being totally unequipped to run a project.

As for delivering on time and on budget, it is a well known truism that the first 90% of the job takes 90% of the budget, and unless you are REALLY lucky, it's likely that the other 10% of the job will also take 90% of the budget....but I digress.

Re:good question

[complete loony]

We did a lot of outdoor gigs with our school band, my mum (the music teacher) would stand to one side and perhaps wave one hand under folded arms while some random person from the audience waved their arms up the front. When you have to read music and watch the conductor you get pretty good at watching something move in your peripheral vision. Generally we only needed a conductor for complicated speed and key signature changes, sometimes to know when to come in after a long break.

Re:good question

[nigelo]

>As for delivering on time and on budget, it is a well known truism that the first 90% of the job takes 90% of the budget, and unless you are REALLY lucky, it's likely that the other 10% of the job will also take 90% of the budget....but I digress.

This is why you should do the hard parts first.

Re:good question

[Bush Pig]

Frank Zappa reckoned it's also because many of them are barely competent drones (he had a bad experience with the LSO). A bit harsh, but probably fair.

Re:good question

[JoGlo]

Agreed, but not always possible, as the hard parts often fall into view AFTER you've done a lot of the work

"Who'd have thought that little bitty piece there would have caused so much trouble?"

Or, as a cartoon I have on the wall of my cubicle says: "We decided to save the total screw up part for the end".

I also once had an entirely approrpiate wall mounted cartoon that showed a really complex diagram of how the application was to work, with a gap in the middle, and the PM explaining the gap as

"Here, we're expecting a great leap of faith to fill in the missing detail."

It never does, of course.

Has anyone here not read Yourdan's "Death March"? If you have any interests in projects and why they fail, then get hold of this book for a good read.

Re:good question

[marcosdumay]

That's the GP's point, it is hard but looking at it it seems easy. That repeats on just about everything, but we know about software development, so we know it is hard.

Re:good question

[LucidBeast]

They've made managers out of those who knew how to code?

Re:good question

[Maxo-Texas]

We use RUP and try to put the hard 10% first.

It seems to help. Esp, when it turns out that the "hard" 10% is really the "impossible" 10%.

Problem with waterfall is you find out those impossible things on the back end after investing 10's of millions.

That being said, I have no experience with projects larger than 10 developers and 20 people total and about 12 months long.

I enjoy being a project manager more than I enjoy being a programmer under the post-sox constraints. PM is about talking to people and solving the scheduling puzzle at my level.

Re:good question

[fireboy1919]

FYI, I was in band in High School and college, and am still active as an alumnus of Kappa Kappa Psi, the "band fraternity" (featured in the movie Drumline, btw).

Conducting, however, is a lot harder than it looks, especially during rehearsal.

For conductors of student groups, they also have to keep the members of the ensemble engaged through tasteful storytelling while not ...

This is almost always only true for band directors - people who have received training as teachers. Teaching!=Conducting. I'd go so far as saying that their very different things. In fact, conducting to teach is different from any other kind. It's must less subtle and requires much less talent on the conducting side of things. This is also true (but usually to a lesser extent) of drum majoring because you can't see subtle hand movements when you're 50 yards away from the drum major, as often happens in marching bands.

Student conductors/drum majors hardly ever have to deal with entertaining the troops or with mistakes (unless you've got a wierd band director that goes in for that kind of thing). The few times that there are student "leaders" are in larger groups where the students are more professional - i.e. student orchestras, universities, etc. They *may* get to deal with interpretations of the music that they want to change, but that's not nearly the same thing. It doesn't mean compiling a huge list of things they want differently. Besides, a good orchestra will respond to a conductor so that he doesn't have to talk to get his point across.

Ask any marching band student turned drum major. Being a good musician does not mean you'll be a good conductor...

Of course, most of the people who actually end up doing this generally have a talent for it. Of all of the Drum Majors I've known well (off the top of my head, I've known 15 very well - watched them practice,hung out, etc.), only one of them couldn't do almost as well by listening to the tape *once* or five minutes looking over the music as they could after doing five or ten performances.

Of those drum majors, only three of them could almost as well with their primary instruments (two were trumpet players, and one was a percussionist). I can only conclude, then, that being a drum major and keeping track of the entire score is significantly easier than playing instruments and keeping track of a single note at a time for people who have the knack of it.

It should be noted, in fact, that at a professional level, musicians are generally so good that they don't need a conductor much (in the exceptionally good quality sound stages we've got today), and the conductor is so good that he needs very little preparation for any specific piece. Of course, the conductors are generally so good that having them adds that extra little bit of cohesion that makes a good performance into a great one.

Being that good is really hard - not because they've got a lot to keep track of, but because they can always get it right the first time.

I think that security is harder because virtually no one gets it right the first time...or the second...or the third...

Re:good question

[JoGlo]

I enjoy being a project manager more than I enjoy being a programmer under the post-sox constraints.

I wasn't trying to say that you couldn't turn a good programmer into a good PM, just that too often that step pushes the person to his/her level of incompetence.

Last big project I worked on had 186 staff, including about 45 testers, and the development effort was split between 5 teams of about 20 each, all reporting to a Progject Manager.. On that project, I was a Project team Leader, with the lives and work of 21 programmers, following a release plan that HAD to work like clockwork. That was over 5 years ago, and since then I've moved into process and quality, which is MUCH more fun.

In the process (no pun intended), I've learnt a LOT about how project management is handled, and how it should be handled, and have taken part in the journey for a BIG company from CMM Level 1, all the way to CMMI Level 5, and have seen first hand just what is achievable by teams that embrace good process, and the improvements in quality, accuracy and (importantly) the estimating process when good process and metrics are adopted.

Re:good question

[Ambush Commander]

Thanks for the perspective into the professional world of music! I am a student musician, so my account is what I see, and probably not representative of the larger world out there, even if it does jive well with our perspectives on development.

I think that security is harder because virtually no one gets it right the first time...or the second...or the third...

Well, it depends on your definition of "the first time." The first time of what? Writing any applications at all? Writing applications for that specific domain?

Re:good question

[Maxo-Texas]

I wasn't trying to contradict you. Most of the developers I actively dislike project management and do not want to move into the role. I happen to like it - and you usually do better at things you like.

I liked being a programmer when I was the hero and not doing 15 hours of paperwork for 5 hours of coding. I enjoyed the "AHA" feeling. The last year I programmed at a major corporation- I did 3 months of programming and 9 months of paperwork, testing, meetings, anything but coding. It was all because of SOX. Very toxic the way it was implemented there.

Re:good question

[nigelo]

It is certainly true that if you don't know what it is supposed to do, it'll be probably be very hard to figure out how to do it.

But don't overlook that it is human nature to say 'I don't really know how this works, but how hard can it be?' (to be an orchestra conductor...) 'No matter, let's do the parts we know, and get them behind us' thus nicely taking up most of the time allowed, leaving little or no time for the little-understood-but-hope-it's-easy 'then-a-miracle-occurs' piece that never got sized or included in the estimating/planning, etc. This (rightly) leads to the claim that 'the hard parts didn't fall into view until after most of the work was done'.

Again, do the hard parts first - if you don't know what it does or how it does it, consider it hard. If you go over budget on the early stuff, at least it was on the hard parts, and maybe your team can subsequently pull off the implementation of the easy parts in less time than you originally thought, especially now that they've learned how to do the hard stuff.

Either way, it'll be easier to deliver the latter parts under the pressure at the end of the project, particularly if you've overrun budget/schedule.

Re:good question

[Ngarrang]

Why is software development so difficult? I know the answer to this one!

The answer is...management. How can anything get written when the software requirements are changed one a week and totally re-written when the new CIO is hired so he can make his presence known?

Re:good question

[EtherMonkey]

Conducting an orchestra isn't that hard when all musicians actually are professionals and thoroughly trained to work under your supervision.

But it's very hard when all the musicians sit 10,000 miles away, they don't natively speak or understand the same language and concepts as you and there's a 10 hour time difference. I guess its sort of like conducting an orchestra full of blind musicians, but with less harmony.

Re:good question

[Anonymous Coward]

Conducting, however, is a lot harder than it looks

It doesn't look that hard.

Let's see...
- lots of hand waving ... check
- reading 20 lines ... check
- time changes (ahem, "rescheduling") ... check
- expressions ... check (statements, too!)
- seeing mistakes but not saying anything ... check

It sounds like being a good conductor is about as easy as being a bad project manager.

Re:good question

[xiphoris]

The thing that orchestras have that most programmers don't is a program design (the score) that they're working with their peers to develop (play) from. The composer knew what he wanted. Most users don't.

True, but a better analogy in this situation is the customers of the orchestra -- ie, the listeners. This might be a concert hall, or the person who commissioned the piece to be played (perhaps more often in older history).

The conductor is still accountable if the piece doesn't sound like what the customer (audience) expects.

Re:good question

[iminplaya]

This is why you should do the hard parts first.

Au contraire (Tish, that's French!) When I took any tests in school, I always got the easy questions out of the way first. Image not getting to questions you could have answered when the time runs out and you are told to put down your pens. And getting some of the easy stuff done first might make the more difficult stuff a little less so. It also gives a moral boosting illusion of progress.

Re:good question

[loki_tiwaz]

To put it succintly, the problem comes from the fact that managers are basically generalists, and programmers are usually as specialised as you can possibly be.

If there could possibly be any other argument for the augmentation of the human brain I could not think of a better one than software project management. Comparing neural augmentation to taking drugs to do better at high level sports is a dumb comparison but sadly one which washes well with the majority of the voting public who are by definition 50% below average.

Re:good question

[vtcodger]

***Why is conducting an orchestra so hard?***

Good point. But it's much worse than that. In software development, you're expected to conduct the orchestra and get the performance right -- while compsing the music on the fly -- without rehersals -- while responding to real time input from the audience -- and likely with one or two instruments in the orchestra whose characteristics are largely unknown. Plus which, the acoustics of the theatre are lousy, the roof leaks, and the guys on the light and sound boards are incompetent if not malicious.

Re:good question

[Matje]

The reason you attack the hard problems first is that you have less money invested if the hard problems turn out to be unsolvable (within your time/money/skill scope).

The big difference between a school test and a software project is that you only need to answer as many questions correctly as necessary to get a passing grade (you can thus skip the hardest questions), whereas in a software project you must solve the hard problems if you want to complete the project.

Re:good question

[swiftstream]

Not always, by any means--things often change at the last minute. Just an example: Yo-Yo Ma played with some students at Patrick Deval's inaugral gala last week (http://news.bostonherald.com/localPolitics/view.b g?articleid=174458). One song they played he finished arranging at 2:00PM the afternoon of the performance, faxed it over to where the students were waiting, and rehearsed with them at 3:30 PM. They performed it a few hours later. Oh, they knew what they were playing in a vague sense before then--they knew the song--but they didn't know the details of what notes they were expected to play (the arrangement). It's not so different from software development after all--in fact, that was the third arrangement of the same song they had been given. Changing requirements, anyone?

Re:good question

[Saxophonist]

This is almost always only true for band directors - people who have received training as teachers. Teaching!=Conducting. I'd go so far as saying that their very different things.

I'd go further to say that Teaching != Conducting != Rehearsing. My undergraduate training was in music education (and composition), and I have conducted student bands and choirs from beginning level through high school as well as a lower-level orchestra and a pit orchestra of some high school musicians (including the chorus on stage) and some professional musicians. Of course, as part of my undergraduate training, and since then in professional playing and in graduate performance study (I am a doctoral student in performance now), I have played for very fine conductors as well as very poor ones. While the actual stick technique does vary in quality, the main difference between a good conductor and a bad one tends to be in rehearsal technique. Particularly egregious is the conductor who spends time in rehearsal trying to "fix" ensemble problems that are in reality conducting and/or score preparation problems -- this issue is the very first one covered in any decent conducting curriculum, but for some reason, this lesson does not stick with some (or, the conductor has no idea how to prepare for rehearsal, which is even worse).

In fact, conducting to teach is different from any other kind. It's must less subtle and requires much less talent on the conducting side of things.

I disagree. Students need to be taught how to watch a conductor at first, but once they pretty much get it, the quality of conducting can dramatically affect their performance. A large part of conducting is knowing how to get the response desired from the performers without saying anything. Obviously, doing so saves rehearsal time in some areas, allowing more attention to other areas.

Of all of the Drum Majors I've known well (off the top of my head, I've known 15 very well - watched them practice,hung out, etc.), only one of them couldn't do almost as well by listening to the tape *once* or five minutes looking over the music as they could after doing five or ten performances.

That probably says more about the music that the drum majors are conducting than about the abilities of the drum majors, though the drum majors would have to have some ability to pull that off. Most (but certainly not all) marching band music tends to be metrically straightforward; after all, the band has to be able to march to it! Further, if the drum major does not have any responsiblity for rehearsing (as opposed to conducting) the ensemble, they may not need to know much more than the time signature, tempo, and basic dynamics. Some drum majors do have more responsibility for rehearsal, though, which makes things more challenging.

It should be noted, in fact, that at a professional level, musicians are generally so good that they don't need a conductor much (in the exceptionally good quality sound stages we've got today), and the conductor is so good that he needs very little preparation for any specific piece.

On the contrary, the conductor has to be absolutely prepared; it's just that the preparation must happen away from the ensemble. If the conductor has already prepared the work (quite common with standard orchestral repertoire, for instance), then, sure, it's a matter of recalling what is already learned, just as it would be for the performers in such a situation. Some practice may still be needed as a refresher, but the amount of time required is generally much less. Of course, the basic conducting skills, just like the skills to play an instrument, rarely change from piece to piece, so an exceptional conductor will apply his or her talents to produce exceptional results.

Of course, the conductors are generally so good that having them adds that extra little bit of cohesion that makes

Re:good question

[TheLink]

That's only fine in the cases where an incomplete project is acceptable.

And even if an incomplete project is acceptable, if the hard parts are _required_ and not optional, it's better to figure out early whether the team can do them or not, or you need outside help, or the project needs to be changed/cancelled.

Re:good question

[dbrutus]

A development project is not a test and it doesn't IMO make sense to use the same strategy for both. Most "do the hard part first" methodologies focus on correct scoping and quickly building prototypes of all the parts. That way you get the prototypes of the easy stuff done quick (a day or two) and the hard stuff taking a couple of weeks and in the end you have a very basic prototype and a very good understanding of which program sections are going to be the problem children. There is real progress, you have a basic prototype out the door after a couple of weeks but more importantly you have much less illusion about what's easy, what's hard, and you might just spot what's impossible before people have to get fired over the issue. Lather, rinse, repeat, you keep making more and more detailed prototypes until what you have starts looking like an alpha, then a beta, and then you ship.

Re:good question

[iminplaya]

Thank you. You spelled it out the way I intended for it to be. I wasn't advocating that you could leave stuff undone. I'm just saying you need to avoid tunnel vision of focusing on only one thing.

Re:good question

[Fulcrum of Evil]

No, do the necessary parts first, and the risky parts of those before the safe ones.

Re:good question

[Anonymous Coward]

I guess that makes me a jazz musician!

Software developers?

[Anonymous Coward]

Software developers?

Re:Software developers?

[lucabrasi999]

Software developers?

End users?

It's not hard

[Oz0ne]

You just have to have a good system and management if it's a large project. That's all.

Re:It's not hard

[daVinci1980]

Spoken like someone who hasn't ever been involved in the development of a large software project.

Large projects bring problems with them that aren't noticeable on small projects. The working set of my project is around 10 gigs, most of which is code and text files. The tree changes quite frequently, and syncing to that tree is painful.

What makes it more painful is when the tree is broken. So we had to develop tools to help ensure that the tree isn't broken, and that we have a way to tell what the last known good submission was.

There's performance issues related to the source repository, because no matter what repository you're using, they all have issues when you have 200 people working in the same place at the same time. (This is true of virtually any database application).

Re:It's not hard

[CastrTroy]

Exactly. Anybody who thinks it's easy to complete a large software project without problems has obviously never worked on real software. Secondly, anybody who asks why it's so hard to develop good software has never worked on a large software project. You can say a lot of stuff about following processes, and doing testing, but until you're working on a large project, with lots of different developers, with time and money constraints, you don't know what you're talking about.

Re:It's not hard

[Oz0ne]

I stand by my original comment. I've worked with multi gig source trees, perhaps not quite 10 gigs. The largest project I've personally managed was 125 member team. Perhaps your situation is a different scale... I really don't think so.

Re:It's not hard

[Oz0ne]

I didn't say there wouldn't be problems. I said it wasn't hard.

In my experience the only time it becomes difficult is when people are NOT doing their jobs. That's a people problem which should be addressed outside of the scope of the software project. Yeah, it effects you a great deal, but in any large project you need to know you can count on your team.

The only other problem I've encountered with software development are when people have unrealistic expectations, despite me telling them exactly what to expect and delivering on exactly that.

Re:It's not hard

[jZnat]

10 gigs? How many different subprojects did you guys have? 20?

Re:It's not hard

[joto]

In my experience the only time it becomes difficult is when people are NOT doing their jobs.

In that case, you have essentially proven that you have no experience with large software projects. Even if everybody is qualified for the job they are hired to do, and are enthusiastic about the project, and doing the things they are supposed to do, things just don't work out right at first try. Or second try. Or third try. Or... you get the picture. And then, when that problem's finally done with, you have 7 new ones, where at least 3 of them are total show stoppers.

It's of course easy to start pointing fingers at people. That guy is an idiot. The management has no idea what we are doing. Writing documentation is killing my productivity. I had to rewrite this assholes code since he didn't follow my favourite bracing style. And so on... The point is, the guy you call an idiot probably knows three hundred times more than you about fluid dynamics, which is what all the project is about. Management isn't supposed to know what you are doing, they are supposed to handle budgets, equipment, hiring, firing, etc. And the last two are kind of obvious...

Software development is hard, because it's not a solved problem. You don't build software the same way you build buildings. There are no rigid rules to follow, no best practices that can be universally agreed upon. The purpose of each new software project is to solve (a) problem(s) that has never been solved before. And because of that, there are great uncertainties involved. You can guesstimate a lot of parameters, but eventually, some of the unknowns are going to bite you in the ass. (As Rumsfeld said: There are known knowns...)

The only other problem I've encountered with software development are when people have unrealistic expectations, despite me telling them exactly what to expect and delivering on exactly that.

Not exactly an easy person to work with, are you?

Re:It's not hard

[lymond01]

Don't quote me, but I think he said his project was Clippy....

Re:It's not hard

[mrchaotica]

no matter what repository you're using, they all have issues when you have 200 people working in the same place at the same time

You know what this means? Your project's organization sucks! I don't care how big the thing is, it ought to be separated into independent modules with well-defined interfaces, each small enough for only a few programmers to work on at a time.

If you need 200 (or even 20!) people to change the same section at the same time, you already have spaghetti code.

Re:It's not hard

[Oz0ne]

You've just described your problems, not mine. What makes you think that these problems are universal? Why don't you get it right on the first time, or second time, or third time? The only reason things don't work out is you either haven't planned them well enough, don't understand the problem, or you've failed. There's nothing wrong with failure, we all learn from it, but it's not an excuse.

No I'm not an easy person to work with. I expect everyone to give their best, and do the work they say they can do. I expect them to inform me if there are having difficulty before the deadline so that I can get them help or get them on something else while the problem is addressed.

I think what you're describing to me are failures you've experienced working in large groups. They are not universal or necessary. It does get progressively harder to manage groups as they get larger, and there are many more variables to consider, but just because JoBlowCo can't get it done, doesn't mean the rest of us can't.

Re:It's not hard

[mollymoo]

[...] just because JoBlowCo can't get it done, doesn't mean the rest of us can't.

Who do you work for, what projects have you done and what methodologies/practices/tools did you use to do what virtually no other large project teams ever manage to do? It's all well and goood saying you can do it, why not enlighten the rest of us as to how, excactly, you do it? Then we can all be great too.

Re:It's not hard

[alienmole]

Would you cut it out, you're making my BS detector overload. I agree with the OP, even the slightest bit of evidence that you've been involved with a successful large software project would make a big difference.

Re:It's not hard

[Brummund]

I have no idea, but according to his blog, he can't adjust his alarm clock. So, my guess is he's in management somehow. ;-)

( http://www.makesitgood.net/ )

Re:It's not hard

[mollymoo]

( http://www.makesitgood.net/ )

I can't decide if that's the cutest dog I've ever seen or, you know, Satan himself. The eyes! The eyes! I bet he gets the dog to hypnotise developers into working super-hard.

Re:It's not hard

[Anonymous Coward]

"Software development is hard, because it's not a solved problem..."

Software development is not a problem to be solved. It should be merely the writing of a transfer function in the form of a state machine that for a given set of inputs will produce a desired set of outputs under a defined set of constraints and initial conditions. 99.9% of software development projects do not know exactly what the outputs, inputs, contraints, and initial conditions are before they start. And these requirements are always way more complicated than they really need to be for political reasons usually, not physical reasons. So the writing of the software is not what is hard. It is the constant massaging of the code to meet the fuzzy requirements.

Re:It's not hard

[joto]

What makes you think that these problems are universal?

Because every large project I've been involved in, or have heard enough about to form an opinion of, have had these problems. On the other hand, I have never heard about a large software project that didn't experience problems.

The only reason things don't work out is you either haven't planned them well enough, don't understand the problem, or you've failed.

Again, a proof that you've never been involved in a large project. You can't plan for things you don't know about. You can't fully understand the problem before you've solved it. And in general, you can't avoid failing, when there are unknown factors at work. Again, I think it's suitable to quote Rumsfeld.

No I'm not an easy person to work with.

In that case, you are part of the problem, not the solution.

I think what you're describing to me are failures you've experienced working in large groups. They are not universal or necessary.

Then please point to at least a single large of medium-scale software project that didn't experience problems. If such a project exists, I'm sure it's well documented in either books, journal articles, or teh Intarweb.

but just because JoBlowCo can't get it done, doesn't mean the rest of us can't

For your information, the phrase "the rest of us" is usually associated with those of us who are not geniuses. And "the rest of us" certainly can't get large-scale software development done without experiencing trouble.

Ignoring this misunderstanding, I'm also a bit curious as to who you would consider good enough to put together in a team. Even if you had Donald Knuth, John Carmack, Dennis Ritchie, Linus Torvalds, and Richard Gabriel, you'd experience trouble. Perhaps especially then.

Re:It's not hard

[Anonymous Coward]

I have worked and also managed large software projects, some with more than 100+ developers.

Basic problem is this. In any school, nobody expects all students to be equally smart. Remember the horrors of grading by the curve? Once these students come into a company, skill level is expected to be identical. Say John1 Doe takes 1 week to finish a task and John2 Doe takes 2 weeks to finish another task. It is implicitly assumed that John2 Doe's task is twice as difficult. Generally people do not make the assumption that John1 Doe is smarter.

In management positions, farmed out hard work to technically sophisticated folks and not by seniority. Mundane or repetitive tasks were given to rest. Seniority determined pay scale though :-( In these cases, software development was a breeze. Software development is NOT difficult. What is difficult is for management to get a clue and stop blaming other people for their inadequacies.

Re:It's not hard

[Anonymous Coward]

He sells hairstyling software for toy dogs... see his website...

SIT DOWN DAMNIT! (there... project on schedule)

Re:It's not hard

[vtcodger]

EVERYTHING you've cited is important.

But the really important thing -- and something that is not done all that often -- is to have the design right and the interfaces documented rigorously before coding starts. Even that doesn't gurantee success. And it certainly doesn't prevent problems. But I've never seen a large project without a solid design and clearly defined interfaces that wasn't a total horror show.

And that's a problem, because just winging it can work quite well on a small project.

Re:It's not hard

[Oz0ne]

Again, I've never said there won't be problems. I'm simply against the tone that large software projects are inherently difficult with volumes of problems that are nigh insurmountable which is the tone of the opposing view in the original article and the discussion thread.

Re:It's not hard

[Oz0ne]

Nice :)

The alarm clock is the new dog. It has since been adjusted.

You guys weren't the reason I got a huge surge of comment spam last night where you?

Re:It's not hard

[Oz0ne]

I completely agree with this. You've put it better than I have.

The problem is not in developing software.

The problem is not knowing what software you're developing before you begin.
The problem is people not understanding their role in the project before they begin.

Of course this is amplified the larger projects you take on, but they are not unsolvable problems.

Re:It's not hard

[Fulcrum of Evil]

That's what you get for having one monolithic tree (I assume). You may have it set up as multiple components that can compile independently, which will help performance (you can run a build cluster that continuously builds the tree) and isolate errors.

Re:It's not hard

[Brummund]

God, no. Sorry to hear that!

(That alarm clock and management thing was just to tempting, sorry. :-)

It's complicated

[fittekuk]

Software is complicated, and obviously the larger the project, the more peices need to fit together, etc. I think a lot of it though has to do with the team involved - both the developers and management. It seems to me, there are lots of "developers" around who really have no business writing software. And even more self-proclaimed architects who really have no clue.

Re:It's complicated

[RingDev]

And there are yet even more supervisors and managers that have no business being in charge of developers.

-Rick

Re:It's complicated

[Thangodin]

There are a number of problems:

1. Clients rarely know what they want. Most software projects are designed and written for someone else's requirements. These requirements often come in the form of "I don't like that, but I have no idea what I actually want."

2. Coders are systematizers, who tend towards the asocial end of the spectrum, which is why silicon valley produces so many autistic and Auspergers's kids. Large projects require communication, and there are a lot of coders out there who may be good with machines but are lousy with people.

3. As a result, lead programmers are often tantrum throwing prima-donnas, intellectual bullies who use their position to undercut those under them and make themselves look superior. You may have to promote and fire half a dozen programmers before you fill the lead programmer position with someone who is actually good at the job. In the meanwhile, you've just lost 5 good programmers. And if you actually manage to find a good one, you'll have to pay him in blood, because he's probably worth more than your house.

3. Salesmen, who deal with the client, are professional bullshitters, who are generally wafer thin on the technical details. That's actually not a slam--this is literally their job--to sing whatever lullaby the client wants to hear. Programmers will generally learn their lesson after being burned a couple of times and will not promise the impossible. The sales people never get burned--it was the programmers who screwed up, right? Calls for heroics on the part of programmers invariably begin with someone in marketing.

So, a few tips:

1. Get someone to hammer out the details of the specifications before signing the contract.

2. Get someone with some technical knowledge to study those specs, and the deadlines, before signing the product.

3. Get a lead programmer who fights for his team, not for the himself, and who is more interested in getting things done the right way than in getting them done his way.

4. Get a project manager (preferrably female--tends to calm the more socially inept coders, less testosterone poisoning) who handles communication between the coding team and the client. Do not, repeat DO NOT allow the coder to meet the client. They will eat the client! You will have a hard time getting paid, and hiding the bones is a bitch.

Lemme Guess...

[Anonymous Coward]

A bad process makes software development hard.

Poor specs, the dreamers from marketing, arrogant programmers, feature creep, unwanted input from people outside the process, lack of caffeine, and the users...oh the god damn users!

Totalitarian dictatorship and "hello world" can be simple if you work hard enough at it.

Re:Lemme Guess...

[necro2607]

Ohhh yeah I forgot about that, programmer skill is of course based on number of lines written per hour...

Re:Lemme Guess...

[TheWoozle]

[sarcasm]Because lines of code per hour is such a *great* metric for productivity.[/sarcasm]

Technically, with certain C compilers I can write entire programs on a single "line" of code. Please tell me how productive I am, then?
PHB's who measure productivity by lines of code/hr. get what they deserve. (Clue: it isn't nice).

Here. You're welcome.

Re:Lemme Guess...

[HomelessInLaJolla]

I didn't say it was a metric of productivity. It was actually in a discussion where the fellow was claiming to have made enormous sacrifices in life, and that he had put all sorts of hard work into his job, and that he had an amazing intellect, and those things justified his salary and his position in life.

So I asked him to prove it... and the proof he offered was to count his number of lines of code in CVS from the previous year.

It wasn't my metric. *shrug* Don't blame me. I offered it as an example of "arrogance", not productivity.

It's design not development

[gilesjuk]

Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

Re:It's design not development

[MindStalker]

Well no one really teaches software design do they.

You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge. Software design isn't really taught in this manor, sure your taught how all the bridge building tools work, and even a lot of the engineering specifications. But I have yet to see the software design school that covered more than a class or two into truly how to design software. Then again, we've been building bridges for thousands of years, and designing software systems for a few decades. It does take time for these things to really get figured out.

Re:It's design not development

[kevin_conaway]

Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

The problem is:

• There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change
• The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

Re:It's design not development

[truthsearch]

The last project I was on which had requirements that didn't change at all during development was 7 years ago. And it was built right on schedule. For me it's been the ever-changing requirements which delay projects the most. The other half of the problem is usually bad estimates, but that's almost irrelevant after the requirements change.

Re:It's design not development

[korbin_dallas]

That and software is more like art than engineering, but no one wants to admit that.
Of all the Renaissance painters that ever painted, only a few became forever famous.

Same thing with coders.

Re:It's design not development

[nuzak]

We've had a couple of thousand years of practice building bridges, and bridges solve a known problem: connect points A and B when there's a gap or a river inbetween.

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

Re:It's design not development

[MeanMF]

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

That was yesterday....We'll be sending you the updated specs soon.

Re:It's design not development

[xero314]

I think you are getting at the point but in a bit of a backward way. The problem is that somewhere along the line the idea of Software Engineering was thrown out the window and replaced with ( or reverted too, depending on how you look at it) software development. If you apply the principles of Engineering to software you end up with far more stable and durable products. If we built buildings like we build software there would be millions of people dying from "unforeseen bugs" or rooms that could not be accessed because the "interface" was not clearly defined (of course you would go back and put the door in by severing an important support beam causing massive loss of structural integrity, but I think you get the point.)

Excuses like;

There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change

and;

The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

are the exact reason software sucks. Not because these things are true, because they certainly are not, but because we are not applying logical engineering principles to the problem.

Re:It's design not development

[HappySqurriel]

That and software is more like art than engineering, but no one wants to admit that.
Of all the Renaissance painters that ever painted, only a few became forever famous.

Same thing with coders.

I personally would use the term "Craft" rather than art because I have always believed that art is about "saying" something through your craft which I don't think is really the goal of software development. The truth is that anyone can be taught to be a reasonably good muscian, artist, or blacksmith and with proper motivation anyone can become great. I would say that there are two problems currently, schools focus on the wrong things (how often did you gather requirements, properly design, implement to given specifications, effectively test, or maintain a program in your school years?) and the industry seems to focus more on technology rather than skillset (does it really make that big of a difference to program in C# as compared to Java?); the result is you have someone who is trained to use a hammer designing your house, and you ask someone if they know how to drive a "Ford" when you are interviewing a truck driver.

Re:It's design not development

[scdeimos]

Mod parent up! Yes!

Re:It's design not development

[Anonymous Coward]

Thanks for making me laugh :-)! Wish I had mod-points for you.

Re:It's design not development

[Anonymous Coward]

I would have laughed, but this hit a bit too close to work.

Re:It's design not development

[CastrTroy]

But people aren't willing to wait or pay for engineered software. Windows already costs $199, and a real OS that followed engineering principles wouldn't cost $199. It would cost probably around $10,000 per copy, if it were being sold on every desktop. And it wouldn't look as nice, and wouldn't do all the whiz-bang things that windows does. You can get engineered software, just ask NASA, but don't expect that grandma will be running engineered software on her $299 Dell box any time soon.

Re:It's design not development

[d3xt3r]

The crossroads of design and implementation (development) is exactly where software projects fail or meet extensive delays. Coding to good design is easy, but coding to design that is always in flux is next to impossible. I'd say that given the current programming toolkits available today, it's impressive just how much gets accomplished.

Where architects need only worry about a single blueprint that is ethed in stone before ground is even broken, software architects and developers aren't so lucky. Software design tends to fluctuate with requests for the feature de-jour or the facade redesigns in the name of "keeping up with the competition".

Software design will always be a moving target because business demands will always require flexibilty. In the business world, service oriented architectures (SOA) have the potential to help this situation by allowing discrete applications to provide services to each other without requiring a common code base. However, SOA itself brings numerous complexities to projects but most of this should be simplied over the coming years.

The UNIX architecture is a perfect example of system designed in small pieces designed to be used in harmony. Every tool has a discrete job that it does well and can be combined with other tools to solve a problem This philosophy, when expanded beyond the bounds of a single OS instance across platforms, machines and physical locations will provide a trully beneficial programming environment to deliver the flexibility software design demands.

Re:It's design not development

[RingDev]

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

Close, but not quite. software engineering is "I want it to do blassie. Make it do blassie by yesterday." Where 'blassie' is a loose concept and the developer has to determine and define the underlying logic.

-Rick

Re:It's design not development

[Mr_Tulip]

The designing of software is much like describing the plot of a book.. by the time you've given the typist enough details to write what you want, you may as well have written it yourself. Since programmers are just the typists, they usually have no idea what the architect has in mind. When the architect / designer leaves a lot of the design to the programmers, the end result will fall short of what the designer had in mind. This is why a lot of awesome software projects are generally one or two man efforts, not designed by a team. If you're writing a novel, you don't bring in a dozen co-authors; at most, you collaborate with on or perhaps two people.

Re:It's design not development

[Anonymous Coward]

        Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

The problem is:

        * There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change
        * The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

Even beyond that... Once the well spec'd solution is put it place, it changes the business processes that surround it (and changes the problem it was put in place to solve). The business solutions that come out of software development make it much more like solving social problems than engineering. Have a look at http://en.wikipedia.org/wiki/Wicked_problems. When you look at the characteristics of wicked problems; stakeholders don't agree, endpoint not well defined..., you'll see all the characteristics of software development that make it "hard".

AC

Re:It's design not development

[StikyPad]

For the love of God! If you want us to troubleshoot your code, you're going to have to use some meaningful variable names and stop putting multiple operations on a single line!

Re:It's design not development

[0xdeadbeef]

Yeah, you're right. Kind of like how writing a novel is easy. It's creating an engaging plot with realistic and interesting characters, that's where it gets hard.

This non-insight brought to you by the discredited waterfall model, the ideology driving the outsourcing fad, and incompetent god-architects with no real programming experience.

Re:It's design not development

[955301]

Hey, that looks painfully like the notes I've taken on my marriage.

Re:It's design not development

[Richard Steiner]

That and software is more like art than engineering, but no one wants to admit that.

I suspect a lot of programmers are fully aware of that (though I would agree with another respondant that "craft" might be somewhat more accurate), but that a large number of managers (mainly those who don't have a software development background themselves) don't fully understand the truth behind the statement.

Software development is still a subjective process in many respects, and I'm not completely convinced that it's a bad thing. Sometimes the effective expression of a radical idea requires a solo programmer who just happens to have a brilliant insight or even just a vision of something.

Writing software isn't like building a bridge -- there are often a fair number of incompatible but equally elegant solutions to a give problem, and sometimes the elegance is readability and supportability rather than cute tricks in the code (which might legitmately save clocks or bytes but make also make life miserable for some unsuspecting support guy five years down the road).

Re:It's design not development

[Tim Browse]

I read once about a programmer who decided to investigate the whole 'if we built bridges like we built software' thing, and did a survey of various bridges that were built in a particular time period. Guess what - a large proportion of the bridges fell down.

Much to my annoyance, I can't for the life of me remember where I read it now - I thought it was Bentley in his 'Programming Pearls', but I can't find it in there now.

Re:It's design not development

[Richard Steiner]

Specs?

*Blink*

You didn't get my voice mail?

Re:It's design not development

[MindStalker]

As a current CS student and Network Admin I'd have to disagree about the teaching a technology rather than a skill set. This was true for a small period of time, most University CS degrees nowadays are heavily theory based and not just based on a very current technologies. Unfortunately there are many places that aren't, and I'm frequently surprised at the stupidity of several of the younger professors who seem to have gotten their Masters in the last 90s

Re:It's design not development

[cyber-vandal]

We've only been doing electronics for about 100 years and yet the failure rate is a lot lower, so it's not a new tech problem. I doubt software engineering will ever get better because there are too many incompetents doing it and too many incompetents managing the incompetents doing it and way too many snake oil salesmen selling garbage to the clueless.

Re:It's design not development

[noidentity]

Fortunately, they do still teach English (at least in some parts).

Re:It's design not development

[saltydogdesign]

That's an excellent point. If you're a bridge builder, you build bridges. If you're an auto engineer, you build autos. Fine. But a computer is essentially a devices for making virtual machines. So you take a guy, tell him how to write C, and then say, build a virtual automobile that builds bridges and can fly and keep an appointment book. There's no job title for a guy who makes one of those, and once you're done making one, you're unlikely to make another one.

Re:It's design not development

[cyber-vandal]

If civil engineers built bridges in the same way as IT departments/software companies build software the bridge would be required to be built in a week and would take 5 years. The bridge would be opened with loads of holes which the drivers would be instructed how to avoid. The holes would never be filled in but lots of ramps would be put on the bridge over the next few years to allow the drivers to jump their vehicles over them. Eventually there would be loads of extra lanes added to sections of the bridge as no one still understands how the original bridge was built.
This of course is assuming that the bridge actually manages to span the gap and doesn't have some of the road left out to "get the damn thing finished".

Re:It's design not development

[virgil_disgr4ce]

Seeing this comment makes me so happy. It frustrates me to no end that software is treated as nothing more than engineering, with no attention paid to the conceptual and/or interface design.

ALL software needs to be designed. To write even a simple CLI program you need to know inputs and outputs, audiences and purposes. The problem is when software makers try to make a fully-featured GUIed program with the same mentality of a CLI.

Here's to hoping and praying that software companies will learn to take a few steps back, research their idea, reconceive their features and interfaces, understand what the goal of the program even is. And maybe even hire ... gasp ... someone whose sole purpose is good design.

Re:It's design not development

[xero314]

But people aren't willing to wait or pay for engineered software.

People wouldn't know if they are willing to spent that on engineered software because they have never been exposed to it. Large corporations on the other hand have been exposed to engineered software and the successful ones do pay for it. There was a time when people lived in ramshackle homes built out of what ever materials they could find lying around, but in time that changed to more robust building practices that worked out for the long term. And the argument that software changes to fast for that to be true in this field is invalid as Unix, in it's many flavors, has shown longevity, as have many other software projects such as Office suits. And just as a good framed building can be remodeled and updated at a fraction of the original building cost, the same could be done with software if it was engineered correctly. This approach that I am talking about is not entirely unique and aspects of it are used by some of the more successful companies, such as Apple's OS X. But there are thousands of software projects out there, such as the one I am currently stuck working on, that are slapped together with no for thought or engineering process, causing them to need to be replaced and rebuilt every couple years rather than updated and enhanced.

I'm not saying the cost won't go up, but I would bet that it wouldn't go up by much, and they you would definitely get more value for your money. I know I have wasted countless hours dealing with software bugs that I should not need to, and I value my time very highly, and assume others do as well.

Re:It's design not development

[musakko]

Well no one really teaches software design do they.
You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge.

This is a good point. The biggest problem is people can't physically see software. If software was a phyiscal thing I'm sure everyone would be shocked by how shonky and ugly it looks, and be much more demanding about it. But as things are, it's totally abstract and if the job/processing etc. gets done, then that's good enough. The question of whether it's done well, or is well designed can only be determined by other software developers.

Re:It's design not development

[Richard Steiner]

The designing of software is much like describing the plot of a book.. by the time you've given the typist enough details to write what you want, you may as well have written it yourself. Since programmers are just the typists, they usually have no idea what the architect has in mind. When the architect / designer leaves a lot of the design to the programmers, the end result will fall short of what the designer had in mind. This is why a lot of awesome software projects are generally one or two man efforts, not designed by a team. If you're writing a novel, you don't bring in a dozen co-authors; at most, you collaborate with on or perhaps two people.

FWIW, most of the mainframe shops I've worked in don't have separate roles such as "designers" and "programmers" -- instead, they have fairly experienced programmer/analysts who perform both roles as a matter of course, or they have programmer/analysts plus at least one experienced business analyst to collaboratively and iteratively create and refine the specs as development rolls along. The latter is far more common.

Project Managerment may or may not exist at any level, and I'm not convinced it's necessary in many cases when the P/A's are experienced (unless a project requires complex resource coordination, a PM can often prove to be more of a liability than an asset).

The Unisys Airline Center I worked in came closest to what you're describing, but that group was releasing fairly static airline applications with a lengthy (6- to 18-month) release cycle and which already had formal PDS (Design Spec) and PFS (Functional Spec) documention created and *formally agreed to* by the main paying customers before work even began on the new version. From that point, the new version's specific feature set was locked for the most part. That was only design at the screen level, though -- implementation details were usually left to the programming staff.

In the three shops I've worked in (an airline, a window-manufacturing company, and a IT/communications company) where production systems were being developed for in-house use and then supported in-house, expertise was aligned vertically along subsystem lines, not horizontally along coder/designer lines, and most of the work was designed, coded, initially tested, and eventually supported by the subject-area experts on the programming team.

With experienced people, that seems to make the most sense, and it provides most of the advantages of a small 1-2 person team even when a large project is being developed (since the net effect is a large number of small functional teams, each working in their areas of expertise in parallel).

Re:It's design not development

[I Like Pudding]

That was yesterday....We'll be sending you the updated specs soon.

There are lies, damned lies, and software specs.

Re:It's design not development

[shut_up_man]

And comments! Good coders always comment their code!

Re:It's design not development

[Richard Steiner]

Well no one really teaches software design do they.

One of the courses I took as part of my BSCS almost 20 years ago (gads) involved 4-person teams doing precisely that -- the coordinated design, documentation, and presentation of a complex software system (well, as complex as you can get in 10-12 weeks).

That course did an effective job of teaching us a LOT of things, including how to compromise when multiple good ideas were placed on the table and a project stage deadline was coming up.

Granted it was only one class, but even one well-presented class can provide good insights into the process (IMO).

Re:It's design not development

[jaydonnell]

I wish I had mod points to give you.

One of my professors(years ago) and a few of my friends work at JPL. They do engineered software and it is ungodly expensive. Millions just to produce the docs/specs. The question we have to ask ourselves is if the cost is worth it. For NASA/JPL it clearly is. For a serious web app it isn't. Agile practices give you reasonably robust code at a fraction of the cost of "engineered" code. And you get to market sooner.

The fact that most software shops have terrible practices/methods does not mean that all shops should do "engineered" code.

p.s. - I'm only using the term "engineered" because the previous posters used it. I don't think it's a fitting choice for the way we're using it here.

Re:It's design not development

[endx7]

Are you implying it takes over a trillion dollars to produce a well engineered operating system?

Re:It's design not development

[Coryoth]

We've had a couple of thousand years of practice building bridges, and bridges solve a known problem: connect points A and B when there's a gap or a river inbetween.
Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

This is a silly comparison. Saying that building a bridge (as an example of civil engineering) is just a matter of "connect points A and B when there's a gap" is akin to saying a software project is just a matter of "present users data from a database when they need to know it": you're making it sound easy and simple by glossing over all the myriad of complex and important details that need to be considered. For the software project you might want to consider what the data is, how it needs to be presented, where the users are (over a network, worldwide perhaps?), what sort of security is needed, etc. For the bridge you'll need to worry about exactly where to build the bridge, what sort of traffic it needs to be able to carry, what the weather is like in the area and what stresses the bridge will need to tolerate, etc.

Each case has a lot of details that need to be properly nailed down and dealt with. This is not to say that software engineering is easy; rather it is to point out that other forms of engineering (be it civil, electrical, aerospace, or otherwise) are, in their own way, just as hard - something that I think far too many slashdotters don't really seem to have an appreciation for.

Re:It's design not development

[styrotech]

You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge. Software design isn't really taught in this manor, sure your taught how all the bridge building tools work, and even a lot of the engineering specifications. But I have yet to see the software design school that covered more than a class or two into truly how to design software

Ummm... that sounds a awful lot like how engineers at taught at engineering schools. Get a civil/structural engineering degree and you'll come out of school with a lot of math, physics and theory about materials and how they behave under loads etc. It's the equivalent of future programmers getting taught theoretical computer science. If your school was at the applied or 'trades' end of the spectrum you might swap some of the more advanced theory for stuff about the local building codes etc - but those types of graduates usually end up doing drafting or project management anyway.

Nobody gets taught how to design a bridge at university - just a huge amount of theory. Then they get chucked in the deep end at their first job and have to pick up a vast amount of real world knowledge on the job from the more experienced engineers. About the only difference is that legally the engineering graduates have to get someone else to sign off on their work until they get registered.

Re:It's design not development

[SixDimensionalArray]

I had to reply to this because hearing someone else say it just felt so good.

It begs a certain question - users often REALLY want a dynamic and flexible system, yet most software is rigid and structured. Users often say they want rigid, structured software (they want the structure to help them drive their process) but then realize that the structure is too rigid and give up using the software.

It's like some kind of a chicken and egg problem - do you give the user a free-text box and let them type whatever they want, however they want, or do you give them 3 entry boxes and force them to input a first name, middle initial and last name? In the end, does it matter which you choose if they just change their mind after you designed, wrote, tested and implemented their software?

This is a bit rhetorical/satirical, but I guess as long as they pay well enough, who cares?

Seriously though, it is an art and a science to accurately project what a user is actually going to need versus what they say they want. It is also a little bit of luck and experience that makes it so you get it right.

SixD

Re:It's design not development

[Coryoth]

* There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change
* The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

Of course that is, to a certain extent, a problem that we make for ourselves by having a forgiving attitude toward such things. If a client that asked for a 3 story building comes back part-way into the construction process and asks for an extra story to be added between the second and third floors they'll simply get told "No". In the case of software projects, however, most people are afraid to say no to clients, or to set limits on defining the requirements. We've trained users to be fickle and change their minds by allowing them to do so. Of course such tolerance and flexibility has also been a boon for the software industry in that the end products have evolved much faster than they would have otherwise. There are pros and cons - but we should accept the consequences of our choices.

Re:It's design not development

[nuzak]

> you're making it sound easy and simple

The execution isn't simple, but the problem is pretty well defined. You don't generally get people asking for exit and entry lanes in the middle of a bridge, or asking it to accomodate aeroplanes, or asking you to build them out of styrofoam just because some magazine said styrofoam was the hottest new material. Or deciding that each piece of the bridge should have a special connector that requires new tools, or finding that your welders suddenly don't work when you choose a different brand of paint.

Software engineering generally sucks because it's surrounded by suckage all around. Programmers are trained and treated as wiz-kid hackers who can do everything ad hoc, projects and tools are designed that way, and it's no surprise that the products come out that way too.

When it's all said and done, it's probably harder to build a good large bridge than a good payroll program. The thing is, most bridges are eventually finished someday.

Re:It's design not development

[Anonymous Coward]

Clap clap clap clap.

Re:It's design not development

[PitaBred]

But at the end of the day, a bridge is still a bridge no matter what parameters you throw at it. If all software was a web browser, we'd have solved the problem a long time ago. But it's not. It's like expecting someone to build a bridge this week, and then next week do some shoring up of the design of the Eiffel Tower, then maybe an undersea tunnel between America and Europe... a software engineer is expected to understand many different disciplines and rules and change those at the whim of management, not just a single, set endpoint they're aiming at, like getting people across a span.

Re:It's design not development

[CptNerd]

One of my professors(years ago) and a few of my friends work at JPL. They do engineered software and it is ungodly expensive. Millions just to produce the docs/specs. The question we have to ask ourselves is if the cost is worth it. For NASA/JPL it clearly is. For a serious web app it isn't. Agile practices give you reasonably robust code at a fraction of the cost of "engineered" code. And you get to market sooner.

Unfortunately, financial software should definitely be designed to be robust, yet from the systems I've worked on, most of it is hacked together legacy code that often started out as spreadsheets. When you have a multi-trillion dollar economy, doesn't it make sense that the code supporting it be as fail-safe as possible? I could never convince my bosses that the time and money for error-checking code and more logical design were worth it. Fortunately the big multi-billion dollar-losing bugs had been hacked around, but there were often million-dollar losses or mistakes that "re-starting and running the data again" would fix. Makes me want to take all my money and hide it under the matress...

Re:It's design not development

[smellsofbikes]

>we've been building bridges for thousands of years

And despite that, they still fall down.

Read Henry Petrowski's book, "To Engineer Is Human" some time. The point of the book is: an engineer, whether bridge or software, is building a compromise between deadlines, money, user requirements, and laws of nature. As time marches on, that compromise will *always* tilt towards deadlines and money, and eventually it will break, at which point we've learned something for next time.

Re:It's design not development

[ColdWetDog]

If builders built buildings the way programmers wrote programs, then the first woodpecker who came along would destroy all of civilization.

Re:It's design not development

[chromatic]

We've trained users to be fickle and change their minds by allowing them to do so.

What kind of software do you write that succeeds despite not giving users what they want so that your users continue to pay you?

In fifty-odd years of software development, has a single, non-trivial project ever had perfectly defined requirements such that the schedule was perfectly predictable and users were completely satisfied at the end of the project? To my knowledge, not even the Space Shuttle qualifies.

If that's the case, why do people still believe that completely nailing down requirements up front before beginning to code is possible, let alone useful?

Re:It's design not development

[chromatic]

If you're writing a novel, you don't bring in a dozen co-authors...

Funny. I've written plenty of software and several books, including a novel. To publish a novel, you need at least one editor, a copyeditor, someone to do layout, someone to do the cover, plenty of people at the printing press, someone to handle distribution and sales, at least one reviewer and hopefully another proofreader, and several other people I've probably forgotten.

Re:It's design not development

[superbrose]

Even if there is a very elaborate design, then when requirements change, or a design problem is encountered, it can easily happen that code gets written and the design adapted in retrospect.

Since programming is like opening a can of worms, unless we are talking about a trivial project, the design is bound to be flawed somewhere down the line - even if the designer is a genius. What happens next is that design becomes documentation, which can be useful at times, though obviously not intended that way.

The biggest challenge is delivering software on time when there really isn't enough time (Google developers ignore this paragraph). Corners get cut, bugs sprout - welcome to the wonderful world of programming.

Re:It's design not development

[Mr_Tulip]

Yeah, but how many co-authors did you use? The design stage is where software development has the highest risks. And in many projects, there are literally dozens of people all with their invaluable contributions to the design. Once the design is complete - to a point where code monkeys can implement it, feel free to hire dozens of coders, testers, project managers, graphic designers, publishers etc.

Re:It's design not development

[Anonymous Coward]

There are many reasons software is so hard. The first is pretty obvious from the comments.

"I was one of the people in charge"
"I don't know much about software development"

The second is obvious from most of the replies here. All the software weenies that only want to code, without any effort to work to a good design with well defined requirements, interfaces etc. "Software is like art?" -- there is your problem.

Software people often get into the mindset of being an "artiste" rather than a professional. They lock themselves away, produce something wonderful, and demonstrate it with a flourish and a complete disregard as to whether it meets the customers needs. Then they complain the customer doesn't know what they want (scroll up for that comment).

Do you think NASA got their space shuttle and went "oh, hang on guys - that's not quite what we were after - do you think you could redo the wings a little?". No. They were an integral part of the requirements definition, design, build and verification processes.

Finally, the reason this hasn't improved at the same pace as genuine Systems Engineering (see EIA-632) is because of the .com boom. Too many people got into "computers" and "software" without the second clue, because those without the first clue were willing to pay a lot of money. They considered anything to do with computers to be difficult, so they got scammed.

Coding is actually very easy, which means there are many poor "coders" out there who consider themselves "software engineers" the second they have completed their 6 week multiple-choice microsoft "engineering" course, with no concept about engineering design or the methods of reducing complexity in large projects.

Re:It's design not development

[radtea]

But people aren't willing to wait or pay for engineered software.

I think you've mistaken "overdesigned software" for "engineered software". In my experience engineered software is cheaper than the usual mess, because there are fewer bugs and they are found earlier in the development process. Modularity is strictly enforced so the scope of bugs is smaller and unit testing is easier.

It is certainly possible to expend huge amounts on designing a thing to death, but given the howling screw-ups that NASA has produced over the years (was that metric or Imperial?) I wouldn't hold them up as anything like an exemplar of good software engineering practice.

Remember, engineering of all kinds is a matter of finding adequate approximations and compromises to meet specified performance criteria. It is not and never has been about making everything "perfect" by some impossible standard. Perfectionism is expensive. Engineering is cheap.

The real problem with software engineering is that the vast majority of people writing software today simply do not have the intellectual discipline to undertake it.

Re:It's design not development

[chromatic]

Once the design is complete - to a point where code monkeys can implement it....

I reject that premise: The Source Code is the Design.

Yeah, but how many co-authors did you use?

For my most recent book? One, but we've integrated suggestions from our editor and a couple of dozen reviewers. Even our outline has changed substantially from our proposal.

For my novel? None, but again, I've had detailed suggestions from one reviewer, input from several other readers, and specific comments from an editor. Also again, I didn't know where I'd end up when I started, and I'm very glad for that; the novel is much better for following the characters than a predetermined plan.

Re:It's design not development

[Anonymous Coward]

Modular Design...like GNU Hurd? I've oft wondered why development is so slow on this sucker. It is so much easier to "make perfect" than a big whopping monolithic OS kernel. At least in theory. Think about it, some day you may even be able to emulate each server as a core on a processor, thereby making the entire OS hardware. How fast would that be? (I would want a FPGA as a processor though, for "updates.")

Re:It's design not development

[Coryoth]

Has anything ever been completely nailed down with perfectly defined requirements and perfectly predictable deadlines? No, of course not. There are, however, cases where requirements have been adequately defined and projects were completed on time and on budget - it does happen occasionally.

It's not about "not giving the users what they want" and still succeeding, and likewise that doesn't require "completely nailing down requirements up front before beginning to code", it's being clear with regard to changes in requirements. Requirements will undoubtedly change as the project progresses, but I really don't think its that hard to make clear to customers that the later on the change occurs the smaller it will have to be, and that there will be some final deadline for changes near the end of development when new changes won't be accepted (and let's be honest, this happens all the time with code freezes when no new features are accepted prior to release and the focus is on debugging). The building client who wanted to add an extra floor when the building was already half constructed can moan that the builders are not "giving them what they want" but I suspect they'll get little sympathy. At some point reality hs to kick in.

Re:It's design not development

[jaydonnell]

I don't know what you mean by financial software and the implications. Also, what was the cost of "re-starting and running the data again"?

Also also :), agile practices wouldn't hack around a bug. You don't have to do NASA level docs/specs to have good code. Good code doesn't mean that bugs don't go live, but it does mean that you don't hack around bugs.

Re:It's design not development

[chromatic]

The building client who wanted to add an extra floor when the building was already half constructed can moan that the builders are not "giving them what they want" but I suspect they'll get little sympathy.

I really don't understand what this has to do with software. Does the building client get more sympathy if he asks for an extra floor when the blueprints are halfway complete?

Does your definition of success value meeting the initial budget and timeline estimates more than giving the customer what he wants?

(Please note that I have said absolutely nothing about meeting the initial time and money quotes despite requirements changes. I'd like to ignore that question for the moment, if that's alright, as I think the definition of success is a foundational belief.)

Re:It's design not development

[Hotawa Hawk-eye]

So THAT explains the Boston Big Dig!

Re:It's design not development

[rho]

Software engineering generally sucks because it's surrounded by suckage all around. Programmers are trained and treated as wiz-kid hackers who can do everything ad hoc, projects and tools are designed that way, and it's no surprise that the products come out that way too.

Kind of. Computer engineering is where regular engineering was back when the pyramids were being built. They built a helluva lot of crooked pyramids. Or bridge-building, to stay on topic. Quite a few bridges went *ffffft* into a ravine.

I used this comparison with an architect once. Imagine that not only are there no standard building details for how windows should be installed, there are no standard brick sizes. Each county in the state makes bricks of a different size. What CAD details you can get are a mix of hand-drawn sketches and computerized masterpieces, but all in different file formats that are incompatible. And finally, your contractor only speaks Swahili. That's why even a simple Web application can entail weeks or months of constant work.

Things are slowly changing. CPAN and PEAR are good examples of "standard details". There are a jillion Java classes to do various things. And with XML-RPC, SOAP and the likes, convenient communication between incompatible languages are possible. It's getting there, it'll just take two more weeks.

Re:It's design not development

[Hotawa Hawk-eye]

That's why you go through usability testing and paper prototyping early in the development process. Draw out the two GUIs on paper, then put each one in front of the user and ask which matches closer to what they had in mind. Watch their interaction with the GUIs -- if they seem to like interacting with the free text box more than the 3 entry boxes, then ask them why they like the free text box and if they have a reason they don't like the 3 entry box version. Your user may have requirements in mind that they can't or didn't explain but that your design will need to match in order to be accepted.

Re:It's design not development

[Mr_Tulip]

Firstly, I love the idea of Source Code as design - I always try to use CASE tools, a good IDE etc. to make writing code as much as possible about the design, rather than just mindless implementation. However, I don't think there are many large projects that are actually done this way.

So for an idealized project, the premise of the programmer being the designer is a great one, and something we should strive towards. However, software development is currently at the stage where novel writing was before the advent of the printing press. We currently still need designers to translate what a designer can easily imagine into something that will become a scalable, robust and well-implemented system.

Until we get there, large projects simply cannot afford to wait around until the designer (author) is able to laboriously code each and every feature to an acceptable standard. This is why we separate the design from the implementation. I know it's not ideal, but it's just the way it is - except for rather small projects.

Re:It's design not development

[Coryoth]

I think my definition of success would be finding as good a compromise as is feasible between customer demands in terms of features, timelines, and costs. I don't see meeting your clients every whim in terms of desired features while running massively over their required timeline and massively over their afforable budget any better than completing on time and on budget but failing to meet fundamental customer requirements. Soemtimes features, timeline and budget all fit together happily. More often they do not. That means late changes either cost a lot more, require a completely reassessed timeline, or don't make it in. If your clients are willing to given you a blank check and indefinite time to produce a shipping product, fine, meet every whim. In the case that they don't then at some point your going to have to draw lines as to when new features or changes can occur.

Re:It's design not development

[Anonymous Coward]

Are you saying it doesn't? Linux is standing on the shoulders of billions of dollars and thousands of man-years of Unix development by AT&T, DEC, Sun, HP, etc and academia.

Torvalds got a free ride and even then the pre-2.4 versions weren't anything to write home about.

Re:It's design not development

[CptNerd]

Financial software == programs run by financial institutions: banks, brokerage houses, etc. I worked on software that handled mortgage-backed securities, keeping track of mortgages traded between investment companies and banks. The cost of restarting was all the time spent by high-paid managers, traders, and business analysts to figure out where the failures happened, when, and what impact they had on that day's transactions. The programs I dealt with when I first got the contract were incredibly haphazard in memory allocation and use, and had little error checking of any kind. I was on a team that was supposed to "harden" the software and make it more efficient, without really changing much of the code. I told them that was contradictory, they said do it anyway, so I nodded, cleaned up what I could, snuck in bounds checking and function return code checking when no one was looking, and turned in my hours. After 4 years of this, my programs no longer crashed or gave bad results, but the rest of the system was still fragile. Design and testing were considered extraneous cost items that couldn't be justified in the budget, so the customers (banks, etc) were the de facto unit testers. There was no end-to-end testing, no test procedures, no test data, no consistent bug reporting or tracking. In short, everything I've come to expect in modern software development houses.

Re:It's design not development

[chromatic]

However, software development is currently at the stage where novel writing was before the advent of the printing press.

Nit: the novel didn't really exist until around 300 years after Gutenberg's printing press, but I'm not sure what you mean here anyway.

... large projects simply cannot afford to wait around until the designer (author) is able to laboriously code each and every feature to an acceptable standard.

Why is there only one designer? Why have you hired monkeys to transcribe the design into software? (Want to help your projects succeed? Don't hire monkeys.)

What The Source Code is the Design means is that you don't know for sure what you want to build or how it all fits together until you actually do it, at least for any project neither so trivial or so routine that you know exactly how to build it before you even write code.

Just as you don't take sentences from hallway conversations with your customer as the final word in design, I take this as saying that even complete requirements documents aren't the final word in design. Until you can actually get useful feedback from the customer using the program (or technical feedback on what's easy and what's difficult to program), I don't think you can have a complete design.

Re:It's design not development

[mollymoo]

It is certainly possible to expend huge amounts on designing a thing to death, but given the howling screw-ups that NASA has produced over the years (was that metric or Imperial?) I wouldn't hold them up as anything like an exemplar of good software engineering practice.

That wasn't a software engineering mistake. When was the last time some NASA code was the cause of a failed mission? Can you give us an example of some better engineered software that that which flies on the Space Shuttle?

Re:It's design not development

[Mr_Tulip]

Ok, the previous posts are entering flame territory, and I don't wish to pursue them anymore - let's just say that I mostly agree with you.

Re-reading my original post:

"The designing of software is much like describing the plot of a book.. by the time you've given the typist enough details to write what you want, you may as well have written it yourself."

Isn't that essentially what you are saying to me?

My point is that the tools we have available for software design fall short of being able to implement complex applications for us. So we still need programmers to do the grunt work. In an ideal world, we would not need programmers, the designer would be able to produce the application themselves, and then simply hire a publisher, some testers and manual writers etc, and all would be well.

My other point is that it is virtually impossible to design an application when there are more than a few people all wanting to 'help' design. This is where my analogy to novel writing began. You simply don't get as coherent a product if you have a team of designers all trying to push their agenda. This leads to internal conflict, compromises being made, misunderstandings between the design and implementation; the end result is unpredictable, but rarely good.

The quote "A camel is a horse designed by a committee" comes to mind.

And finally: in an ideal world, I would be able to design an application without having to ever see a single line of source code, yet still have a finished product at the end of the process. Although this would probably mean a huge downscaling of the software industry, as everyone would just write their own applications.

Oh, and I probably should have said 'book' instead of 'novel' with the printing press thing. You learn something every day.

Re:It's design not development

[chromatic]

My point is that the tools we have available for software design fall short of being able to implement complex applications for us.

That's a fair point. I agree.

My other point is that it is virtually impossible to design an application when there are more than a few people all wanting to 'help' design.

I don't see why that has to be the case. Certainly I don't trust more than a dozen people to agree on the fine points of any single non-trivial item, but I believe strongly in emergent design. If a dozen programmers can agree that approach X is reasonable and sufficient for the current features, and in presence of good development practices that encourage effective refactoring, I believe it's possible to produce software effectively.

... in an ideal world, I would be able to design an application without having to ever see a single line of source code, yet still have a finished product at the end of the process.

I've never seen visual design tools work in the general cases, and I've seen little evidence that they can produce a finished application to the satisfaction of their users. I remain skeptical.

If you mean however design tools that work on a specification language, then that becomes the source code, just as higher-level languages such as COBOL, Fortran, and Algol replaced assembly languages.

It seems we have a philosophical disagreement. That's also fair.

Re:It's design not development

[shagymoe]

OMG, I wish I had mod points, that should be a 6 LOL.

Re:It's design not development

[jbengt]

spoken like someone who's never designed a bridge

Re:It's design not development

[Maxo-Texas]

Yes that's why buildings and bridges never fail either. Especially when not maintained or used in ways they were not planned to be used for in the first place. And never when they were made out of new poorly understood revolutionary materials invented only 24 months previously.

Software is not as strong as bridges unfortunately so it is much harder to make it work under those constraints.

Both structures and software fail when management pushes unrealistic budgets or deadlines.

Re:It's design not development

[GileadGreene]

You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge.

Christ, I'm sick of the bridge analogy. How many Slashdotters have ever actually designed and constructed a bridge, or really have any idea how hard it actually is? Are there any former bridge builders who are now software engineers reading this? Could you perhaps step in here, and tell us just how well bridge design actually compares to software design? For that matter, how many people have gone to school to learn bridge-building, and subsequently gone on to design and build bridges straight out of school? I'm guessing none. My own experience in engineering (electrical and aerospace rather than civil, but I'm guessing that civil is similar) is that kids straight out of school don't know anywhere near enough to tackle a major design project by themselves.

Re:It's design not development

[Coryoth]

But at the end of the day, a bridge is still a bridge no matter what parameters you throw at it. If all software was a web browser, we'd have solved the problem a long time ago. But it's not. It's like expecting someone to build a bridge this week, and then next week do some shoring up of the design of the Eiffel Tower, then maybe an undersea tunnel between America and Europe

You jest, but I think you underestimate civil engineers. They don't spend their whole careers designing very particular inds of bridges. You refer to the Eiffel tower, so perhaps that can furnish an instructive example. Gustave Eiffel, who designed the tower that bears his name, also designed the armature for the statue of liberty, an array of different (and quite notable) bridges, the observatory in Nice, a church in Manila, along with train stations, gas works, and viaducts, among other things. All of those have their own unique demands and required him to understand different disciplines (the needs of a church are quite different to the various things one needs to know to design a good functioning gas works, or a working observatory). And lets' be realistic, very few software engineers are asked to design a robust transactional database one week, a word processor the next week, and then maybe an air traffic control system. Sure they have a lot of variety, but so do other engineers - you just aren't as familiar with the odd and diverse specialisations within their fields. Again, I'm not claiming software engineering is easy, and certainly there is plenty of complain about in the field of management of software engineering, but you really shouldn't dismiss other fields of engineering so easily.

Re:It's design not development

[swordfishBob]

Actually, designing and building bridges does have complexities, but at least the requirements don't change too rapidly. A multi-storey building is in between - the basic principles are known but details still need to be managed and requirements and assumptions can shift. Joel Spolski wrote about attending a course on managing construction projects, and was stunned to discover how non-static the designs and schedules can be there. That said, software is all about complexity. We try to model real-world processes that real-world experts struggle to understand and describe, let alone real-world joe-average types. And the tools and preferred methods are only a few years old.

Re:It's design not development

[rickb928]

"That was yesterday....We'll be sending you the updated specs soon."

So you can not meet the new spec pretty much the same way you didn't meet the old spec.

I'm watching a failing attempt to replace a VB - based application with a Javascript/Java browser - based solution. We just started our second attempt to get it into beta testing, and it looks like a crucial function is failing. And if this function fails, there is *no* point to the application. Like a spreasheet app that can't add a column of numbers, or a word processing app that can't save your work. Literally, that basic.

But it's only a year late, 100% over budget, and cannot possibly meet 40% of the original specs. If you ranked the specs according to important, or 'weight', it would miss about 60% of the specs.

If I can figure out how to sanitize the details, it will make a great magazine article, a cse study in how not to do 'IT'.

The best part - It's intended to reduce support costs, among other things. I'm pretty new in the support department. None of the experienced analysts believe it will do anything but INCREASE support costs for the forseeable future.

So I'm not sure I'm too disappointed. At least I'll have a job.

-rick

Re:It's design not development

[turing_m]

"The truth is that anyone can be taught to be a reasonably good muscian, artist, or blacksmith and with proper motivation anyone can become great." Only in the movies. In the real world, there are IQ thresholds for skills, and there are upper limits for IQ set by genetics. Only once you are past that will motivation even play a role.

Re:It's design not development

[SixDimensionalArray]

I don't disagree with your comment, as I have spent many hours in usability testing, however, I guess I have even had the experience that a group of users has confirmed through usability testing that a certain UI was what they wanted, the final product was produced as agreed upon, and then the user decided that the UI was not really what they wanted. They "flip-flopped".

This is one reason I admire the simple UI of Google, for example. One simple search box does nearly everything, or at least, so it seems to the end-user. The dynamic aspect of the UI seems to appeal to many people.

SixD

Re:It's design not development

[Anonymous Coward]

That wasn't a software engineering mistake. When was the last time some NASA code was the cause of a failed mission? Can you give us an example of some better engineered software that that which flies on the Space Shuttle?

Oh, yeah, the space shuttle, great example! Yeah, the software for that thing is truely awesome, so long as you don't keep it in orbit during a year date roll over! Sesh... what a horrible design limitation, not being able to plan missions that run through the end of December into January. Not very well engineered if you ask me. Sure, it does some impressive things, but the shuttles are far from a great example of well engineered code.

And to be honest, if you look into it the software on the space shuttles are not all that complex or "well designed" by todays standards. It's mostly a bunch of independant systems running x86 code on 8086/8088 processors, more of a collection of firmware for embedded devices rather than an coherent single peice of software running on one computer... I would say the computers used by the ground teams for control and course plotting are much more impressive!

Re:It's design not development

[Stamen]

Yes, yes, and yes again; you are exactly right. People, time and again, choose cheap software over higher priced higher quality software. People refuse to pay for backups, for testing, and for high quality.

People get the software they deserve.

Re:It's design not development

[TheLink]

(Disclaimer: I am not a Software Engineer, I'm an EE grad who programs)

The problem is most people don't even understand the basic differences in software engineering and civil engineering (or other engineering).

Creating software is design. But most people manage software creation the way they manage the _construction_ phase of a bridge, not the way they manage the _design_ phase of a bridge.

Even you mix it up a bit by saying "building bridge" vs "designing software". Perhaps people do that because for bridges the bulk of the cost is in the construction phase not the design phase, but for software the bulk of the cost is in the design phase, not the "make all/compile" phase.

See my rant here: http://it.slashdot.org/comments.pl?sid=215690&cid= 17519988

Re:It's design not development

[TheLink]

I disagree. Making software is more like engineering than art.

The real problem is most people incorrectly manage the design phase of software the same way they manage the _construction_ phase of a bridge/car/airplane/etc.

They should actually manage it like the _design_ phase of a bridge/car/airplane/etc.

Rant here: http://slashdot.org/comments.pl?sid=215690&cid=175 19988

Re:It's design not development

[dubl-u]

For me it's been the ever-changing requirements which delay projects the most.

I think changing requirements are a fact of life. The only solution I see is to work in small enough increments that requirements change is a business problem, not a technical one.

The teams I run do weekly iterations, and that seems about right to me. A week is long enough to get something useful 100% done, but short enough that even the most antsy product manager can be persuaded to let the team focus on the work at hand. And then you ship the work and make a fresh start the following week.

Well, most of the time, anyhow. Occasionally a real emergency will come along and throw an iteration into disarray. But very worst case, you lose a week's work. And because the beginning of the next iteration is so close, most "emergencies" can really be handled in the normal course of things.

Re:It's design not development

[dubl-u]

My other point is that it is virtually impossible to design an application when there are more than a few people all wanting to 'help' design.

I disagree. And I'd bet what you really mean is "I don't know how to..." not, "It is virtually impossible to..." Unless you have some mathematical proof that I'm unaware of.

You simply don't get as coherent a product if you have a team of designers all trying to push their agenda.

Yes. If you have a bunch of people trying to push agendas, that would be bad. What if you instead had a whole team that shared one agenda: the making of a great product? That's how I spend my days, and it works well for us.

Novels are typically written by one person (although chromatic's right, it's more "one primary person with a lot of collaboration" from what I've seen). But there are much more collaborative creative works. With my own eyes I've seen ensemble theater pieces with substantial creative input from a dozen people. Improvisational theater groups regularly produce extended coherent works with no central control. And I understand improvisational music works similarly.

For more information on this, see the excellent book Artful Making.

Although this would probably mean a huge downscaling of the software industry, as everyone would just write their own applications.

I doubt it. Take the example you mentioned, the printing press. There are now millions of people with the basic skills and material circumstances necessary to write and lay out a book and have it printed. How many do? And of those, how many write good ones? And how few write great ones?

Re:It's design not development

[chromatic]

I think I understand what you're saying, but I'd like to set up a hypothetical situation.

What would you do differently if you worked on features small enough to complete in a week, if your customer were capable of prioritizing features even as he devises new ones, and if you could deliver working software every week or two to the customer?

Would you still focus as much on long-term deadlines as a measure of project success?

Re:It's design not development

[dcam]

All bow before the waterfall model of software development.

Re:It's design not development

[ebbe11]

It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct

...all of which are impossible to do until after the project has finished. For further enlightenment, please see A Rational Design Process: How and why to fake it (PDF) by David L. Parnas, especially section II.

Re:It's design not development

[Antity-H]

It is not only true about software :
The effective expression of a radical idea requires an individual who just happens to have a brilliant insight or even just a vision of something.

This is true for many things, science, craft, politics,etc you name it.

Re:It's design not development

[devonbowen]

I contract once in a while for a company that does great specs. Good enough that I've never even met them or talked to them on the phone. They specify exactly what they want and I write it - on time every time. Sure, maybe I need to ask one or two questions in a short email, but generally everything is well thought out and described. Not once have we had an "oh shit we need to redesign". And this stability allows us to work on a fixed-price-on-delivery basis which means they can precisely budget and be sure they won't have any surprises. It can work that way. I've done it many times.

But most companies don't want to do that. Rather than tell the upper managers that the design will take much longer (because they don't understand that good design should be the lion's share of the project time) they get the software people working before the specs are done. This shows the upper managers that there is progress even if it's not real. Of course, doing this makes delivery later and increases final cost. But, hey, everyone knows that's how software development is, right? Whatever. I'll take their money, too. But it doesn't have to be that way.

Devon

Re:It's design not development

[Jesus_666]

Close, but not quite. software engineering is "I want it to do blassie. Make it do blassie by yesterday." Where 'blassie' is a loose concept and the developer has to determine and define the underlying logic.

You forgot requirements like "has to work like amazon.com even though it's a GUI app and I don't want to pay royalties for the patented stuff you have to implement".

Re:It's design not development

[qwijibo]

Each of the details in the bridge project have obvious consequences, so bridges have large budgets. A catastrophic failure in a bridge means people may die. A catastrophic failure in a software project usually results in more meetings.

If you tried to build bridges like companies try to build software, you'd end up with something that can handle 10% of the expected load, assumes that the bridge will not have to sustain hurricane force winds or any amount of moisture, and can be built in 6 months by 3 people, the leader being the guy who once saw a movie with a bridge in it. And by the way, the bridge must be built from the pile of styrofoam that doesn't fit in the dumpster. The saving grace for the 3 people with no budget or time is that the end users will be responsible for 100% of the testing.

That reminds me, I need to check on my current project which involves rewriting significant chunks of functionality that I've never seen before in the middle of our production cycle. Strangely, I don't remember ever seeing anyone replace major structural members on a bridge while it's being used.

Re:It's design not development

[vtcodger]

***But people aren't willing to wait or pay for engineered software. Windows already costs $199, and a real OS that followed engineering principles wouldn't cost $199. It would cost probably around $10,000 per copy, if it were being sold on every desktop.***

I'd guess closer to $2000 than $10000 because coding and testing a system with clear specifications (I worked on exactly ONE of those in forty years) is just barely more than trivial. I'd guess that if you actually had specs for an engineered OS, your programmers would be knocking off 50-100 lines of code a day after averaging the system test people's hours and a bit of rework. And everyone that wanted to would be going home at 5:00.

But that's still more than most people want to pay up front. Much easier to pay later in security problems, near total rewrites, lost user time ... etc, etc ,etc.

Also GUI OSes are Complicated. We never really got text mode right. But Unix seems mostly right, and even MSDOS will run reliably for years despite a plethora of bugs. I think that maybe we could engineer a text mode OS that works. I'm not at all sure that we have the technology to engineer a GUI that is internally consistent and is usable.

Re:It's design not development

[MindStalker]

HAHA, I totally meant bridge designing not building. Thanks for the correction.

Re:It's design not development

[Peter La Casse]

The real problem with software engineering is that the vast majority of people writing software today simply do not have the intellectual discipline to undertake it.

Amen to that!!

I think most of them work for a company called Microsoft... which explains how C# came to be...

That's not actually true, which is one of the great conundrums of software engineering. For years Microsoft hired the best people, really smart people, good software engineers, and the result was Windows. I took a class once where the testing methodology of Windows NT was studied and praised; they had something like 1500 test engineers. Jinkies! What went wrong?

Re:It's design not development

[menugarit]

Sorry, the bridge specs have changed

Re:It's design not development

[synesis]

The real problem is that everyone thinks that software development is an engineering discipline (or should be). It is not. It is like trying to engineer a living organism.
The spec evolves, the solution evolves, the tools change, the programmers move on and get replaced - and if you are lucky enough to get a working system the company gets bought out and the new company wants it all done again with Linux.

If you were using engineering to produce the human eye you would currently be about 5 billion years behind schedule.

Re:It's design not development

[mollymoo]

You don't know what you're talking about. The shuttle's flight computers are IBM AP-101 systems and there's basically one which flies the shuttle, catastrophe notwithstanding (there are actually four running the same code for redundancy, with a fifth running independedntly developed backup code). The 8086/8088 systems, which I presume you're aware of because NASA were on the hunt for processors a few years ago, are ground-based test rigs.

Yeah, the date rollover things sucks by modern standards, but with highly restricted memeory (424KiB of core on the original flight computers) niceties such as that were often left out. The AP 101 shares architecture with the IBM/360. You might have heard of Fred Brooks - he of "Mythical Man Month" fame. In another classic essay "The Second System Effect" he complains that OS/360 wastes 26 bytes of core to deal with correct rollovers of leap years. He thinks that should have been left to the operator. The computing world was very different when the shuttle's flight computers were developed.

Re:It's design not development

[MindStalker]

Also meant to say that the Ford comments almost made me spew my Dew.

Re:It's design not development

[strikethree]

"But people aren't willing to wait or pay for engineered software. Windows already costs $199, and a real OS that followed engineering principles wouldn't cost $199. It would cost probably around $10,000 per copy, if it were being sold on every desktop."

Horseshit. Microsoft has made billions of dollars off of software without charging $10,000 per copy. No, they have made more than a trillion dollars off of software. I guaran-fucking-tee you that I could build a well engineered OS that is more featureful than ms windows for less than a trillion dollars. I am sure I could do it for less than a billion even.

Wake up and smell the greed.

strike

Re:It's design not development

[Anonymous Coward]

In the scenario you describe, I would be ashamed because I would have failed to solve the customers problem, and the payments for those "weekly snapshots" was just a con.

Go home XP boy - your ideas are being discredited even as you think up new priorities.

Re:It's design not development

[BlueStraggler]

The building client who wanted to add an extra floor when the building was already half constructed can moan that the builders are not "giving them what they want" but I suspect they'll get little sympathy. At some point reality hs to kick in.

Yes, but the key difference is that the final expression of a building is a physical object, whereas software is not. As long as the building is also virtual (ie. you're drawing the blueprints), buildings can go through all the same shit as software. I'm sure if there are any architects here, they could tell us all manner of horror stories of changing requirements and clients who want to add floors or do things even more outrageous. And if the architect is charging a flat fee based on the initial requirements, at some point they'll put their foot down and say no. If they are charging by the hour, maybe not. It's not a whole lot different, really. The error is in thinking that the analogy to construction is coding. In fact, the best analogy to construction is deployment (or manufacturing of CDs, etc.).

Re:It's design not development

[Whomp-Ass]

You should read this Article about NASA Software...

Here is a snippet...

But how much work the software does is not what makes it remarkable. What makes it remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.

This software is the work of 260 women and men based in an anonymous office building across the street from the Johnson Space Center in Clear Lake, Texas, southeast of Houston. They work for the "on-board shuttle group," a branch of Lockheed Martin Corps space mission systems division

Now, I have a software project, that has been in the works for five years, has had 2900 bugs, and has had a sum-total of 5 developers on it, on a budget 1/100th the size, that changes enourmously weekly...and it's true; the user's have no idea what they want...and my users are engineers (!) who don't want to pay for 'engineered' software.

Re:It's design not development

[Coryoth]

Well you're talking about a blank cheque (I presume you're getting paid for each new feature or change the customer decides he wants) and an indefinite timeframe - the customer doesn't care when the "final" product with all the changes he can dream up is delivered, it an be next week, or 25 years from now. As I said, with no such constraints of course you aren't going to worry about deadlines - there are none - or costs - they can inflate as high as you like.

Most people, however, generally have some idea of core necessary functionality, the costs for that functionality, and a deadline that they'd like to have that core functionality up and fully running by. These demands, of course, provide long term deadlines that make a measure of project success - if you manage to get all their changes and new extra features included but miss a couple of core features by the deadline then yes, I would consider that a failure (unless of course the client renegotiates deadlines for core functionality along with the change requests). In other words: more often than not people hve some fixed idea of when they'd like certain things working by even from the outset; those ideas define long-term deadlines that are measures of product success. If your clients never have such things, more power to you - it sounds like you have very understanding and flexible clients.

Re:It's design not development

[Coryoth]

Oh, I agree. My point was more that people often are prepared to put their foot down and say "No, we can't do that", and no one blinks when it happens. Sure someone working by the hour might make late changes, but they'll also be making new deadlines and other renogotiation a condition on such changes. The end result is that, if a fixed deadline or fixed costs exist then you have to put your foot down and have a harsh attitude toward late change requests - the is much less common in software development however.

Re:It's design not development

[jaydonnell]

sounds like you worked for idiots. I don't think there is any solution to that problem :(

Re:It's design not development

[Fulcrum of Evil]

Writing software isn't like building a bridge -- there are often a fair number of incompatible but equally elegant solutions to a give problem

Also, it's really hard to duplicate a bridge. If you could build bridges like software, a 4 lane expressway would mostly involve deciding where the supports should go.

Re:It's design not development

[Fulcrum of Evil]

There was a time when people lived in ramshackle homes built out of what ever materials they could find lying around, but in time that changed to more robust building practices that worked out for the long term.

Not by choice, though: the places where people live got tired of collapsing buildings killing people and enacted building codes, which some builders follow and others ignore because paying off the inspector is cheaper. Meanwhile, a lot of people build extensions using any old crap they have lying around, as you may well discover if you by a house from someone else.

Re:It's design not development

[xero314]

So I guess what you are saying is that the pyramids, of egypt and central america, as well as the great wall of china, the temples of rome and all the other durable ancient structures where built the way they were because of building codes. Or maybe the cathedrals and castles of europe, and the Sky Scrappers of the modern world. I'm sure that if it weren't for building codes the Petronas Towers would have been built from match sticks and bubble gum.

Somethings in the world were built for longevity, and I think the same principles could be applied to software to improve the quality as well as the cost effectiveness. Personally I would like to see companies held responsible for faulty software, same as a car manufacture would be if their automobiles failed to perform as advertised. There are no laws, at least in the US, that states an auto manufacturer has to warranty their products for 3+ years, yet they all do. This is because we expect more from manufacturers of physical products and I happen to thing we should demand the same from software.

I know I would easily pay 5 times what I have on my current OS if someone could offer an alternative guaranteed not to crash, ever. That, even though my current OS has never crashed on me an probably never will (obviously I don't run Windows).

Re:It's design not development

[Dacmot]

The problem is:

        * There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change

Which is why you should design for change, like Parnas always said. Any design methodology that doesn't take change into account is intrinsically flawed.

Re:It's design not development

[Fulcrum of Evil]

So I guess what you are saying is that the pyramids, of egypt and central america, as well as the great wall of china, the temples of rome and all the other durable ancient structures where built the way they were because of building codes.

More or less, I was referring to houses (duh). Building codes are formalizations of things that have failed - they built a lot of crappy pyramids, and a lot of people died building the great wall of china.

I'm sure that if it weren't for building codes the Petronas Towers would have been built from match sticks and bubble gum.

Your house probably would be. Who knows, they might have done that anyway.

I know I would easily pay 5 times what I have on my current OS if someone could offer an alternative guaranteed not to crash, ever.

And so long as MS offers a $40 OS that is compatible with what you already have, noone will buy that OS, so you will pay $200,000 instead of $200.

That, even though my current OS has never crashed on me an probably never will (obviously I don't run Windows).

The tricky part comes after you turn on the computer. Everything fails.

Re:It's design not development

[xero314]

Building codes are formalizations of things that have failed

The problem with software is that regardless of the countless failures companies are not making the necessary changes to avoid those same failures in the future, and consumers are putting up with it. You will notice that there are very few start up software companies that survive, but those that due tend to use more standard engineering practices (you can use slashdots favorite Frog Creek as an example if you would like)

Your house probably would be. Who knows, they might have done that anyway.

Actually my house is built of Block, which is both more expensive to build and more durable than the standard Wood Framed house, and far above the minimum requirements for the building code of when it was built (other than the lack of GFI sockets it meets todays more stringent codes). There are shoddy constructed houses out there, but a fair number of people prefer something that will last.

And so long as MS offers a $40 OS that is compatible with what you already have, noone will buy that OS, so you will pay $200,000 instead of $200.

This must be why everyone has changed over to using the many free operating systems that support everything they need (word processing, email and web browsing for the majority of the populous).

It still stands that software sucks because of lack of real engineering practices, and regardless of people being cheap or not, no one has but up an argument against that. People may have given reasons why we don't use solid engineering practices, valid or not, but that still doesn't debunk the idea that it is the lack of these processes that are the reasons behind shitty software.

Re:It's design not development

[Fulcrum of Evil]

This must be why everyone has changed over to using the many free operating systems that support everything they need (word processing, email and web browsing for the majority of the populous).

Normal people don't install OSes. they take whatever Dell gives them, and MS says that that will be windows.

You will notice that there are very few start up software companies that survive, but those that due tend to use more standard engineering practices

It has little to do with that and a whole lot to do with producing something at a price people are willing to pay. engineering is often secondary to that.

Re:It's design not development

[xero314]

the user's have no idea what they want...and my users are engineers (!) who don't want to pay for 'engineered' software.

You can't have it both ways. Either the don't know what they want or they don't want engineers. And they are paying to cost of software designed and built using engineering principles, they just aren't getting it. Admittedly I only have anecdotal evidence, but experience has shown me that well engineered software, like all products, cost less than hap hazard development over a long enough time frame. In you example you have spent 5 years and have yet to actually deliver a working version of what your users want/need (I don't consider software with 2900 bugs to be complete). I don't know this one hundred percent but I would guess that had your project been run using solid engineering principles it would have been completed, bug free, to the exacting specifications signed of by your users, or at least your clients, in less than half the time and requiring less than one full time employee (not developer or engineer) to maintain it. But that is generally what happens when you have software that "...never crashes [or]...needs to be rebooted."

Re:It's design not development

[Bloke down the pub]

If you could build bridges like software, a 4 lane expressway would mostly involve deciding where the supports should go.

I can answer that question easily - the exact same place they did on the old bridge. Yes, I know it was a triple span arch and this one is a single span subvention ... surtension ... whatever ... can't you, umm, adapt it? Anyway, just get on with it, mmmmmkay?

What makes software development so hard......

[Chineseyes]

Viagara, Cialis, red bull, two brazilian hookers and a swiss midget.

Re:What makes software development so hard......

[SoulRider]

Um, thats "What makes software development so hard", not "What makes software developers so hard"!

Re:What makes software development so hard......

[iamdrscience]

two brazilian hookers and a swiss midget.

Herein lies the problem, how often are you going to find two brazilian hookers and a swiss midget in the same place? I would think there are probably more brazilian hookers in Switzerland than Swiss midgets in Brazil, but it's really unlikely either way. I think the best solution to this would be to fly from Switzerland to Brazil with a Swiss midget as your carry-on, that way you're not leaving anything to chance.

Re:What makes software development so hard......

[owlnation]

hmmm, not impossible in Geneva, you may yet be lucky.

Thank Goodness

[pyite]

That was a close one. If this was another Joel on Software article, I was going to have to promptly defenestrate myself.

Re:Thank Goodness

[xlordtyrantx]

Are you trying to educate slashdotters?

Re:Thank Goodness

[morgan_greywolf]

Joke: ---> . --->
You: ___ O __
  _ _ _ /|\ _ _ _
  _ _ __ | __ _ _
  _ _ _ / \ _ _ _

Re:Thank Goodness

[xlordtyrantx]

heh... maybe, but I was asking about the reasoning for linking the definition of the word defenestrate...

One word: Skill, Talent and Knowledge

[rudy_wayne]

Anyone can write a book. But, writing a book that's *GOOD* and that people will want to read is an entirely different matter.
Computer software is no different.

Don't treat it like a science.

[Anonymous Coward]

The idea is that if we're going to turn the creation of software into a true science, we need to first have principles.

Treat it like an engineering discipline.

With science, you're observing and making measurements of the natural world and testing hypotheses to develop a theory on the how and why of nature.

Engineering, OTOH, you are applying natural physical laws to create a man made piece of technology or something.

Ok, here comes the Wikipedia cites on science and engineering from folks who are completely missing my point.

Simple, really. Three words

[Anonymous Coward]

the goddamn users

Re:Simple, really. Three words

[Archangel Michael]

Exactly!

Except we tend to call them L-Users, or function 1d10t faults. As in "We suffered a fault in function 1d10t and are currently patching the system."

Seriously though, Never underestimate the power of a group of average persons to exceed your expectations and find the one (many?) undiscovered bug. Whatever you can think of, they can exceed in mere minutes.

People expect too much too soon.

[Dex5791]

Good software takes time to develop. There is sometimes a tendancy to set unrealistic goals and when they aren't met the people in charge feel let down.

Re:People expect too much too soon.

[rrohbeck]

Good software takes time to develop. ... during which the specifications usually change, or holes are found in the specs. It's chasing a moving target that makes it hard. Interfacing with other [sub]systems that themselves aren't final/stable. Trying to nail Jell-O, basically. Try to build a bridge out of Jell-O!

Re:People expect too much too soon.

[Anonymous Coward]

Mod parent down. This is what I get when I pay for the +5 highlight version?

Re:People expect too much too soon.

[Larus]

Raising children is another typical case where people expect too much too soon. The different is that most parents are just disappointed when the children don't turn out as they wanted. In software development there are status and salary on the line for everyone, so we don't want to accept failure.

Maybe we'll be disillusioned once we look beyond our cubicles and see that other projects had problems too. But unlike biology, failed software projects cannot maintain themselves and bring forth future generations that may be more successful than they are. In other words, built-in resilience is not a trait for failed engineering projects.

Cosi triste...

What large projects DON'T have problems?

[panaceaa]

In related news, humans still can't seem to bridges with any reliable schedule or budget. Despite the fact that bridges have been built probably since the dawn of man, and we've been building suspension bridges for at least 500 years.

Re:What large projects DON'T have problems?

[Anonymous Coward]

we've been building suspension bridges for at least 500 years.

That was before the formal invention of capitalism, back when people did it right the first time because that made the rich people that fed and housed them look good.

Now, it's simply a function of how much money can be extracted by doing the least amount of work for the most money possible. Employees do this because they're "lazy good for nothings communists who think they deserve a free ride", corporations that do this are "shining examples of capitalism who deserved every dollar that was coming to them".

Re:What large projects DON'T have problems?

[Anonymous Coward]

Large delays and costs are caused by the endenmic stupidity of politicians trying to second guess engineers and architects, as evidenced in the case of the San Francisco Bay Bridge. This can happen with other clients as well, it's just that politicians screw it up worse.

Actually, they can build bridges and skyscrapers nearly on time and close to budget. You can't have both. There are always unforseen circumstances that cause either, or both, to rise a bit. It's just the politicians and bribery(lobbyists) that screw things up. Structural engineering is very definitive. Most severe budget overuns and delays are usually caused by incompetence(clients changing their minds, bad contractors, or inexperience), unforseen or unforseeable accidents or trouble, but that's generally expected and does not cause significant delay or costs.

Any engineering project that comes both under budget and ahead of time is highly suspect. Someone skimped along the way or lied about the timeframe or costs. The only way for it to happen is if material goods suddenly expeirence a price drop. For example, steel futures suddenly dropped mid project, causing steel prices to drop, reducing construction costs. Construction, construction labor, and engineering are all highly predictable and accurately measured quantities. General costs for standard construction, barring stupidity, range from 100% to 110%. Rarely, it may be as low as 95%. Anything well below that range is suspect. Anything well over that range is caused by bickering and last minute changes by clients, not the engineers. For fancy or unusual architectural designs, the costs may be more, but the cost estimates and time frame of standard construction is predictable.

Repeat after me.
Programmers are not engineers.
Programmers are not engineers.
Programmers are not engineers.
Programmers are not engineers.
Programmers are not engineers.
Programmers are not engineers.
Programmers are not engineers.

Except for NASA rockets and mission critical high end mainframes, I have yet to see any programming strictly follow engineering methods. Not that it can't, just that no one out there understands how to do it properly, or no one is willing to follow strict engineering methods. No Computer Science student follows any engineering methods. "Software is easily changed," is the mantra that's been ingrained into people's minds.

Electrical Engineering, however, does follow engineering methods more. There's material and time wasted when you actually have to fabricate a physical object, so those fields tend to follow engineering methods.

Re:What large projects DON'T have problems?

[Pollardito]

In related news, humans still can't seem to bridges with any reliable schedule or budget. Despite the fact that bridges have been built probably since the dawn of man, and we've been building suspension bridges for at least 500 years. that's only because bridge building isn't nearly as easy as bridge building, there's a lot more that goes into it

Re:What large projects DON'T have problems?

[BovineSpirit]

The Millau bridge. At over 300m it's the tallest bridge in the world, the only delays appear to have been weather related. It opened ahead of the revised schedule. A quick google search for "millau bridge" and controversy/delays other words threw up lots of US projects, but no real stories about the bridge. When engineers push boundaries there will always be problems, but I think mankinds' overall record in bridge building is pretty good. Judgeing by recent Slashdot comments the US does have a problem with large civil engineering projects, but I don't think this is a worldwide problem. Then again, maybe the rest of the world is better at covering things up...

Re:What large projects DON'T have problems?

[panaceaa]

No Computer Science student follows any engineering methods. "Software is easily changed," is the mantra that's been ingrained into people's minds.

What you describe is not how things are taught at a proper university. At the University of Virginia there was extensive discussion on applying engineering project planning to computer science, how the waterfall process works (including its derivatives) and how changes at different points along the waterfall affect the financial cost of the change. At Cornell, there's a computer science program in both the liberal arts and the engineering school. I'd hazard a guess that the engineering school program definitely discusses the finer details of computer science as an engineering discipline.

Finally, there are well publicized standards for how to run a rigorous software development project. There are many companies throughout the world that follow these standards, mostly consulting companies, but you're correct that these practices are not the norm. (Especially in the West.)

Re:What large projects DON'T have problems?

[IfritZero]

bridge building isn't nearly as easy as bridge building I disagree with that statement. It's most definitely harder.

Re:What large projects DON'T have problems?

[micktaggart]

If I am a contractor for a government agency or department, does it really make a difference if a project gets delivered on time or on budget? Of course not. The contractor doesn't care, the bureaucrat doesn't care, and the politician doesn't care either. To get back on topic, I dare to say that software development isn't really "hard" when your customers are private organizations.

Re:What large projects DON'T have problems?

[darknite1979]

Ummm well the problem with building bridges is that if they are not built properly people die. I have no experience with medical software but when people make medical software they make damn sure its built to VERY tight specs or people die and they people who want medical software know this.

Two things make software "hard"

[mandelbr0t]

First, the people who make the technology decisions often don't have the required technical know-how, and have a terrible tendency not to listen to people who do. OTOH, the people who have the technical know-how have absolutely no idea how to write a business case. Thus, there's usually a disconnect between the people who understand the business requirements, and the people who understand the technical requirements. Vendor loyalty has been known to be a sticky issue as a result.

Second, there's always a problem getting a bunch of talented, egotistical (ok, so not all software developers have ego problems...) quirky, eccentric and generally difficult people to work toward a common goal. The common analogy to being a successful director/manager of a software project is to that of "herding cats". My experience has been that business types don't react well to the often-emotional developer types, hearing the emotional outburst, but ignoring the content of it. Developers would do well to learn some more social skills, and director/manager types would do well to listen better.

mandelbr0t

Re:Two things make software "hard"

[Anonymous Coward]

First, the people who make the technology decisions often don't have the required technical know-how, and have a terrible tendency not to listen to people who do. OTOH, the people who have the technical know-how have absolutely no idea how to write a business case.

Actually, they usually do, that's why there are sales departments to prevent them from communicating to the customer the expected budget and time..

... Posting as an AC for obvious reasons

Re:Two things make software "hard"

[professorfalcon]

Congratulations. You just described traditional engineering, as well.

These personality and management conflicts are found in other types of engineering. It's not new to software. While other types of engineering do benefit from a longer history and more standardized parts, you still have companies wanting to approve things that can never work, and you still have jerks working in the company.

Construction has lots of problems, like software engineering, but construction companies manage the risk and timelines fairly well, overall. Software companies should, too.

Re:Two things make software "hard"

[Anonymous Coward]

I second your opinion.

I have worked on many projects as a contractor/consultant. For most business applications, the biggest hurdles are the people problems. Trying to get "hot-shot" programmers to conform to a sane standard is almost impossible, even when they are the ones that finalized the standard. Then there is always the Not-Invented-Here syndrome as well as the You-Don't-Know-Nothing crowd (always dissing proposals but never suggests an alternative).

I've gotten to the point where I have to pick my clients very carefully or the blood will flow (mostly mine as I burst some vessels :))

Re:Two things make software "hard"

[k12linux]

In a prior job I was asked (in front of everyone in the boardroom) to estimate how long it would take to do a project which had just come up and only been discussed for 15 minutes or so. When I said that I would need some time to spec it out I was told that would not be possible.

After thinking about it for a minute or two I decided that 90 days would be reasonable based on what I knew of the specs at the time and my current work load but that I could manage 60 if I could put this one first on the list of priorities.

I was then told that 60 was too long, how about two weeks? I finally compromised at 30 days with overtime and absolutely no other work on anything else in the company.

I finished the programming on the evening of the 30th day, went home and wrote up some instructions I could use for training the people who needed to use it. First thing the next morning I was taken to a conference room by my boss and literally *yelled* at because nobody was using the program yet. (Thisw as the 31st day.)

I explained that the program was done but I had to train the users that morning so they could be operational in the afternoon. (Only about 10 people needed to use it right away.) This was dismissed with a *shout* of "Nobody is using it, therefore it is not finished!" He considered the program a failure.

Do you see anywhere that management could have been at fault for the project "failing" here? It's all too common to have management fall into the Capt. Kirk role. (Scotty: "I have the engines at 100%", Kirk: "Damnit Scotty, I need more power!") Maybe it's the techies who do screw around that make them think the rest of us just need to be pushed harder to get better performance out of us. I don't know but I've seen it more than you might think.

BTW, when I say "yelling" I mean it. Face red, blood pressure skyrocketing, the works. BTW #2, the program worked just fine starting late that afternoon after staff were shown where it was and how to use it. BTW #3, that was the day I decided to find a new job and stop blowing off offers from headhunters. (I was somewhere else with a 40% pay raise three months later.)

Re:Two things make software "hard"

[Richard Asbridge]

Add to that, that the people who are supposed to *know* what the business requirements are often not schooled in communicating abstract thoughts in the written word. They communicate by (imperfect) example that does not actually tell the software designer/developer what the problem is that needs solving, they simply explain one example of the problem and gloss over the hard bits because they are hard.

Re:Two things make software "hard"

[Anonymous Coward]

When I said that I would need some time to spec it out I was told that would not be possible.

After thinking about it for a minute or two I decided..

There's your problem right there. Your answer instead should have been "Then I can not give you an estimate." and left it. Repeat it, if needed. Then, once the meeting is finished, you go an update your CV and apply for jobs with companies that aren't run by monkeys in suits who think bullying is an effective management technique.

In other words, software engineering processes will only improve when software engineers stop acting like pussies.

Re:Two things make software "hard"

[Fulcrum of Evil]

I explained that the program was done but I had to train the users that morning so they could be operational in the afternoon. (Only about 10 people needed to use it right away.) This was dismissed with a *shout* of "Nobody is using it, therefore it is not finished!" He considered the program a failure.

At which point you reminded him that your original estimate was 60 days and that you only agreed to 30 days because they badgered you, right?

BTW, when I say "yelling" I mean it. Face red, blood pressure skyrocketing, the works.

Best response here is to smile a bit - nothing takes the steam out like finding out it's hilarious.

Re:Two things make software "hard"

[jnana]

Absolutely. Developers really need to learn how to handle these kinds of situations.

It would help to have some pre-canned responses for this kind of bullshit, which comes up again and again in dysfunctional environments. Perhaps a 'pattern language' for dealing with common scenarios, such as the original poster outlined. A possible solution in this case being, "would you ask a civil engineer to give you an estimate of how long a building will take and how much it will cost after 15 minutes discussion?" "And if you would, would you have any confidence whatsoever in the estimate?"

Then again, if that's the kind of environment you're in, you should just look for another job, because fighting ignorance and incompetence from above is as frustrating as it invariably is futile.

Re:Two things make software "hard"

[Reservoir Penguin]

So what is the moral of this story? You got a new job with a pay increase and a new friendlier manager? Did you call the old one and thanked him for pushing you to make an importnat change in your life?

Theory is great

[hsmith]

practical application isn't so. Requirements change, clients see something and want something different. You work a month on something and find out it isn't working anywhere close to the way it should. Tasks are more complex then people thought at first, the list goes on. It is a complex thing with infinite ways to do things. It isn't any easy process.

Technology Review tackles the same subject...

[mcho]

...but, instead, they profile Charles Simonyi and his quest to "re-program software".

He's developed an approach he calls intentional programming (or, more recently, intentional software), which he hopes will overturn programming. If Simonyi has his way, programmers will stop trying to manage their clients' needs. Instead, for every problem they're asked to tackle--whether inventory tracking or missile guidance--they will create generic tools that the computer users themselves can modify to guide the software's future evolution.

Read the article at Technology Review.

Re:Technology Review tackles the same subject...

[clickclickdrone]

>they will create generic tools that the computer users themselves can modify
That'll be Excel then? Job's done. Next?

Slashdot.

[Anonymous Coward]

Slashdot makes it hard.

PHB?

[Jumper99]

I find most of the issues that arise on projects are not so much that the developers are incompetent, but that no one sets realistic, or are not allowed to set realistic expectations. How many times does the business change the specs? How many times are the developers given more time to allow for the changes?

The last project I worked on had a hard deadline. We were in the UAT phase when the business decided it wanted a change. A big one. Despite the fact that we were mere days from production, we were expected to make the changes, test them, and still meet the schedule. IT managements take was as usual. The business pays the bills so the business gets what it wants.

Until IT management can stand up to the business side, software development will always suffer.

Re:PHB?

[Anonymous Coward]

I agree. Our managers typically agree to customers' demands without having any idea of what the development impact might be. It's devolved to the point where insane delivery schedules are set with both sides knowing they won't really be met. Instead, a buggy first draft is delivered on time and the real requirements are fleshed out during an expensive and time consuming test and enhancement cycle.

The summary of the article illustrates a common problem: this guy has no experience developing software, but was in charge of a large software development project. Having a little development experience before getting put in charge of a large development project would be beneficial.

Hard is better than impossible

[composer777]

The reason it's hard is because we are doing the impossible, at least if you had asked someone 50, 20, or in some cases even 10 years ago. Solving previously impossible problems requires quite a bit of innovation and novel thinking. But, it's better than it was decades ago, even if it is difficult. What we need to remind people is that even with fast computers, the problems are still difficult to solve.

Also, automating tasks that we used to perform ourselves is also difficult. It's one thing to walk and chew gum, but another thing entirely to precisely describe the mechanics of doing so. (One is very easy, the other requires creating precise models of how things work, a very hard problem).

Re:Hard is better than impossible

[Anonymous Coward]

The other reason it's hard appears in the article summary:

I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development.

Indeed.

Re:Hard is better than impossible

[ralphdaugherty]

...20, or in some cases even 10 years ago.

      Oh, what do you think you whippersnappers do that we didn't do 10 or 20 years ago, and in a few k of memory with a few megahertz of CPU.

      As for the internet thing, it's just a bigger network than our Novell or phone line connections, other than having a bigger network the last 10 to 20 years have nothing on us.

  rd

Ideas come faster than code

[nharmon]

What Makes Software Development So Hard?

Simple. The current state of technology in regards to human imagination makes it many times easier to think up of good ideas or improvements for computer software than it takes to create or implement them. Thus the requirements for developers are often made by the ones with the ideas with no idea of the costs (in terms of time or money) involved in making those ideas a reality.

Joe Sacks says "Gee, it would be a great idea of our flight tracking program also tracked fuel usage and recommended fuel loads to minimize weight on subsequent flights". See? It only took Joe all of about three minutes to think that up. He figures that a reasonable amount of time to implement this idea is one or two days. After all, those "computer guys" are pretty smart and can "do anything".

Two days later Joe receives a report that the "project" is over budget (needed to check with the FAA on this one, that cost money) and overdue.

Sure, you could just say that these problems need good project management. But seriously the problem is that these "problems" are not considered worthy of project management. It only took Joe three minutes to think the idea up...why should he hire a project manager for *THAT*. It's not like he's building a new hangar.

Re:Ideas come faster than code

[iangoldby]

The current state of technology in regards to human imagination makes it many times easier to think up of good ideas or improvements for computer software than it takes to create or implement them.

Hmm. I don't think that is generally true. While it might be true in a few cases, in my own experience of developing software the hardest part is not the implementation, but coming up with really good ideas.

Implementation of ideas in software is generally not especially difficult. (I know, sweeping generalisation!) The bits that take time, that derail the process, are when you are half way through an implementation and suddenly ask "What do we do about X? It's not mentioned in the design." If the idea is less than great, there may be no obvious answer, or 'right' way to do it. You then end up either making a horrible kludge, or going back and making major changes to the design, refactoring, throwing away great swathes of code. (Guess which usually happens.)

It takes an extraordinary amount of foresight and insight to come up with an idea that is fully consistent both internally and with all of the related existing software infrastructure.

Re:Ideas come faster than code

[Richard Steiner]

The parent seems to be describing the types of ideas you sometimes see from end users and/or mangement, while you seem to be describing ideas from someone who has a better idea of the software development process.

Sometimes those sets of ideas intersect. More often, unfortunately, they do not.

Re:Ideas come faster than code

[inKubus]

That's why I like the Dr.Dobbs quick-kill project plan--you make a list of features you're NOT going to develop also, and have the stakeholders sign off on it ;)

One wag of my finger, one tip of my hat

[iPaul]

First the wag: The idea that the interveiwee states early on, that software development is not introspective, is horse-hockey. We think about it all the time. We invent new, clever ways to diagram software, capture requirements, interview users, validate functionality and come with all sorts of certifications. More than, say accounting, we are process focused people. Maybe our processes suck, but we spend a lot of time, energy, certification exams, etc, on those processes.

Tip of my hat: He does mention starting small and iterating. I think that's the best way to build software.

Re:One wag of my finger, one tip of my hat

[Beryllium Sphere(tm)]

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

Re:One wag of my finger, one tip of my hat

[BalkanBoy]

That's what RUP (Rational Unified Process, now owned by IBM) is all about - iterative & incremental development.

Following RUP + SCRUM based approach has done wonders for multiple projects where I currently work. It seems to work every time, if everyone is on board, daily engr. meetings are conducted to check on status, or adjust expectations if something is taking longer than expected/calculated. But because the breakdown of tasks is such that we have brief sprints, realistic, well defined smaller tasks, and management knows exactly where we are with development at any point in time, it seems to work very very well (at least for us).

The day where you design/capture everything up front for a complex system, and then just build it out as if it's set in stone, are officially over. Software programmers today and software programmers 20 years ago, are two different beasts (probably same with hardware engineers, though I'm a CS major, not EE/CE).

Re:One wag of my finger, one tip of my hat

[inKubus]

Tip of my hat: He does mention starting small and iterating. I think that's the best way to build software.

Object oriented programming. If you do anything more than once, make a method that does it. Then it's easy to make changes withour editing 1291 different places. If you export/import a veriable more than once, put it in the object and make an interface.

The problem is this takes time. Sometimes it's better to just write in long hand, the whole app, and THEN filter it down into smaller modules. Meanwhile you are editing a lot but pledge to yourself the second time you make a major edit in many spots you're going to cut that out and replace it with a function!

the devil is in the details

[prgrmr]

Rarely can a single person manage all of the details. Coupled with the generally poor estimating skills I've observed at all levels of staff and management, technical and otherwise, significant portions of projects tend to be guesses. Granted, usually educated guesses based on a certain amount of information, but ultimately still guesses. Getting management and users to want to have the patience to quantify everything is all but impossible.

For Me

[KermodeBear]

What makes software development so difficult for me in my particular case:

1. Requirements are not clearly defined
2. Requirements that are defined change constantly
3. No existing documentation on the system I work with
4. Outdated technology (Not a big issue, but many things are just easier to do with newer software)
5. The sales department sells features that do not exist, promise a date, and do not consult IT at all about the feasibility or time estimates

4 out of the 5 things I have listed above have to do with bad information / lack of communication.

Re:For Me

[cryfreedomlove]

> 5. The sales department sells features that do not exist, promise a date, and do not consult IT at all about the feasibility or time estimates

My advice to you, for this item, is to begin looking for a new job right now. If your sales force can't make enough money selling the product you already have then you don't have a great product. Then, couple that with the fact that your sales force seems to have free reign to put the company into lose-lose deals that will end with bitterness for the customer.

Be candid with management. If that does not work then run away. There are alternatives at the same or greater income level. That's been my direct experience.

Re:For Me

[KermodeBear]

Working on it right now. (o: However, my last job had the exact same problem, and I fear that I'm falling prey to "The Grass Is Greener..." mentality. I'm also not quite how nice it is to ask in an interview, "So, does your sales staff sell things you don't actually have?" I suppose I'll find out soon enough.

Re:For Me

[Jeppe Salvesen]

You can beat around the bush a bit. Ask about typical overtime. If they say "rarely" and don't snicker while saying it, you've probably found a pretty good place to work (since you're not one of those people who like the adrenaline rush of making stuff on the fly).

Re:For Me

[CoolCat23]

Sales force *always* have the last word... Because "they generate revenue". And, obviously, as a mere "technician", you don't.
So when they say the product will be on the shelves on a given date, you just have to bend your head around it and work twice as hard to release on schedule. When you complain about this, they come and tell you that they have to take into account thousands of parameters you "can't understand from your technical point of view", such as marketing pressure, being the first to release a product (well, at least *something*) on a given market, and so on.

If the product works... well, this is good marketing, good market analysis, etc.
If it fails... It must be that the product was not good enough, heh.

Once upon a time... computer science was a highly elitist science. People had to have a mathematics background, spend nights making holes on little plastic cards, wait for hours for the Computer to digest them and spit out some result. Those people were voodoo priests for the common man.
Now, everyone and his ape have a computer at home. Everyone thinks they are doing "computer science" when it is just "bureautics", "programming" when it's only recording a macro in Excel. They are used to use MS Office, Google, 3D Games with physics simulation... It's so ubiquitous now, that they don't see the magic in it anymore. They don't understand the complexity behind all of thoses products.
So three things happen :
1) executives don't believe it's hard, choke the budgets, shorten dev time.
2) they hire just anyone under 25 and think they can turn them into Java (or whatever) gurus in a few weeks. Anyone with, say, some geography master will do (seen that with my own eyes).
3) "computer science" is considered a whole. You know PHP ? good, go tune that Oracle database. Oh, when you're done, set up some backup scripts on that Solaris system, please. And after that, you'll be promoted "senior" Java developer, because, hey, after all, you've been in this company for two years, so it's time to get you a more shiny title on our resume...

Guess the result ?

Re:For Me

[computational super]

my last job had the exact same problem

And the job before that, and the job before that, and the job before that...

Seriously, I finally quit job-hopping when I realized that all programming jobs are the same.

Re:For Me

[plopez]

You may ask questions like:
1) What is your testing process? (this is a big one).
2) Who defines your requirements? (better not be the sales droids)
3) How much time is spent on requiements? How much on design? How much on coding? (Too much time in any one area is a warning).
4) How are deadlines set? Who sets them?
5) How closely do you work with your clients? (the closer the better, the programmers had best not be in a 'black box').
6) Have you read 'Brooks'? GoF?

If you ask your prospestive employer a few questions like these you *will* know if you are stepping into it.

Re:For Me

[0xdeadbeef]

I'm also not quite how nice it is to ask in an interview, "So, does your sales staff sell things you don't actually have?"

It depends. Do you think they'd be offended if you asked "Is the Pope Catholic" or "Does a bear shit in the woods?"

Re:For Me

[anomalous cohort]

I found this one minute risk assessment tool to be particularly lucid on the issues.

the five p's

[Anonymous Coward]

Pre-planning Prevents Piss Poor Performance

Why is it so hard?

[CasperIV]

It's hard because of stupid people. I swear, users get increasingly more challenged as time goes on. An incompetent user would rather push the wrong button and see what happens then read anything. My new system is fool proof; each button will be a color coded geometric shape....

Re:Why is it so hard?

[RetroGeek]

My new system is fool proof;

No program is foolproof, because fools are so ingenius....

Re:Why is it so hard?

[CasperIV]

True. Never under estimate stupid people in large numbers... or people with too much time on their hands.

Re:Why is it so hard?

[mollymoo]

It's hard because of stupid people. I swear, users get increasingly more challenged as time goes on. An incompetent user would rather push the wrong button and see what happens then read anything.

If your software doesn't take account of the fact that your users prefer to experiment than read the manual it's you who is incompetent, not the user.

Re:The answer is easy

[Anonymous Coward]

smart programmers come out of school thinking that the real world is just like college where everything must be perfect and "elegant".

It's not just a thought, real world code is so repulsive it's almost unbearable. Software, like art cannot be forced and the world is only interested in paying for cheap prints sold by marketoid parasites. Often the more aesthetically pleasing app is superior in the mind of those holding the purse strings. Microsoft know a thing or 2 about software in the real world, Superficial enhancements for the win.

Complexity & Culture

[denoir]

There are two major reasons why software engineering is today not comparable to more traditional types of engineering.

The first reason is the complexity of the systems. There are essentially extremely many interacting parts that cause all sorts of emergent phenomena. This is a field that we don't really know how to handle very well - good quantitative models of complex systems simply don't exist. We need more study of systems theory and chaos theory to build some form of predictive models on which in turn can base planning.

The second problem is a sort of 'freedom of thought' culture that sees coding as the vaguely mathematical expression of ideas. As there are a huge number of degrees of freedom most attempts to regulate it in practice have been abandoned. So instead of just the problem of describing a mathematical problem, you throw individual human preferences, thinking and biases into the equation.

Of course it can't continue that way forever and we need to move to a meta level of coding. We do have well-constructed systems, such as the software on-board an airplane - but they are laughably primitive (because you have to account for all possible states the software can get in to). And we have advanced software, but it is laughably unreliable.

Still, there is reason to be hopeful. We're not inventing the wheel every time any more (just the horse carriage and so on). Modern programming systems have extensive class libraries that are on average a more stable foundation than making a custom implementation from scratch every time.

Ultimately however we need to know how to handle complex systems and how to enforce convergence/stability to systems where the huge number of possible combination of states can be analyzed. And sad as it is, if we want it to be reliable, the liberal individual approach to coding has to be abandoned in favour of a much more strict industrial way of thinking. Right now we have handcraft and we need industrial precision, standardization and quality.

Re:Complexity & Culture

[eipgam]

You obviously don't know much about traditional fields of engineering if you think they're simple.

Re:Complexity & Culture

[Anonymous Coward]

For all the similarities that you can infer about the design process for both software and traditional engineering, there is one big fundamental difference between the two. With traditional engineering you know how a piece of material, electrical circuit, pipeline, etc. is going to behave over a range of values. It is a continuous system. Plus you can touch it, see it, hear it, and smell it. For software engineering you only know (with limited certainty) how a piece of software is going to behave at individual points of the system - those points in the system that you can actually test. It is a discrete system.

Re:Complexity & Culture

[denoir]

Well, I'm originally an electronics engineer and that field is orders of magnitude more organized and structured than software engineering. Ultimately I ended up in software and I think that have a reasonably good view of both fields. Admittedly, I've only led groups of at most a dozen of programmers or so, but I doubt that the problems go away when you increase the number of contributors to a project. A good example of how increasingly bad things become with project complexity are the horror stories coming out from Microsoft about their Vista development (QC) troubles.

EE is intrinsically simpler because it is based primarily on physics - which offers simple, computable models. There are orders of magnitude fewer degrees of freedom. And the freedoms that exist are far more bounded.

If anything this is evidenced by the fact that most modern electronics design is actually made by computers. Human design is today at a fairly high meta-level. Software is far away from this.

As for traditional engineering being "simple" - those are your words not mine. Next time reading the text before attacking it might be a good idea.

Re:Complexity & Culture

[clickclickdrone]

Another issue is that in many cases, modern development is using third party components, be they UI controls, OS calls, database features etc. These are effectively blackboxes so if there's something dodgy in there, you could waste days or weeks trying to pin down a bug.
I heard of a recent case where a bank had a mainframe process, which had just gone live, kept abending. For 3 days they threw people at the problem, managers blaming the coders and testers for letting something so important fail. It turned out to be a new version of the compiler had been put on the mainframe that had a bug not in the version on the dev and test systems.
It those sort of ass-biting issues that can knock estimates right out the window.

Re:Complexity & Culture

[jsight]

We do have well-constructed systems, such as the software on-board an airplane - but they are laughably primitive (because you have to account for all possible states the software can get in to).

I don't think you've seen too many modern airplanes then. They don't tend to be "laughably primitive" any more. :)

If it was easy

[dantal]

I would be out of a job ;)

Re:The answer is easy

[DavidR1991]

I would reeeally like to see your code. Seriously. Then we could have a flamebait style code-review just like your post, where we humiliate your coding skills (if existent in any form at all)

Engineering and Education

[TrappedByMyself]

What is the pedigree required to build a large bridge, a skyscraper, a rocket, or to perform brain surgery?
Right now software engineering is such a young field. Success isn't dependent so much on process and education as it is on the skill and intuition of individual developers. And there are a heck of alot of software developers out there with neither education nor skill
Software engineering is still becoming it's own thing. I mean, we still keep trying to compare building software to building buildings. We're still pounding things with rocks, but the difference between software and things in the past is that it's so easy to get something up quickly that is moderately useful.

We'll get there.

Re:Engineering and Education

[Javagator]

we still keep trying to compare building software to building buildings

I read a book about the proof of Fermat's Last Theorem a few years ago and it reminded me of software development. First some brilliant mathematician proves a theorem that no one has ever proved before. Then the proof is circulated and an army of mathematicians look for mistakes and bad assumptions. The mistakes are corrected and if the assumptions can be shown to be true, then the proof succeeds, otherwise it fails. This is like software development except for the brilliant part.

Part of the problem

[grasshoppa]

Part of the problem is this; When asked if we ( software developers ) can do something, the answer is typically yes; Of course we can do something. Given enough time and money, we can write software to accomplish just about anything. The problem lies in that we may not have experience in that field, or we do but this request is different in a way to make it difficult.

Software development is a lot like engineering, only without the hundreds of years of experience to pull from. And it's a lot like janitorial work, only without the experience to draw from there as well. Managers, marketting ( and schools really ) like to see it like a burger to be flipped.

It's simple

[ozzee]

When you ask a carpenter "are you done ?", almost anyone can verify the answer just by looking at it.

When you ask a programmer "are you done ?", it is difficult to verify the answer.

There are so many ways to accomplish a goal in software that you need a number of very expensive controls in place to validate the result. Very few managers know how to do it and very few C*O's are willing to spend the money to put in place even the minimal control measures even though I can say in every case where ther were good controls put in place the benefits far outweighed the costs.

This leads to a plethora of problems that I won't repeat here but I know too well.

Eventually everything breaks down when the C*O's start making technical decisions that they are not qualified to make and it's all downhill from there.

Re:It's simple

[Tankko]

>>When you ask a carpenter "are you done ?", almost anyone can verify the answer just by looking at it.

You've never had a house built, have you. :-)

Re:It's simple

[sesshomaru]

"So, what I'd like you to do is carve me a cross between the Sistene Chapel and Notre Dame Cathedral out of wood, well not exactly but in that realm you understand, and make it out of this kind of wood that we got from the vendor with the best salesman. Oh, and unfortunately we bought hammers but not carving tools, but I'm sure you'll figure it out. What? Oh no, you can't use your own carving tools or any of those GPL carving tools they aren't authorized. Oh? You're asking, 'Should it be a scale model or full size?' Good question, I'll get back to you on that but you'd better start making it in the mean time. Well, get going the schedule on this is fairly agressive."

Re:It's simple

[arthurpaliden]

Or how about:

I want you to turn this Volkswagen Beatle into a Porsche and no we cannot stop the Beatle you have to work on it while it is moving and no we cannot afford to buy another beatle to test stuff on. Oh and by the way you have to use only VW parts.

Re:It's simple

[PitaBred]

Using VW parts on Paul McCartney? What do you do, just shove them up an open orifice? Seems a little medieval to me...

A Beetle is the bug or the car. A Beatle was a member of the most famous band the western world has ever known.

Re:It's simple

[pipingguy]

When you ask a carpenter "are you done ?", almost anyone can verify the answer just by looking at it. When you ask a programmer "are you done ?", it is difficult to verify the answer.

This reminds me of process plant design. It's fairly obvious when "it's done" and constructed, but will it work and not explode when started up? And yes, today's process plants use a lot of software and electronics that provide finer control than we had even as recently as 20 years ago, not to mention that there are a lot of legacy facilities that are retro-fitted.

Re:It's simple

[clickclickdrone]

Amen to that. I've worked on RAD projects where we've dummied up a screen to get user feedback and they say 'Looks great - about a week to finish it?' when we're probably looking at 6 months of development time.

Re:It's simple

[cmat]

If you'll pardon the different point of view :)

--

When you ask a carpenter "are you done ?", no one can verify the answer just by looking at it without knowing fist what he is supposed to be building.

When you ask a programmer "are you done ?", it is possible to verify the answer by simply using what he's built.

There are so many ways to accomplish the primary goal in business (to exchange something for something else) but you need basic arithmetic to validate the result. Very few programmers take the time to learn how to do it however many managers are able to accomplish this.

Eventually everything breaks down when the programmers start making business decisions that they are not qualified to make and it's all downhill from there.

Re:It's simple

[ozzee]

When you ask a programmer "are you done ?", it is possible to verify the answer by simply using what he's built.

In many cases yes, in some cases no. Often the level of specification does not go into the level that is needed to specify everything and so lingering and expensive issues may remain and no-one will be able to tell until the most important customer ends up using the product and it goes down in flames.

As for programmers making business decisions, they're paid to do that within the scope of their job. I have never seen a programmer ink a business deal or choose a bank or leasing company. However, I have seen too many C*O's create an overconstrained situation where there is no winnable result.

Mod parent up!

[EmbeddedJanitor]

While the comment might just sound like a silly grab for a funny, I think this is a very valid point. We really have far too many developers who were not intended by the Intelligent Designer to be so. Software quality and robustness etc (which impacts on delivery schedules etc) are dictated mainly by the lowest performer in the group rather than the highest performer.

There's a whole industry of dev tools out there trying to convince management that SilverBullet will make your software come out on time, or make your crappy programmers more productive. I don't think so.

What we really need is to promote the concept of the CodeSmith like a balcksmish, silversmith or whatever, coding is a skilled artisan occupation. Instead of trying to keep good coders coding, most organisations try promoting them and making them managers. Eventually you're left with just the dregs coding.

Re:Mod parent up!

[Anonymous Coward]

What is the hell is a "balcksmish"?

Re:Mod parent up!

[tepples]

What is the hell is a "balcksmish"?

Typo for blacksmith. (No connection to Will Smith, right?)

Re:Mod parent up!

[Beryllium Sphere(tm)]

>Instead of trying to keep good coders coding, most organisations try promoting them and making them managers.

Whereupon it turns out that they have exactly the wrong skill set to be managers and things get even worse.

I once saw a deep-thinking and productive coder promoted to management. The result was a catastrophe of Biblical proportions. In terms of morale alone, when employees got together to compare notes they found that they'd independently been thinking of where to take cover if the guy went postal.

Re:Mod parent up!

[FecesFlingingRhesus]

Actually,
Though the comment was party in jest, though I do believe that it is a prevailing issue. I believe there are far too many developers in the industry that are not competent in their ability to write quality software. This is evident in many developers adhering to patterns for the sake of using pattern when the patterns do not fit their needed problem set. Web programming is saturated with this kind of mentality; this is evident in new frameworks coming out weekly that rehash the same old problems and poorly I might add. Your comment about development being an artisan type endeavor might be more true than anyone wants to admit. The business side of the industry is scared of the idea that software may not be able to be manufactured. Crafting conjures the realization that each peace is unique and unlike an assembly line, carries unpredictability and risk as it cannot be quantified until it is completed. This takes away businesses ability to look like they are in control and masters of the software domain and makes people who otherwise feel as thought they are all controlling realize that in fact they have no control nor can they even begin to comprehend what needs to be done to control software development. Yet everyone goes on quietly knowing this and not saying anything because no one wants to look like a jackass. So the business goes on acting as though they are in control while not treating the process as a creative one because no one wants to admit that software cannot be manufactured. The sooner companies realize this, the better off the industry as a whole will be. Developers will be looked at as skilled contributors and not an cog that are replicable by the next guy with the same skills on the resume, Timelines will be more realistic and business personnel will realize that in fact they have no freaking clue how to manage a development environment and that they had better get someone who does know.

Re:Mod parent up!

[mrchaotica]

What we really need is to promote the concept of the CodeSmith like a balcksmish, silversmith or whatever, coding is a skilled artisan occupation.

Or promote the concept of the title "Software Engineer" requiring a professional certification, the way real engineering does. Any slob off the street can't suddenly proclaim himself a "Professional Civil Engineer" and start building skyscrapers; why should he be allowed to proclaim himself a "Professional Software Engineer" and start coding medical databases?

Re:Mod parent up!

[Javagator]

This is one reason software development is hard. Most of us can figure out what a "balcksmish" is, but the compiler can't figure out that

if (x = 0)

Should be

if (x == 0)

Coding is inherently error prone. Even the very best programmers make stupid mistakes.

Re:Mod parent up!

[Aladrin]

You know, despite the fact that I'm a 'Software Engineer' without 'certification' (I have a 2 yr degree. Whoopee!) I agree with you. People doing work that is THAT dangerous SHOULD have papers to certify them as being competent enough to handle it. But where do you draw the line? Programming games definitely doesn't need that. Hospitals definitely should have it. Courtrooms? Police Stations? Fire Stations? There's a lot of grey-area.

Civil Engineers, though... Every single thing they design has the possibility to take a human life if they do it wrong.

Re:Mod parent up!

[SuluSulu]

Don't forget that if you want to have a Codesmith you have to start with an apprentice, and that means hireing people with very little or no real world experience and putting them on projects where they can learn.

Re:Mod parent up!

[jc42]

Or promote the concept of the title "Software Engineer" requiring a professional certification, the way real engineering does.

Actually, this has been attempted repeatedly in the software business. It has always failed, for pretty much the same two reasons.

First, the only thing that the organizers (invariably software managers) can agree on is testing for knowledge of the mundane introductory-level details of IBM/Microsoft software. This means that in the more technical areas, where other (higher-quality) systems are used, there is no certification.

Second, even if the customer wants software that runs on IBM/Microsoft systems, they quickly realize that that "mundane introductory-level details" part can be rephrased as "It's a totally Mickey-Mouse certification", and the people who get certified tend to be the novices who think that a certification will get them the big bucks that their knowledge doesn't qualify them for.

So in software, "professional certification" always turns out to be a big joke. Everyone knows this, but pretends that a good certification program would help.

And no, I don't know how to make a non-joke certification program, either, despite several decades of building software. If I did, I certainly wouldn't be hired to take part in the committee that manages the certification program. (But I'm disqualified anyway, because I've worked too much on non-IBM, non-Microsoft systems to be considered for a position in a certification committee. ;-)

Re:Mod parent up!

[Jerry Coffin]

...promote the concept of the title "Software Engineer" requiring a professional certification, the way real engineering does.

For this to do any good, you'd need reasonable criteria upon which to base the certification. IMO, TFA was mostly correct in pointing out that right now, we don't know what those criteria should be. The Mythical Man-Month remains relevant today largely because relatively little progress has been made in solving the problems it explored. As Brooks pointed out in No Silver Bullet, we've cured a lot of the accidental problems of software engineering, but done relatively little about the essential problems.

In terms of the certification itself, we're just about as lost. The correlation between formal schooling and success as a software engineer is weak at best, and quite possibly negative. Experience indicates more, but still very little. Certification based on such criteria would do more harm than good.

There are other problems as well. With buildings, it's pretty easy for most people to realize that "skyscraper" and "dog house" have different requirements as to the level of expertise involved in their design. In software those differences often aren't nearly so obvious. The managerial people who'd need to enforce the requirement for certification don't know the difference between a skyscraper and a scale model of one for a train set (and sometimes the developers don't either). "Web developers" (in particular) seem prone to jumping (metaphorically) from doll houses to 18-lane suspension bridges in high-wind, earthquake-prone areas.

Re:Mod parent up!

[mrchaotica]

I used civil engineering as an example because that's my particular field. However, AFAIK all kinds of engineering (i.e., stuff like electrical engineering too) have professional certifications. And even if you don't bother to get your P.E. (Professional Engineer) certification, pretty much everybody needs to at least be an E.I.T (Engineer In Training), which requires passing the Fundamentals of Engineering exam. So yes, even projects that aren't dangerous enough to warrant a P.E. still have certified E.I.T.s working on them.

Re:Mod parent up!

[alienmole]

What is the hell is a "balcksmish"?

It's from Lord of the Rings. Deep in the Mines of Moria live horrors which man was never meant to meet: Balrogs, Balcksmishs, and worse.

The Balcksmishs were originally Maiar, of the same order as Sauron, Saruman and Gandalf, but they became seduced by Morgoth, who corrupted them to his service in the days of his splendour before the creation of Arda. During the First Age, they were among the most feared of Morgoth's forces. When his fortress of Utumno was destroyed by the Valar, they fled and lurked in the pits of Angband. Fighting alongside the Balrogs, the Balcksmishs were nearly destroyed at the end of the First Age. Unlike Balrogs, Balcksmishs have real wings -- much of the muddle about Balrog wings comes from confusion with Balcksmishs and the related Balckribers, which the Nazgul sometimes style themselves after. Sheesh, weren't you paying attention during the movie?

Re:Mod parent up!

[Kjella]

Luckily, any decent compiler will warn you on that (and those who ignore warnings should be summarily executed, they're there for a reason).

What I've found hard about software development is that there's a thousand ways to do things, and if you look at someone else's code it's going to be different. Not just a little bit different, but to the point of "what the fuck is going on" different. Sometimes it's easier to just redo the work than to understand it. I was taking over a report which contained a piece of code to select months or quarters and it was full of dateadds and datediffs and godness knows what else - a mess. I rewrote it as a 10x simpler, easy to read statement which anybody who saw could verify with a glance was correct. For a major project, there are as many ways to structure it as there are projects. I've looked at design patterns up, down and sideways and it always comes down to a personal mix-and-match. Which is great if you have 20 years of "feel" for what works and what's needed, but it doesn't do squat to ensure you and the last guy or next guy has similar opinions.

Re:Mod parent up!

[Anonymous Coward]

Making matters worse is the distressing tendency for one doghouse to scale quite nicely to handle sky-scraper sized problem spaces (despite the fact that it was never intended to do so), and for the next one over to collapse killing all occupants when three small dogs wander in... and yet both doghouses look remarkably similar.

Re:Mod parent up!

[Rakshasa Taisab]

Lucky for you, GCC does know it should ask you to make it either "if (x == 0)" or "if ((x = 0))".

On the other hand, I have no idea what "balticsmash" is supposed to be.

Re:Mod parent up!

[Anonymous Coward]

As far as I can make out, the term "Software Engineer" is pretty much only used in the US. Here in the UK the term is usually "Software Developer". Let's say you started to require certification for Software "Engineers": one of two things would happen: 1) A lot of software development would be off shored 2) A lot of non-certified people would begin calling themselves "Software Developers". In the long run, what difference would it make?

Re:Mod parent up!

[Aladrin]

I assume you used electrical engineering as an example that was relatively safe in comparison to civil engineering... But wiring a house wrong could EASILY kill someone in multiple ways. I can't think of anything that any kind of physical engineer could do that does not fall into the class of 'could hurt someone badly'.

If D&D: Tales of Boredom crashes, nobody gets hurt. Ever.

Like I said, I definitely agree with the idea that anyone doing dangerous programming work having to prove they are capable, but most programmers do not fall into that category.

Re:Mod parent up!

[Anonymous Coward]

They were crappy programmers before that guy went into management; they were crappy programmers while he was a manager; and they are still crappy programmers now that he has left.

Re:Mod parent up!

[hotdiggitydawg]

• You do not need electrical engineering certification to wire up a house. See your local hardware store for all the requisite DIY equipment.
• Here in the UK you do not even need training to rewire mains plugs for appliances. Companies are required to supply equipment with plugs that can be opened by the user (possibly to change the fuse inside, I'm not sure of the reason why). Any muppet with a screwdriver can do it.
• If the software controlling the space shuttle crashes, people die.
• If an electrical engineering fault causes a cheap pocket calculator to crash, nobody gets hurt.

In essense I agree that dangerous work should be done by certified people, but I fail to see how anything else you said really illustrates that point. My point is that the certification requirement should be applied to the danger rather than the specific field of work, as most fields of work will cover many levels of danger.

Re:Mod parent up!

[Anonymous Coward]

You do realize that people who wire houses are usually not engineers, right? At least in the U.S., they're electricians. The professional engineers are the ones who design the lighting schemes, not the ones who install the fixtures.

I'm not saying that these people don't know what they're doing (I've known a lot of smart journeymen), but they're definitely not all certified professionals. I spent several summers running conduit, pulling wires, and installing devices/fixtures when I wasn't even an apprentice. And, in agreement with what you said, anyone doing very dangerous stuff (high voltage, for instance) has already shown they are capable.

Re:Mod parent up!

[Bastard of Subhumani]

Not all languages work like that. But am I the only person who thinks it's a bit of a dumb decision to design a language that way?

Re:Mod parent up!

[Simetrical]

Luckily, any decent compiler will warn you on that (and those who ignore warnings should be summarily executed, they're there for a reason).

Unfortunately, by that metric PHP does not have a good compiler. Python, of course, won't allow assignment in if conditions to begin with, which I think goes rather too far. Possibly my favorite solution is to use some special syntax for assignment, like := instead of just =. But the idea of getting in the habit of using 0 == x instead of x == 0 is a good one, definitely, although it doesn't work for comparison of two variables. (Actually, I believe 0 == $x is even faster than $x == 0 in PHP, although I could be wrong.)

Re:Mod parent up!

[Fulcrum of Evil]

Sure it can - it will warn you that (x=0) is an assignment, but will still compile because it's legal to do that. If you get in the habit of doing the weird (0=x) convention, then it won't compile and you've eliminated one error class.

Where software developers sell themselves short

[Digital_Quartz]

When you've contracted an engineer to design a bridge, and you want to make several large-scale changes to that bridge, the engineer will come back and say "If you want these changes done, this is how long it will take. If I don't have that much time, I can't make the changes".

In fact, I'd go a step further; software developers tend to say "This is how long it will take to make the change, and this is how long it will take for me to hack something together." Bridge engineers don't say things like that. They don't put that "hack something unsafe together" option out there on the table, and neither should we.

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

Re:Where software developers sell themselves short

[Cro Magnon]

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

It's not just that we put "unsafe" software out there. It's that our bosses expect us to put unsafe software out there. I try to test everything, but when the requirements are changing long after we SHOULD be in beta, and the boss wants it out by such-and-such a date, something has to give, and often that something is proper testing.

Re:Where software developers sell themselves short

[kthejoker]

The chief reason for this, though, is the lack of clear requirements on the initital project manager / architect's part, though.

When you're building a bridge, there's a base requirement: something can cross it. Now what can cross it, how fast, structural integrity, etc, all play a factor in the specifics, but at the end of the day, either something can cross the bridge (safely / quickly / profitably) or it cannot.

If you were told to write a program to add two numbers together, and that was all, you'd have a pretty easy time rejecting "hacky" options, too - because you know all of the requirements.

But most of the time you don't, or you misunderstand them, or (in the classic case) you offer up something and the project suddenly takes a dramatic turn because you did one little neat thing in it and they want the whole system programmed around that tiny widget (at the expense of all others).

Comparing software to a bridge is useless. A bridge is like one specific program - a calculator, or a no-frills text editor. Building software is more like urban planning. Now go ask engineers how many times they've screwed the pooch on that one.

Re:Where software developers sell themselves short

[0xABADC0DA]

I would say the fact that a bridge engineer can give a realistic estimate on how long it will take whereas the software engineer cannot, or does not, indicates that the bridge is easy in comparison.

Just for example, ask that same bridge engineer how long it will take to make the entire bridge out of a single piece of plastic. Now suddenly it's a much more difficult problem. How do you support the form as it is molding? How much plastic do you need to for integrity? How do you cast something that large at once? The answer will start with "uhh...". Now hold him to the estimate and see what kind of bridge you get. This is more what software development is actually like.

Re:Where software developers sell themselves short

[Beryllium Sphere(tm)]

>If you were told to write a program to add two numbers together, and that was all, you'd have a pretty easy time rejecting "hacky" options, too - because you know all of the requirements.

Can the sum of the numbers exceed MAXINT? MAXINT on what platform? Should overflows set a flag, throw an exception, or be reported in-band?

Can the numbers be floating point? If one is much smaller than the other, how large a fraction of the smaller number's precision are you willing to lose?

Can the numbers be multiple-precision? Is the multiple-precision library you're using compatible with the rest of your system? Does it fit into your memory requirements?

Do the numbers come from some real-world source that need to be checked for sanity, for example the output of a sensor?

In what form does the output need to be? Encoded in ASN.1? BCD? IBM "unpacked decimal"? String? If output is variable length, is there a maximum size that must be enforced to prevent buffer overflows? If the maximum size is exceeded, should the result be truncated, or should the function throw an exception, or ...

The sickening thing being that I've probably left out some important issues. Like thread safety.

Look up the bug history of "IEFBR14", originally a single return instruction that required four or five revisions to meet shifting requirements.

Re:Where software developers sell themselves short

[professorfalcon]

Agreed. If the software fails, the developer should get tasered. But just a little bit.

Re:Where software developers sell themselves short

[Anonymous Coward]

You can ship an automatic update far a software project after it's already in customer's hands. That is sort of hard to do with a bridge. The main difference between software development and other engineering disciplines is that software is never finished. Hence the tendency to go with the "ship early and ship often" approach -- after all, it works for Microsoft, doesn't it?

Re:Where software developers sell themselves short

[TheViewFromTheGround]

Mod parent +6 insightful. This has to be the most succint and true statement of the lack of accountability in software development I've ever seen.

Re:Where software developers sell themselves short

[Grishnakh]

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

You seem to be missing something: software engineers are not "professionals" in the sense that civil engineers are, and they're not licensed. There's no liability to a "software engineer" if anything happens, only to his employer.

You can't say "no" to your boss and expect to keep your job long. Bridge-building engineers don't have that problem because they ARE the boss, just like doctors don't work for bosses as they themselves are the boss.

You want to make software engineers accountable for their work? You'll need to mandate government licensing, and then you'll need to pay those developers 2x-3x as much because they'll have to have malpractice insurance. Somehow I don't think any software companies want this.

Re:Where software developers sell themselves short

[rk]

"Comparing software to a bridge is useless. A bridge is like one specific program - a calculator, or a no-frills text editor. Building software is more like urban planning. Now go ask engineers how many times they've screwed the pooch on that one."

Spot. On. The construction metaphors applied to software systems always bothered me for this reason, which I could never put into words as clearly and succinctly as you have right here.

Re:Where software developers sell themselves short

[belmolis]

To add to your point, what about the input format? Are they going to come from text files or user input in the form of strings? Can you assume ASCII decimal standard format? What about scientific notation? Can they be in hex, octal, or binary? What about numbers in Chinese or Gurmukhi or Thai?

Re:Where software developers sell themselves short

[raduf]

"and neither should we"?

Well, how about the competition? You know, the one that says, ok, we'll hack something right up. And unfortunately, in most cases the hack works.

No, the "hack" option is very much on the table. The thing is, right now the software industry WORKS. True, there is way to much software to be done and too little good programmers. True, most development has turned into training code monkeys as quickly and cheaply as possible and letting them follow (better or worse) specifications. In a perfect world all software developers would be either herdened profesionals or young aprentices, but in the real world, most don't have any talent whatsoever. It just pays good.

And things probably won't change, not in this respect. It becomes easier and easier to build software, so there is more and more reason to hire cheaply and quickly. I don't really know where the change will come from. I do know that software is needed, it's hard to build and harder to automate, so things will probably limp for several more decades. But in the end the big technical hurdle (how to LINK/SEPARATE UI from logic) will be solved, and then... many programmers will lose jobs. But it's still in the distant future.

Re:Where software developers sell themselves short

[tknd]

Software is probably the only industry where you can get paid to fix your mistakes.

Re:Where software developers sell themselves short

[Anonymous Coward]

>It's that our bosses expect us to put unsafe software out there.

If you were behaving like a professional, then your ethics would prevent you from releasing unsafe software regardless of what your bosses think about. You can go to let your boss know about that, document it, go to P.E. or whoever has the sign off authority (even to the legal department!). All else fail, you can alway quit.

This is the part that you do not get.

Re:Where software developers sell themselves short

[Tablizer]

The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

I usually tell the manager or customer what the tradeoffs are. If they want a quick hack to reach a deadline, that is their decision. If I delay past the deadline to "do it right" without permission, then I would be "insubordinate" and get "bad grades" or fired.

Perhaps if people died when software was not right like it does with bridges, things might change.

There is a saying: soon, good, cheap: pick 2 out of 3. You see, as peons, we usually don't get to make that choice.

Re:Where software developers sell themselves short

[LQ]

Bridge engineers don't say things like that. They don't put that "hack something unsafe together" option out there on the table, and neither should we.

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.

Like most things in life, it boils down to money. The vast majority of programmers are in the wrong line of work and are only in it for the cash. There is just too much demand for software and not enough supply of good software engineers.

Re:Where software developers sell themselves short

[Digital_Quartz]

There's no liability to the software designer's employer, either, or next to none.

As the old joke goes, Microsoft made IIS susceptible to worms like Melissa and Code Red, costing billions of dollars to companies around the world, and didn't have to spend a dime in restitution, while Intel made the Pentium with a floating point error that affected a handful of people doing extremely precise simulation work, and had to spend a small fortune recalling chips.

Even if we just had software liability, we'd see a much greater focus on quality in our industry. Although we tread a thin line trying to introduce that concept without having horrendous effect on open source.

Re:Where software developers sell themselves short

[figa]

See this FAQ for how hard decimal arithmetic really is. Few programmers understand this.

Re:Where software developers sell themselves short

[Grishnakh]

There's no liability to the software designer's employer, either, or next to none.

No, not right now, but that's really beside the point I was trying to make, which is that the way things are structured right now, software engineers are just paid employees, just like janitors. If a janitor doesn't clean up a spill in a bathroom and someone falls, the janitor isn't liable, only his employer. The janitor might lose his job, but big deal, he'll just get another one; his employer has to spend millions in a settlement to the guy who slipped and cracked his skull on a sink. It's the same with Intel in your example: the hardware engineers who designed the Pentium didn't have to pay a fortune recalling the chips they badly designed. With licensed engineers like those who design bridges, they're personally liable for their mistakes.

As the old joke goes, Microsoft made IIS susceptible to worms like Melissa and Code Red, costing billions of dollars to companies around the world,

That's because all those stupid companies willingly paid MS for their crapware, even after knowing the license agreement absolved MS of any liability. They didn't have to do that. If some piece of medical equipment has a software failure and someone dies, do you think the company who made that equipment can just say "too bad"? No way, they're liable for that death (whether they wrote the software themselves, or it was contracted out). There are industries where software companies are indeed liable for their products. Think aviation, medicine, military, etc.

Even if we just had software liability, we'd see a much greater focus on quality in our industry. Although we tread a thin line trying to introduce that concept without having horrendous effect on open source.

We can have liability if customers demand that from their vendors. I think the way things are now is fine: if people/companies are too cheap to demand liability (which carries higher costs of course--see the aforementioned medical equipment: $$$), then they can suffer with lower-quality software. As for open source, that's another choice: I don't have any illusions about my open-source desktop being NASA-grade-reliable, but then again I didn't pay anything for it either so I'm OK with that. What's more, it seems to me that the OSS community is proving that the OSS process produces quality superior to that produced by no-liability proprietary software companies at much less cost. It's not up to the bar set by truly mission-critical software (yet), like that provided by some embedded vendors, but it's getting closer all the time, and may even reach that.

Re:Where software developers sell themselves short

[Reservoir Penguin]

I hope the time when software developers are treated as engineers never comes. A license required to use a compiler and write some free software? NO WAY!

Mommy, rocket science is too hard.

[Anonymous Coward]

Complex subjects are... ummm... complex.

Every day there is more software created that makes creating new software easier. We're still in the toddler phases of computing. It will get better as computers get faster and have more storage. Even if you combined all the computing power in the world it's still pitifully slow in the grand scheme of things. Compare biological functions for example and the amount of computing power it takes to model even a single atom (and that's without even modeling the quantum state... and whatever is smaller than the quantum level...etc).

Re:Oddly...

[HomelessInLaJolla]

As a medicinal chemist I second this.

It's not that complicated...

[Gunark]

The answer to this question isn't complicated. Software development is hard because it requires that the developers impose an artificial abstraction on the real world -- a real world that could be conceptualized in an infinite number of ways.

Once you decide on some particular conceptualization, you make certain assumptions about what is and isn't possible. This is the root cause of most if not all of developers' headaches. Despite our best intentions, we find that our carefully engineered abstraction does not capture everything about the real world relevant to the given task (either because we got it wrong the first time, or because the task itself has changed).

This, in fact, is a deep problem in computation in general -- a huge obstacle in artificial intelligence, for example. We need an algorithm that can pick out the salient aspects of a problem and ignore the irrelevant ones. Until the day that we have this (and that day may never come) the hard part of software development will remain an art form based on intuition and creativity.

Re:It's not that complicated...

[Mean Variance]

The answer to this question isn't complicated. Software development is hard because it requires that the developers impose an artificial abstraction on the real world -- a real world that could be conceptualized in an infinite number of ways.

And then give management an "estimate" for how long it will take you to do your part of implementing this little abstraction ... and make sure it's close the time that management thinks it should take.

Therein lies the rub. You can do this little dance in many software building scenarios using shortcuts and programming practices that make the coder (engineer, whatever) wince. It'll get fixed up later. As long as no one dies using your software, the bag of hacks and tricks to get things done on manager's time are endless.

...because planning isn't doing and we want to do!

[jofny]

I see more projects go bad because of poor communication and lack of architectural efforts than anything else.
A sufficiently detailed question needs no answer. Likewise, with software development (and really most computer systems development), everyone wants to rush out and start coding as fast as possible...but if they'd made sure there were no assumptions, defined their problem specifically, and spent way more time than they wanted to developing an architecture, the coding afterwards would just be data entry for the most part.
But...people tend to be averse to taking the up-front costs of not only gathering their requirements, but also -understanding them- and making them -fit together-. Frameworks and Architectures tend to be poorly developed and understood ahead of time. Features are developed in isolation from each other. Test cases need to be developed at every step to keep people in agreement and to make find issues early. If your project isn't working in a manner where every new addition to it can't be tested as it's written, you're doing something wrong.
Requirements need to be reviewed and revised and reviewed and revised constantly with all staekholders against those test cases.
But...maybe I'm off base.

Who says it's hard?

[Animats]

Software development isn't really very hard today. At least not at the level at which 90% of programmers work. The tools mostly work, many of the languages are quite forgiving, there are plenty of books, you can get help from Google, and much code can be downloaded. Yes, there are people doing hard stuff - real time control, database internals, game physics, stuff like that. But ordinary web programming is almost a no-brainer today. Look at the people doing it.

And if you need to do something hard, it's amazing how much one person can get done today. It's a great time to be a serious programmer.

There are design problems, yes. Microsoft's world is overly complex because they need it to be complex; if it were straightforward, nobody would need Microsoft. The Linux/Unix world is overly complicated because it has too much legacy dating back to the 1970s. The CSS world is overly complicated because the approach to page layout was botched. (Layout is a constraint problem, but isn't being approached as one.)

What we do have is too much tolerance for the mess at the bottom. What we need is, say, a "Just Say No to Bad Software" from Fortune 1000 CIOs. As in "We're not buying Vista until this list of problems is dealt with".

Re:Who says it's hard?

[FlyingGuy]

Hmmmmm "Web & Programming" a new oxymoron!

A few reasons

[scdeimos]

Three reasons I see regularly that cause development not to be delivered "on time" and "on budget":

• Specification - specifications given to developers are always incomplete. Something always gets left out, from subtle behaviours to business logic for whole subsystems.
• Scheduling - although we're working on changing this, other people who have no hands-on development experience set the schedules. They either use some metric about lines-of-code-per-person-per-day that has no relevance in the real world, use a dart board, or just pull the figures out of their proverbial. (I'm thinking the last.)
• Integration - Green Fields development is always easier. Extending old systems is always a pain in the proverbial due to the lack of accurate documentation. Even the perfect documentation becomes imperfect if it isn't kept up to date (or nobody knows where it is).

There's heaps more, but this is my top three.

No such process

[camperdave]

I think that a big part of the reason that software development is difficult, is that people don't have, or don't follow a program development process. At what stage is the data definition and data flow analysis done? Before coding starts? After? Are there milestones in place, specific points at which progress is measured and design changes made? Are these planned for and documented? How is this program related to others? Is there code from other projects that can be re-used? What parts of this project can be re-used?

An old-timey programmer looks at this...

[troll]

We promise according to our hopes; we perform according to our fears.

Managers -- down to mid-level -- need to have dates, so the programmer gives the earliest possible date. The software goes out buggy, untested. More pressure on the programmer and management doesn't trust programmer estimates anymore.

Solution? None, with the current business model.

Re:An old-timey programmer looks at this...

[rastos1]

so the programmer gives the earliest possible date.

I give the latest date that I can get away with.

Re:An old-timey programmer looks at this...

[szelus]

Yeah... ...and it doesn't help either...

Short Answer

[955301]

Software is still a Science and not an Engineering practice.

As long as the design can also be the implementor and estimates based on actual analysis are optional, Software will NEVER be an engineering study.

These aren't changing quickly for what I believe are a few reasons:

IEEE has not created a Professional Engineer - Software and noone really wants them to.
Companies don't like to be told they have to hire something that sounds expensive to build something they cannot see.
Companies will NEVER open their software to outside inspection the way construction companies must open their buildings because of the concept (flawed, I think) of Intellectual Property.

If a company had to have their software inspected by a Software P.E. before using it in production or selling it to end users - If Software P.E.'s had to adhere to standards which included concrete estimates and testing - If companies were not allowed to use anyone they could find that has seen a computer to write their software... commercial software development would be much further ahead.

Do I believe any of this should happen? No. Why? Because I want it to continue to fail. I do not believe software development should be put on a pedestal or only performed by "experts". I believe shoot from the hip software projects allow open source software projects to exist and to succeed in the market.

Open source works without accurate estimates because the contributors can flock to good projects and don't have to adhere to a labour budget. Company employees can't get wind of the cool software project and leave the crappy one's - corporate structure and budget's won't let them.

I don't believe companies with more than 120-150 people are stable - once they breach this range empire building occurs and massive uncontrollable monsters result. If a company truly needs more than 150 people it should split into two and partner on the project at hand. I believe this is a human condition - humans work best in tribes where they can personally know all of the members.

All of this might be completely and utterly wrong. But it's my hypothesis.

Re:Short Answer

[Mr.Radar]

I don't believe companies with more than 120-150 people are stable - once they breach this range empire building occurs and massive uncontrollable monsters result. If a company truly needs more than 150 people it should split into two and partner on the project at hand. I believe this is a human condition - humans work best in tribes where they can personally know all of the members.

All of this might be completely and utterly wrong. But it's my hypothesis.

It's interesting that you should choose that number, because that's the number of people we can keep in our Monkeysphere.

Re:Short Answer

[toddhisattva]

Engineering is the application of materials sciences.
+
Software is not made of material.
=
There is no such thing as "Software Engineering."

Re:Short Answer

[Anonymous Coward]

"Companies don't like to be told they have to hire something that sounds expensive to build something they cannot see."

This is why code and functions needs to visualized in 3D desperately, so you can see in solid form what is actually happening.

Re:Short Answer

[955301]

That would certainly make it more suited for our senses and spacial recognition as humans.

Now, if only we could get the 3D rendering software done on time :)

No one knows ahead of time what to write

[scottsk]

What I've always seen in college classes and books on methodology is that there is some mythical "requirements" known in advance but in the real world I've never, ever seen any project which actually has known requirements and sticks to them over the project. I guess software development would be easy if that were the case. What usually happens is as the software is developed, requirements emerge and diverge and go amok during the time it takes to develop the software. Not true in other fields: When you've got a die or something you pour stuff into, you aren't going to change it after you make a few widgets. You're not going to port your die to Oracle after casting it for DB2. You're pretty much stuck, and if people don't like tail fins that year, you've got to start over from scratch. Software isn't like that, and any known methodology from engineering and science breaks down. The closest thing I've seen that works is some of the agile methods that say deliver as little as possible as soon as possible. But even that doesn't tell how to do it. What I have discovered is that the best way to meet changing requirements is to build applications with an object model that does the base functionality and a scripting language like Python to drive it. Then, as new requirements emerge, you can mold and shape the object model into new "applications" using the rapid-development scripting language. If the user wanted a desktop GUI and suddenly wants a web interface - if the user wanted a GUI but now wants it to run in batch - if the user wants something new no one thought of six months ago - whatever, the scripting language makes major changes much easier as long as they're not asking for big changes to the basic object model that does what the application does. (Or, you can add new stuff to the object model without breaking the working scripts.) Push the changeable stuff off into a scripting language that makes it really easy to change stuff, and write the guts of the app in your core language like Java or C++. I don't have a catch name for this approach, but it's worked remarkably well in practice.

The Truth

[carrier lost]

What Makes Software Development So Hard?

It's all that goddamned thinking you have to do.

MjM

Because software design isn't construction

[Todd Knarr]

Software design and development is more akin to an architect designing a building rather than the more common analogy of building the building once it's designed. Except that software developers are often burdened with requirements that any architect who valued his license would reject. For example, management often dictates which parts are to be used (ie. "We're going to use MS SQL Server as the database engine."). What architect would design a skyscraper after having been ordered by the client to use pine 2x4s instead of steel beams, or design a 1-story residential house under the requirement that he use titanium box beams instead of 2x4s? And then there's what the article notes at the top of page 3: software requirements change right up to the day of release. When an architect goes to design a building, the requirements are fixed before he starts. Square footage, height, number of floors, number of people each floor has to accomodate, number of elevators, number of toilets needed on each floor, how high the ceiling of the lobby floor will be, all that's fixed in stone before any design work begins. Sometimes that requires back-and-forth between the client and the architect to get everything clear, but it all happens before major design work begins. No architect's going to design the foundations of a building until he knows exactly (or at least very very closely) how much mass is going to have to sit on those foundations and how much horizontal shear they're going to have to handle. If the client can't decide what he wants, the architect just goes "Fine, call me when you decide what you want.". And even when this kind of analysis is done, software requirements change constantly after design's started. See above, no architect's going to accept the client changing the number of floors or the square footage of floors without also agreeing to a complete redesign to accomodate the changes with all the delays and additional costs that requires. If the client didn't like it, the architect would just hand the client the work to date and tell him to have a nice day. And no there wouldn't be any refund of money, the architect's held up his part of the deal as best the client will let him.

And then there's another parallel: architecture is only half engineering, the other half is art. Every building is different, and it's accepted that the architect's going to have to have time to come up with the parts that aren't off-the-shelf standard. There isn't a single standard floor-plan for a single-family 3-bedroom house, there aren't any rules that can be mechanically applied to create a floor-plan, and it's accepted that the process of creating a floor-plan is more creative than anything else and people without the knack for it just aren't going to be very good at it. And that on the next single-family 3-bedroom house much of that work's going to have to be done over again because the new client doesn't want the same thing the previous client wanted.

Re:Because software design isn't construction

[rrohbeck]

In architecture, the transition from the artistic to the engineering phase is well defined because the engineering discipline uses a mature restricted set of components. E.g. the 2x4. Now imagine if one of your suppliers came in and told you or your management or even your customer "look, we have this great new design, a 2x3.8! It'll save you 5% in material, and through our new and improved design, we have the same strength as the old stuff! However, you can drill holes only in specific locations, and only when the moon is half full."

Components and subsystems (middleware, runtime, OS, firmware...) du jour that aren't fully specified is where the difference lies. If we used the same set of components that we learned in college we would be totally confident in applying them after a couple of years (heck I can do POSIX C in my sleep but who wants that these days?) But the lifetime of a specific SW/FW revision is generally measured in months, and in a few years you are through a number of major revisions or on a completely different project or product. If our product life cycles were measured in decades the disciplines could settle down to "real" engineering.

It's basically Moore's law at fault.

Re:Because software design isn't construction

[cspariah]

You'd think that this sort of thing -- changing requirements and demanding that the cost/time not change -- wouldn't happen to architects. But sadly I can tell you that it does. My stepfather is an architect. He spends a lot of time in court because of these types of situations.

Re:Because software design isn't construction

[BrotherZeoff]

I'm not exactly sure why so many people get architecture confused with structural engineering. Architects have a lot of freedom, and in fact do help the owner decide upon number of stories, square footage, elevators, etc., as well as help decide what it looks like. Structural engineers consult during this phase (helping tell them what is within the realm of possibility and giving cost estimates) as well as design the structural members to fit within the building's shell. Structural engineers design foundations, slabs, columns, beams, and roof members.

Re:Because software design isn't construction

[Anonymous Coward]

I'm an Architect. As in, building Architect. And I gotta say that you're actually way off-base here.

"Except that software developers are often burdened with requirements that any architect who valued his license would reject."

Right. Because we don't have to deal with Building Codes, Zoning Laws, or Neiborhood review.

"When an architect goes to design a building, the requirements are fixed before he starts."

I wish. They are constantly changing, just like everything else, even right up to the day they cut the ribbon on the door sometimes.

"See above, no architect's going to accept the client changing the number of floors or the square footage of floors without also agreeing to a complete redesign to accommodate the changes with all the delays and additional costs that requires."

An Architect that totally rejects client's demands is an Architect that looses the job. Period. The two worlds aren't as different as you think, except our stuff has to work or we get sued, where y'all get away with murder.

"And then there's another parallel: architecture is only half engineering, the other half is art. Every building is different, and it's accepted that the architect's going to have to have time to come up with the parts that aren't off-the-shelf standard."

Unlike cars, airplanes, and software, which are largely made using totally custom individually designed compoenets but then are mass produced; Buildings are the opposite, they are totally custom one-of-a-kind products that are largely made of off-the-shelf standard items. Yes, how those parts are put together is 'art' but the part themselves actually are pretty boring, normal, and common. Like the great example above in the comments, talking about 2x4's that are '2x3.8s', the 'stuff' we deal with isn't as 'custom' as you might think...

"And that on the next single-family 3-bedroom house much of that work's going to have to be done over again because the new client doesn't want the same thing the previous client wanted."

Seems like to me you gotta get out more and talk to some real Architects. The Fountainhead was fiction, you know...

Re:Because software design isn't construction

[carpeweb]

I agree with the mod-ups, but I still have some serious issues with your analogy:

When an architect goes to design a building, the requirements are fixed before he starts. ... Sometimes that requires back-and-forth between the client and the architect to get everything clear, but it all happens before major design work begins.

IANAA, but that just doesn't sound accurate, especially when dealing with bigger projects. I think the architect (in reality, a team of architects) usually spends a great deal of time and effort working with the client to define and revise requirements. How else could it get done? The client has no expertise but knows he wants a great big building. He knows he wants things like a big atrium or skylights or modular offices or ... The [good] architect knows a lot about not only what features are possible but what they cost and good ways to arrange them (like the things you mentioned) But, the requirements definition process needs a lot of interplay between the architect and the client -- and I think your analogy more or less supports this, so I don't think we're in much of a disagreement. I just think you oversimplified your analogy because we'd all like things to be linear. I think meaningful architecture probably always requires back-and-forth between the client and the architect to get everything clear, and that is an essential part of "major design work", at least in terms of value-added.

If the client can't decide what he wants, the architect just goes "Fine, call me when you decide what you want.".

Well, in my experience, any professional (from architect to plumber) who takes this attitude rather than soliciting input and guiding the client, usually doesn't get that call later. I know they never get that call from me.

There isn't a single standard floor-plan for a single-family 3-bedroom house

Actually, there are tons of cookie-cutter floorplans for single-family 3-bedroom homes, which is part of what has driven the rise of large-scale home builders in the U.S. It's no longer a cottage industry (sorry, intentional), and in fact the widespread availability of standard plans is a positive factor imho (lowers cost). However, I think the single-family 3-bedroom home is the analogy for the "easy" development project, not for the hard ones. For example, a client who wants an intranet that has discussion forums, shared links, maybe a few other similar features ... can choose from a ton of off-the-shelf content management systems ("portals", or whatever the hell anyone wants to call them ... still lots of cookie-cutter systems available). Yes, I know that the client might also want features not available in the off-the-shelf systems, but that's like the home buyer who wants a marble kitchen island and an oversize master bath suite ... and the analogy moves more toward the big project. We just don't hear about the easy IT projects, because they're, well, easy.

You're still dead-on balls accurate about the way IT requirements change after design.

Re:Because software design isn't construction

[jbengt]

The problems of large projects are not limited to software.

I'm a mechanical engineer in the construction industry and I can assure you that the architectural requirements are not fixed before the design is started, I'd consider myself lucky if the requirements were fixed after we issued finished plans for bids.

Also, the foundation is often designed (for large projects, by a structural engineer, not the architect) before the engineer knows exactly what loads will be required by the final design. Like most things in any large project, designs are almost continually refined as the project progresses.

In the last 25 years everything has gotten quicker, but hardly more productive. It used to be that people would plan ahead a little bit, and take changes seriously. But with the advent of the fax machine, it less common to plan ahead and more and more common for last minute requests. Then CADD made it "easy" to make drawing changes, so it became more and more common to have more frequent and more major changes thrown at us. Then e-mail made it easier to send those changes, and now it's not uncommon for us to get newly revised drawings the night before the project is due.

So I'm blaming it on all you software guys for enabling all these fast-paced changes and turning my work life into a series of emergencies.

Re:Because software design isn't construction

[pipingguy]

I think you're bang-on with your comments, FWIW.

Re:Because software design isn't construction

[inKubus]

They need to pay us a shitload more so I don't have to worry about doing my laundry or making food etc. Sort of like a General in an Army. Sales in my eyes will always be the foot soldiers. Put sales UNDER IT.

THe problem is that "business" is "making money", not doing things right. And so, it's only a slow evolution, even with well capitalized places like MS and GOOG (who are both really publishing companies, NOT software). Software really IS interactive art. Any usefulness it has is just a side benefit. Just like music or movies, if something's good, people pick it up and use it. If it's garbage, you can't force it down their throat. You make it better until they WANT to use it. Thus there are a lot of failed bands, bad movies and.. abandoned software projects. Of course, if you make the right app, you might just change the world.

Sure is a different way to look at things...life without sales.

Re:Because software design isn't construction

[Maltheus]

I'd agree that projects fail (or drag on forever) when non-tech types dictate what tools to use. I'd also say that's the single defining factor in whether I'm happy at my job or not. I've had many great jobs turn to crap because some distant manager decided that we should be using their new pet platform (untested, undocumented alpha platforms), even when we had something mature that was working great. I can't work the miracles everybody wants if they're gonna tie my hands and blindfold me. I can't tell you how tired I am of seeing this pattern play out again and again.

changing requirements

[mrcubehead]

Writing good software is hard because while the result is just a tool for a job, identifying the job can be very, very difficult. Sometimes it's like planning to create a square peg for a hole that may or may not be circular, and whose circumference might no longer be known in six to twelve months.

Evaluate at every stage

[digerati00]

Management should re-evaluate the project at every stage and choose 2 out of following 3 to make sure it gets done .... Quality , Time or Money.

For good Quality product to be delivered on Time - you need to make sure the Price is right!
For a good Quality product which Costs less - you need to make sure you have enough Time to develop it
For a Cheap and Quick product - you will have to compromise on Quality

Software development is hard because....

[Anonymous Coward]

... of all the hot babes running around the offices in miniskirts giving massages that make it tricky to type. That, and the martinis that keep spilling in the keyboard. Combine that with the constant parties, sailing trips, ski trips, etc and it's a wonder anything gets done.

What's that you say? It's not the .com bubble anymore? Glad I left software.

Re:Software development is hard because....

[Plutonite]

You left software to do what? Has anyone else here been a developer and moved on?

I ask because I plan to do this myself, but the prospects are daunting.

Re:Software development is hard because....

[xtracto]

You left software to do what? Has anyone else here been a developer and moved on?

I ask because I plan to do this myself, but the prospects are daunting.

Well, I think you can answer that question yourself. You should look for things (outside the computer) that you like to do. Me for example, I started to cook as a hobby and I am planning on starting a restaurant (might even start a REAL [no texmex] Mexican food restaurant in the UK).

And then, you could use your "computing" knowledge to help you in your new job (wannalearn.com).

Gap between computer science and person problems

[lawpoop]

I believe that these days there is a gap between the kinds of concepts that you learn in computer science, which are mathematical things, and the everyday social problems that people are trying to solve with computers. It was summed up nicely a few years ago in a criticism of open-source software someone on the internet -- someone was complaining that the developers of GNUcash were dealing with memory leaks. I think they were using c or c++, and the complainer was saying they should migrate to java or python or something. If you're trying to make a computer program to solve problems, it's a waste of time to be solving computer problems. You should be solving the pre-existing human problem.

Every tool has its own problems. One problem is accounting, or keeping track of money. With pen and paper, you can run out of ink or paper. Humans aren't good at adding numbers in their heads. With computers, you can completely erase all of your work with a few clicks of the mouse or keyboard. So there are ways that problems inherent in the tool can emerge, which take energy and attention away from solving the pre-existing problem.

Currently with computers we are trying to abstract problems such as banking and business into simple logic puzzles. I think that's too much of a simplification. I think we need to create a virtual world full of basic human-percieved concepts, such as time, weather, humans, animals, etc. and create programs by manipulating those basic ideas and objects.

An example is an ontological system like OpenCyc. An ontological system holds hundreds of thousands of logical assertions like "Animals eat Food" and "Paris is the Capital of France". Basically, an ontology system has some basic common sense. From all of these assertions, it can make logical conclusions. So, if well tell Cyc that Duke is a Dog, it can conclude that Duke has a tail and eats food. If we tell it that Duke lives in Paris, it knows that Duke lives in France.

Now imagine, instead of dealing with animals and where they live, it has a bunch of assertions about generally accepting accounting principles. One day, you might be able to just sit down and talk with an ontological system via email or IM, and say, "We got a check from client A for $575, another check for $440." and then the computer balances the books with all the other accounting principles it 'knows'.

Current programs seems to be exclusively a digital re-creation of paper-based forms and filing cabinets. It's a sheet of paper with a bunch of field:value pairs, and reports are the resulting logical operations you can do with all of that data. This is *basically* the relational database. I think we are hitting the limit of the field:value model of reality. I think there are other models, such as virtual realities like online worlds, knowledge systems like opencyc, etc.

Programmers are working exclusively at too low of a level. Yes, of course, we will always need to teach and understand basic boolean logic and computer science terms, but we need to start working at higher-level, human friendly concepts.

Why it's so difficult

[BluedemonX]

BLOAT - Look at J2EE and COM.

EGO - Noone will admit they don't know what they want, noone will admit that what they wrote thet were learning while they were coding it, etc.

GENERAL CLUELESSNESS: Figure out what you want and build it. In that order. "If I don't write down clear specifications I can't be held accountable if they're wrong" is the manager's mantra.

MAGIC BULLET THEORY: Technology X will solve all our problems! And our neighbour's problems (see first point)

HR ISSUES: Must have 10 years experience in .NET (posted a year after .NET was released), Must have experience in Java, J2EE, J2ME, J2SE, JAX, JNI, JWD, JQV, WWJD, SOA, SOL, SQL, SPL, APL, OBE, CBE, ... (see point one) and AN EXPERT IN ALL THE ABOVE.

Corollary to the above: don't hire someone with a clue who architects well and can translate the design into whatever language you want (or tell you why "object oriented" and "EJB" don't co-exist) find someone who either guesses the exact answer to your clever pointer question you thought was a cool hack, or find someone with tons of certifications and ultrageekery in the language du jour (the GOLDEN HAMMER antipattern)

Re:Why it's so difficult

[Ungrounded Lightning]

EGO - ... noone will admit that what they wrote thet were learning while they were coding it, etc.

I'll freely admit that. I built a carreer on it.

A programmer is a particular sort of lazy person: One who will spend six hours doing something right ONCE rather than fifteen minutes doing it twice. As such he is usually working on something new - and bored stiff if he's writing-from-scratch a third version of something he's done before. Further: Having him do the same thing over is a horrendous waste of resources. He's already done it: Take that code, make minor tweaks if necessary to get it to work in the new context, and use that.

So he's always working on something that's at least partially new to him.

A good programmer (especially a consultant) comes to the job with maybe HALF the knowlege needed to complete it - plus the ability to quickly learn the rest.

Re:Why it's so difficult

[BluedemonX]

Actually, I've found it's more a problem trying to convince a manager that it would take LESS time to redo the damn code than it would be to figure out what the last guy did, and "resuse it" in the "new context" e.g. cut and paste and edit the hell out of it, rather than rearchitect it to the new requirements and take what you need.

A LOT of managers have this strange idea that lines of code are mined out of unobtainium or something, and that using as much as possible "from the existing codebase" is a REALLY GOOD IDEA.

Refactor. Rebuild. Check the unit tests. Duh.

But hey, so long as the checks clear, knock yourself out wasting BOTH our time....

Re:Why it's so difficult

[rockmuelle]

"BLOAT - Look at J2EE and COM"

Have you looked at COM? The core of COM (Component Object Model) is a wonderfully designed binary interface for component-based development. It's clean and simple, solves the component problem in a language neutral fashion, and was the foundation for hundreds of thousands of software projects. Microsoft solved a problem that still isn't as nicely solved anywhere else. For an example of bloated component system, see CORBA.

Of course, the marketing spin, productivity tools (ActiveX, Dev Studio wizards, et. al.), and many of the component hierarchies developed with COM are bloated messes. The marketing spin (calling everything COM) led to the confusion about what COM really was. The productivity tools were just plain bad. And, most people couldn't design good component systems (which is a point of the original article).

-Chris

Re:Why it's so difficult

[BluedemonX]

You are correct, that is what I meant to say. Sounds like I'm going "oh, uh, yeah, I meant to say that". I was referring to the environment of COM, as you very clearly enumerated, as opposed to the standard and the binary implementation.

Re:Why it's so difficult

[Ungrounded Lightning]

Actually, I've found it's more a problem trying to convince a manager that it would take LESS time to redo the damn code than it would be to figure out what the last guy did, and "resuse it" in the "new context" ...

Hear, hear!

I'm not advocating you reuse OTHER people's stuff.

Reusing your own older work, however, may be far easier than starting over. Also much faster and less mind-numbing. (Of course that's IF it turned out to be a good fit to the new prolem and IF he documented it well enough that HE understand what YOUNGER HIM wrote. B-) )

The point I'm making is that the only way a programmer knows nearly 100% of what he needs to know when the project starts is if the project is a repeat of one he already did. If that's the case, why is he restarting from scratch?

My brain doesn't work that way.

[Anonymous Coward]

I'm a sysadmin and not a coder. Scripts make sense. Do this, then this.

3D Matrix? Pointers? Sorry but those things confused the living hell out of me.

I took C++ in college. It was nice and fine. I understood it but after getting stuck on syntax, lost commas, single quotes and pathetic "Hello, world";. scripts it just lost ALL the magic.

My cousin does some high end 3D world modeling (and math) and he got turned onto it. That's why it is hard: our brains are wired differently.

Lack of a specification language!

[vinn01]

Home builders have architectural plans. Machinist have blueprints. Electronic equipment builders have schematics.

Software engineers are stuck trying to figure out the incoherent ramblings of marketing/sales/business analysts/corporate executives/users and a host of others who have no means to specify what they are asking for.

Software specifications are uniformly deplorable.

Re:Lack of a specification language!

[the eric conspiracy]

Part of the reason is that software development is so hard is that change and design are not managed correctly. In the old school engineering disciplines introducing change after construction begins is obviously a disaster in terms of budget and delivery. Deciding that you want a 6 lane tunnel under the Hudson River after you have a 4 lane tunnel half built obviously introduces MAJOR costs. Software, not being similarly physical appears much easier to change, and requirements are constantly shifting becuase of this perception. In response to this we see all sorts of development models (Agile etc.) that are purported to let the application evolve along with changing requirements.

This is utter crap.

Software change introduces costs just the way changing a half constructed bridge does. But management does not have the experience to realize this. A half assed program that sort of runs is accepted as the norm today because that's what everyone is used to. But a bridge that has an reliability factor of 99.99% (1 car in 10,000 going over the bridge ends up in the water) is NOT what people expect, and if they got it there would be holy hell to pay.

The other part of the reason is that engineering is a hueristic discipline. Yes there is all sorts of math and so on, but the real core is a collection of rules people learn the hard way. How long did it take to learn to build tall buildings after the invention of concrete? Centuries. Now you have a brand new tech and think all of a sudden you are going to be able to tackle large construction projects without misteps? LOL. It takes time to learn how to build anything this complex. Pull a van Winkle and come back in a century or two. Betcha then software development will be a lot more cut and dry.

Re:Lack of a specification language!

[iangoldby]

The source code is the blueprint/architectural plan/schematic.

The incoherent ramblings of marketing/sales/etc and the software specifications are the equivalent of the initial meeting with the architect saying we want 20 stories and a south-facing balcony on each external wall.

Three Essays by Jack W Reeves.
The Source Code Is The Design.

Re:Lack of a specification language!

[Coryoth]

Home builders have architectural plans. Machinist have blueprints. Electronic equipment builders have schematics. Software specifications are uniformly deplorable.

Software does have robust specification languages. Try Z, JML, SPARK, CASL etc. or things like CSP or CCS for specifying concurrent systems. Specification languages exist, and they provide sufficient power to nail down very specific specifications against which implementations can be verified. Sure, some of them are rather technical, but then you have to spend some time learning how to read complex electronic schematics too. For a variety of reasons such specfication languages are rarely used.

Re:Lack of a specification language!

[tshak]

Home builders have architectural plans. Machinist have blueprints. Electronic equipment builders have schematics.


Software developers have C#, Java, Ruby, and dozens of others. It's more likely that the problem is that we A) have too many languages and B) don't think of code as a specification. Remember, in software, the computer does the building, not blue collar workers.

Re:Lack of a specification language!

[istartedi]

Now, if only the specification language could be typed into a computer and compiled, then you'd really have something. Specs, except at the higher levels are, IMHO, over-rated for precisely this reason. As the specifications become increasingly detailed, they approach the complexity of the code. You end up with something that's every bit as complex as the code, except it doesn't compile. Worse yet, while it's a given that computer code is hard to read, people approach specs with the delusion that they'll be readable.

The trick is to figure out what's observable and testable by the user, and specify nothing more. Wherever possible, cite existing specs. If possible, break down your specs into logical parts and cite those (e.g., define what a "range of values" is for each datatype, so people have a prayer of knowing what you mean when you say "the user can enter a range of values".)

Even so, it's pretty much a lost cause. Some of the best software out there probably has no spec. Where's the formal spec for Apache? Linux kernel? GIMP? The libjpeg library (not the image standard, the lib itself)? Or, for that matter, any Open Source project? If you look at Open Source in general, you see a general lack of design docs and management in the traditional sense. You do see, however, other things that while not software, go along with software: documentation, bug tracking, versioning. It seems that Open Source has naturally weeded out the "metatasks" that aren't really integral to good software development, and formal specs (at least in some cases) appear to be unnecessary. Standards, however, are vital. A standard, however, unlike a formal software design spec, simply specifies what is to be implemented not how to implement it. Perhaps this is where most specs go horribly wrong.

Re:Lack of a specification language!

[Anonymous Coward]

Home builders have architectural plans. Machinist have blueprints. Electronic equipment builders have schematics.

Software engineers are stuck trying to figure out the incoherent ramblings of marketing/sales/business analysts/corporate executives/users and a host of others who have no means to specify what they are asking for.

Home builders, machinists, and equipment builders have another advantage: their products obey the laws of physics.

Gravity will not change to tear down a home that previously would have stood fine. The tensile strength of a given steel alloy will not alter to frustrate the machinist's intentions. The electrons will flow in the expected paths built by the electronic builder. Their goals are all physical. Predictable. Reliable.

The type of software that business folks moan about not being easy deals with business rules, business constraints, business procedures. All made up out of whole cloth by humans, and always containing exceptions and contradictions. Humans are more often than not, not logical. They are not predictable, reliable, or even believable. Hence, neither is business -- which is just a niche of human behavior.

Software is hard because humans are human.

Re:Lack of a specification language!

[chromatic]

I wish I had something more insightful to add than "I wish I'd said this" or "I'm a programmer as well as a writer. Natural language makes a terrible specification language." Excellent post.

Re:Lack of a specification language!

[Larus]

Software has pseudocode, and I rarely see people use this concept when they write their spaghetti. The pseudocode is also rarely passed down with the final product, thereby obliterating any possibility of future expansion and maintenance. Even when they are, they are handled by the wrong people who never read them. In the low probability cases where people have access to the pseudocode, the complexity makes any effort to understand the project moot. Smart programmers usually say, "What the heck!" and start coding everything from scratch, thus repeating the cycle.

No you don't, you silly manager troll.

[Anonymous Coward]

Linux is a large project. It was managed effectively with no source control and no managers for years. If that's what you consider good management and a good system, I might agree with you; but somehow I don't think that's what you were saying.

Too much Management and Systems *ARE* what makes software hard.

Re:No you don't, you silly manager troll.

[Richard Steiner]

Uh... What tasks do you think Linus has performed over the years w.r.t. the kernel? Or are you referring to the entire distribution universe (which is still effectively decentralized)?

Re:No you don't, you silly manager troll.

[Oz0ne]

No, I think the linux kernel project is managed well. I didn't mean to specify a type of management, just quality.

You wanna know

[MrCopilot]

What Makes Software Development So Hard?

two words: The DarkRoom

My Guess

[Anonymous Coward]

• Marketing
  
• Software "engineers", the most difficult group of people to get to work as a unified team known to man.
  
• Clueless managers
  
• Software patents
  
• Lawyers
  
• Users never know what they want
  
• lack of communication
  

Software has HYSTERICAL complexity.

[Ungrounded Lightning]

What makes software so hard? The enormous complexity of the software constructions.

Why is it so complex? Because it's so EASY to build it.

Example: Pre-computers, the moment-to-moment computations necessary to run an automobile engine were performed by mechanical devices, mainly the distributor and carburator. Every term in every computation was manufactured as a number of physical components, several of which are moving parts.

For instance: The RPM input to the spark advance was computed by two weights on pivots, with springs and stops, rotating a sleeve on a shaft. The shaft was driven by the camshft through a gear and the sleeve carried the cam driving the contact points or (in an electronic ignition) the starwheel that passed the sensor coil. Adding this computation (compared to no RMP spark advance) added five moving and four stationary parts, to be assembeled, and a test stand the volume of an office cube to test the result, and allow a worker to adjust the constants by bending the spring supports with a screwdiver.

In software this computation can be done by PART of ONE line of code. (In real engine controls it's actually done by more - mainly so the computation can be more complex and thus better approximate ideal running conditions.)

Software changed the game completely: When adding a piece of a compuation requires a moment of thought, a minute with a text editor, and issuing a compile command (plus whatever testing is necessary to convince yourself you got it right), rather than months of an engineer's and draftsman's time, manufacuture of dies, lab work to check the result, repeat through three layers of departments (to prove it can be done, to prove it can be done reliably, and to prove it can be manufactured affordablly), the amount of work and time to implement one bit of complexity reduced drastically.

The result was that the amount of complexity that can be afforded rose in proportion. Given that the proportion was hysterically large, the amount of complexity handled by each person became enormous.

Unfortunately, programming is NOT formulaic. Portions are - and as they are identified they are rendered into algorithms and software is written to perform them. The result is that the part people work on is ALWAYS the part that is "fuzzy" and difficult to formalize.

Programming consists of rendering a set of requirements into a correct specification for meeting the requirements. (The reset is automated.) This is not an easy task - and it gets more difficult with increasing complexity of function. Unfortunately, methodologies for performing it have remained in a catch-up game: The better the tools, the more complexity a worker can handle. The more he can handle, the more he is assigned.

To quote McClary's Third Law of Computer Technology: Software complexity expands to exceed the capability of any software development methodology.

Re:Software has HYSTERICAL complexity.

[Ungrounded Lightning]

By the way: Though the example was process automation, the same applies to all fields of software application. The drastically lowered cost of creating a complex computation was enabling - allowing things to be done that were too complex, and thus too costly, to do by non-software-driven methods.

When the blueprint IS the product it's possible to do far more than when it's a first step in an expensive process.

Re:Software has HYSTERICAL complexity.

[LordMyren]

I really like this, and think its pretty dead on.

We had a software engineering & ethics lecture in our software engineering class back at uni. The nutshell of the opener was, every other engineering department has far better success/failure ratios, and its coders fault for being sloppy and not following proper practice. My first reaction was that every other engineering has tolerances built in, deliberate overcapacity. There's no such thing in software product, yet software is a very long chain/directed graph of co-reliant structures where a single failure will in most cases cascade out to mass system failure. Any tolerances or overcapacity is implicit in the development process, not so much the code itself (how exactly do you give code tolerance or overcapacity?), yet we're constrained to very serious deadlines. We're relied upon to produce large complex rube goldberg devices that function perfectly, you'd damned well better expect failure and a lot of it. The only compensation we have is culture and practice, to manage the process and the complexity both.

It'd be so convenient to say that programming is just spec writing and product development. The truth is much further out there though. 99% of the time the requirements are a moving target. That is one of the most important features of code and programming, its ability to be responsive. It'd be great if we could coax our clients into deciding what they want and then building it, but no matter how many spec processes we run through, the end product is invariably 50% spec at best, and 50% ad hocracy. Specs are not the only way to scope and define and manage complexity.

My other main gripe is that OSS has less than done nothing to build itself better coding environments. Past the compiler and the object, its hard to see real forward progress in managing complexity of code. The code environment we work in and the practice of coding is largely shaped by vendors, and they've been feeding us the same worthless excrement for years now, and no ones built a simple enough powerful enough comprehensive environment to demonstrate how much more natural programming should be (try as Smalltalk/Squeak might). I think vendors have added huge mental overhead to managing what is often a reasonable consumable chunk of complexity, and in many cases its deliberate. Its poisoning the entire developer gene pool. And no one is stepping up to make better coding environments, we're so busy building more crappy product with the existing tool set.

Re:Software has HYSTERICAL complexity.

[Xugumad]

Agree, absolutely. Considering the number of possible states, for example, that a car can be in. Okay, start with the gears, then you've got lights on and off, windscreen wipers, air conditioning, CD player... maybe 100-200 states in total, right? Now, consider how many states a computer program can be in, it's many, many orders of magnitude higher.

Consider each instruction a moving part. I'll accept high level language instructions if it makes you feel better. That puts most of the programs I work on, each of them relatively small (except their web apps, so they're bloody massive in comparison to most web apps), at around 100,000 moving parts. If you went to an engineer and told them to design, build, test, and roll out to users a machine with 100,000 parts in 4 months (okay, okay, so it took us 6 to get these stable), they'd just laugh at you, yet it's not an unreasonable request for a programmer.

Re:Software has HYSTERICAL complexity.

[qzulla]

For instance: The RPM input to the spark advance was computed by two weights on pivots, with springs and stops, rotating a sleeve on a shaft. The shaft was driven by the camshft through a gear and the sleeve carried the cam driving the contact points or (in an electronic ignition) the starwheel that passed the sensor coil. Adding this computation (compared to no RMP spark advance) added five moving and four stationary parts, to be assembeled, and a test stand the volume of an office cube to test the result, and allow a worker to adjust the constants by bending the spring supports with a screwdiver.

What about the crankshaft? It is needed for the piston movement. I like how you added the camshaft. Now how about the cooling system to make it all run without locking up? How many part are in a carb? That is another essay in itself.

So it is not so easy when you start breaking it down. Each part becomes infinite, or finite in a big way.

Heck, trace the gas pedal to the carb with the cable and springs and cotter pins to hold it all in place.

Complexity builds quickly.

qz

Re:Software has HYSTERICAL complexity.

[pipingguy]

Excellent post.

Re:Software has HYSTERICAL complexity.

[__aailob1448]

Excellent post. Thank you.

Re:Software has HYSTERICAL complexity.

[Anonymous Coward]

Your sig is hilariously bad. You Fox News mouth-breathers are astonishingly ignorant about the world you're helping to destroy and the brown people you're helping to kill. But then, u probably have no idea what I'm going on about.

Bay Bridge

[whoever57]

The Bay Bridge is an interesing parallel -- the delays have been caused by unclear and shifting requirements (which are focussed on the aesthetics). These changing requirements have led to delays and cost overruns.

A manager was asked how many programmers s/he had:

[troll]

Oh, about 1/3 to 1/2.

Some people can, some people can't. Unfortunately they're all in the IT industry.

The automated development platforms that apparently make someone off the street into a programmer are at the base of a lot of bad code.

To turn it around....

[GeorgeMcBay]

What makes software development so hard?... What makes anyone think software development should be easy? It is simply an inherently difficult job that very few people are truly talented at, just as with any other inherently difficult job. All of these people who think software would be easy if people just followed process X,Y and Z would cause Fred Brooks to spin in his grave... if he were dead.

Fucking C

[SensitiveMale]

IMO, Delphi is still the best and fastest option available.

First and formost : Management

[Shivetya]

Management who is more concerned with the process than the product

Management who never met a feature they didn't have to have added

Management who sets rules for design lockdown but permits themselves exceptions

Management who never seems to discipline those who don't work but loads obligation upon obligation on those who do work

Management who is more concerned with initials on parking spaces, titles, and new furniture.

Explains our 3 year project currently in its 6th year with 3 more years scoped out !!!

Software development is hard

[goldcd]

as our brains provide very fuzzy solutions to problems and tend to just think about what we have to do, rather than handling all eventualities.
I talk to a customer and come away with an idea of what they want to do.
I write a High Level Design - and it makes sense and everybody is happy. I've basically just written down how we think.
The tricky part starts when you break it down to the low level. The questions start to come up:
What happens if that's that?
How do we stop X
If we have Y & Z htf do we calculate blah (without locking the db for a week)
Now when you think you've got the low level you then still forget stuff, implement a feature in a less than idea way blah blah.
Then the spec changes and you through band-aids all over the code.
Oh I could babble on for hours about this.

Another way of looking at it is to think of Chess. The objective is simple - to win with check-mate. The rules are simple - this piece can do this, this one can do this. The problem comes with the bit you have to fill in in the middle. The all but infinite permutations are there, so there is rarely ever a 'right' answer at any point. Some options are obviously better than others, but at most points you can never definitely say that was the absolutely best option. Pulling customers back into it, if you suddenly find a piece 20 moves back's just been fiddled with, you're screwed.

Just one last mention of customers, they often get a harsh deal from developers. Developers only see a small part of what customers actually 'do' - but tend to walk away assuming they know everything. Most bugs seem to arise when assumptions are made by the developer.

Development Tools are always Under Construction

[cyclocommuter]

You can't master the tools you use if they are constantly evolving. Case in point dotNet 1.0, 1.1, 2.0, 3.0 all within a span of 5 or so years. Yeah you the fundamentals (design patterns) stays the same but vendors such as MS try their best to obfuscate the basics by providing layers of "code generators" which are suppose to to make it easier for developers but in reality does the opposite. For example, how do you migrate generated code to the next version when the new tools come out?

It's just what I asked for....

[arthurpaliden]

But not what I want.

The problem is that no one takes the time to really analize what it is the customer "needs" and how they will "want to" use it.

Once this is done creation of the design document and coding is easy.

Software development is hard because.....

[mlwmohawk]

(1) MBAs who do not understand technology. They think of software s widgets and they expect software developers to "convince" them that one way is better than another, unfortunately, the gulf between what management knows and what it needs to know to make that decision is too vast. The MBA can't understand the reason and blames the developer for not being able to explain. This is frustrating because sometimes it takes years to understand something well enough to make decisions, but MBAs think they can be briefed.

(2) MBA negotiation. A software developer with some experience can make a good time estimate, but when the proposal is made, the management of the product says that's too long. So, they start creating hypothetical scenarios about better machines, more people, different specs, etc. until they reduce the time estimate. so, surprise! it takes longer.

(3) Bad developers. some developers just need to leave the business. they started in visual basic and have no interest in computer science or "the craft," the hack stuff together and say the things managers want to hear. Management likes these people and makes them project leads, then they get fired and move on to new unscrewed companies as "project leads.'

(4) Bad managers, some managers just don't know how to bring productivity out of people. when people are behind, they don't need nagging, they need help and they need an environment where they can talk about it.

(5) process over results, nuff said.

(6) Not enough process :-)

(7) Bad specs. A product spec is like a writing 'outline" it should be complete enough to know what to do, but be concise enough to give the developer the room to do it. specs that are too loose or too rigid make development difficult. During development, there is no possible way for a spec to encompass 100% of the possible problems, but there should not be too many surprises.

Programmers dont listen to users...

[PermanentMarker]

Well programmers dont listen to users, instead they listen to the one who pays their bill, he's often not the user who will use the software. Probaply he's an interim manager with different targets. So when the product is finished and the people should start using it there is often a to big mismatch, to much klicking everywhere and a teribble GUI. And most likely you come close to a product which allready exsisted but seamed to be more expensive in the beginning (later all changes and fixes make it a lot cheaper probaply..) So also take a damn good look on the market to check if something allready exists. As real breaktroughs of new software are rare.. Hack even distributed mp3 downloading is an old idea. So you realy think you have something new ??? Next.. People poorly organize, themselves have difficulty to explain their isseu to a programmer and the programmer well he just has to finish something. He simply hits real problems (sad he missed that math lessons). Advice go to the users dont think of anything stay open to them. Let them draw on a drawing board what they would like to see. Then after that part start programming, when programming use technology that the company is planning to use also after your product, probaply something they can support themselve. If they understand VB or C+ or C# or whatever stay with that. If they have SQL dont force to use a lotusnotus DB or MYSQL or whatever. Use current (but not outdated) technology. Ehm sorry to say MS quite future orientated and all their products mixup quit reasonable within their programing environment. So it might be handy to use that. Ofcourse you could bringin linux and your assembler knowledge combined with an oracle DB, would probaply use smaller code, but how easy would it be to extend or fix that code.. Ooooh and learn to program in multiple threads dont have that single process halting your performance because now in real life it has to deal with 50000 records instead of your theory test model who used 100 records. Understand the performance of the hardware talk with System engineers. I'm a guy like that and sometimes work together with a programmer because of a performance issue in their software.

mod this coward up!

[Anonymous Coward]

+5 funny/insightful

It does not have to

[trydk]

I find a general tendency today towards KICS (Keep It Complex, Stupid) programming instead of KISS. This means that people have modules with tens (or, shudder, hundreds) of thousands of lines of code, which are hard to maintain (an average programmer can probably handle a few thousands lines of code in her head, if that much) and even harder to test/debug. I personally try to make my programs very modular, where the modules are simple, single-function and testable. I do put a lot of thought into planning ahead and designing the module interfaces so I rarely have to change the interface specifications during the project. And I put comments into the programs, often way above the norm, in fact, some of my code could be read and understood completely just by reading the comments. Waste of time, some would say, but I like it. To be slightly more specific about the modularity:

• Simple means small and relatively linear, i.e. as little convoluted code as possible.

• Single-function means, well, that it performs only a single function. In the program I may have a general formatting module that can format dates, strings and numbers, depending a bit on the complexity of each type of formatting, I might choose to have further modules that each handle one type, i.e. a date-formatting module, a string-formatting module and a number-formatting module.

• Testable means that each and every module can be tested independently, which normally implies no side-effects (not an absolute demand, though). I always write a test section for each module and make the tests cover as many parts of the code as possible (hopefully all sections of the code, but for slightly complex modules this may be infeasible or even impossible).

By sticking to these rules and by documenting the code thoroughly, coding does not have to be too difficult. These rules/principles, I know, are part of some of the major design methodologies, but you do not have to make it more fancy than this to make it work. The above principles are very hard to retrofit into existing (legacy?) code that was not built this way, but extensions could still be made in a modular way that, if not to the letter, then in principle, adheres to the rules.

Re:It does not have to

[cmat]

Thank you for the setup for my point. At the rules you follow are NOT precise for anyone except yourself. You know your exact definition of "simple", "small", "single-function", "testable" and your experience is what allows you to adjust your definition to a given problem. This is why it is so difficult to build software, because you can't explain to me in static way how you do what you do. I believe this is why building good software is difficult to do and teach others to do. I think a previous poster's comment that the state of software development is a "craft", and should be taught much like plumbing, carpentry (think apprenticeship, tradesman, etc).

SW dev is hard because big CO's gain power/control

[SimBuddha]

I've been writing software and hacking hardware for nearly thirty years. It used to be much easier to write software. The machines were less capable and the software environments were less capable, and yet making software was easier.

Why?

Imagine you are one of the current software industry IP exploiter/plunderers: a senior manager at a large computer software/systems company. Your company didn't create computing nor invent most of the foundational technology you now control, leverage and profit from. You have a huge staff of marketers, managers and engineers to keep busy and an array of divisions and product lines. You have other smaller companies and open source initiatives nipping at your products trying to compete and share some fraction of the market. How do dominate the market? you protect your turf... You make the development environment and the software itself as complex as possible and influence the key hardware vendors to do the same. Why? so it takes a huge staff of engineers to do anything useful. That way that the cost of trying to compete exceeds the value of a competitive product, discouraging such competition. In order to take over certain product sectors, you under price products in that area while over pricing products where the competition is now extinct. When the new product sector is dominated after a few years, that becomes a new product area you can over charge for while under charging for some new sector of conquest.

The real problem with this "complexity as a weapon" strategy is that people who are making totally unrelated software, like most of us, have to suffer with insanely complex, poorly designed and not refactored, unreliable, no real time guarantees or QOS, barely interoperable, no security to speak of... systems and development environments that compare very poorly to 30 year old mainframe/mini technology.

Is this a conspiracy? or is this just a habit that happens to serve the very powerful and disempower the rest of us... YMMV

One day the motivation for making products and services will switch from "Winning" to "Serving" and on that day I will CHEER!!!

SimBuddha

Lesson:

[Anonymous Coward]

Don't do a massive project. You'll always end up with something that's a mess to maintain. Use the UNIX philosophy:

1. Small is beautiful.
2. Make each program do one thing well.
3. Build a prototype as soon as possible.
4. Choose portability over efficiency.
5. Store data in flat text files. (I disagree with this, a DBMS or SQLite is a better choice, IMHO)
6. Use software leverage to your advantage.
7. Use shell scripts to increase leverage and portability. (COM, DCOP, XML:RPC, SOAP, etc)
8. Avoid captive user interfaces.
9. Make every program a filter. (see 7)

Re:Lesson:

[Marxist Hacker 42]

Make each program do one thing well.

Wish somebody had told that to the first guy who coded ls. Three screens worth of switches is NOT doing "one thing well".

Re:Lesson:

[BSAtHome]

> Wish somebody had told that to the first guy who coded ls. Three screens worth of switches is NOT doing "one thing well".

You are wrong. ls does _one_ thing very well: it lists files; and it does a very good job at that IMO. The switches are not to increase the information stream, but are meant to decrease it (your milage may vary). The formatting is an important part (for compatibility) but the function is still to list files.

Although the options act visually as formatting, they are, in fact, filters. They filter the information in a very well defined way (well, most do ;-). The real use of ls is in scripts in combination with other `small' programs.

Re:Lesson:

[Marxist Hacker 42]

The point is, if you need 3 man pages to describe your switches, EVEN IF THEY ARE JUST FILTERS, you are not doing ONE THING. You're doing 3 man pages worth of things. And from a UI standpoint, that's just damned confusing to the user.

Re:Lesson:

[tepples]

Choose portability over efficiency.

Unless you have to deploy in environments where there is a demand for efficiency because your competitors are using efficiency as a selling point. For instance, Nintendo DS battery lasts 2 to 3 times as long as that of the PSP, and no ad for a cell phone is complete without an estimation of talk time.

Avoid captive user interfaces.

There has to be a captive user interface somewhere, as that's partly what a user is seen to expect.

Make every program a filter.

To filter keystrokes and mouse clicks into pixels? Does the filter approach work in GUI programs?

Re:Lesson:

[pclminion]

So instead we should have 25 versions of ls? I propose we call them: ls1, ls2, ls3, ls4... Much better!

Or are you saying that it is never useful to say, sort a list of files a certain way? Are you really saying that the user should be required to write a little shell script every time he wants more than just a list of filenames?

When you say "ls", you get an informative, simple report. This report looks practical identical on every flavor of UNIX that exists. If you want something different, you use a switch. I fail to see the problem in this.

Re:Lesson:

[Fulcrum of Evil]

Once the user can depend on sane defaults, he'll just ignore the options that don't do what he's trying to do. That' s the downside of something that's really flexible.

What Makes Software Development So Hard?

[Ari Rahikkala]

Slashdot. :(

Making software is not hard... building teams is.

[Dystopian Rebel]

There have been successful small software projects and successful large software projects. Good software ~can be made~ and ~is made~. Sometimes it's because of a tremendous individual effort. Sometimes it's a tremendous group effort.

Good software can be made. But building good teams and protecting them is very difficult in a world of self-consolidating middle-managers, overpromoted beancounters and confused Senile Management. This is where everything goes to hell in a handbasket.

Oh yes, then there are the consultants.

Most companies need to be protected from their internal incompetence, power struggles, and misplaced faith. It's really no surprise that on average, the modern dysorganization is a disaster at making good software.

There are a few exceptions. In general, good things come from small organizations or ones that have a clear authority and are guided by a few people who understand how to manage, how to inspire, how to plan, how to delegate, and how to code.

I'd like to continue this discussion but I have to prepare a Powerpoint presentation.

Re:SW dev is hard because big CO's gain power/cont

[cdrguru]

Very funny. Wow.

OK, really now. You don't actually believe any of that, do you?

Given that Open Source development would be completely outside of the conspiracy to make things harder, any Open Source development should be orders of magnitude simpler than writing something for Windows. The APIs would be simple, clear and well documented. Everything would be much simpler because it could be and nobody would be pushing back to make it more complex.

Unless of course Richard Stallman and Linus had been somehow bought and subverted. Yes, that would explain it - obviously Open Source development is just as polluted by greedy CIOs to make it more complex because ... well, otherwise it would be simple. Right?

Overstated it...

[Ungrounded Lightning]

A good programmer (especially a consultant) comes to the job with maybe HALF the knowlege needed to complete it - plus the ability to quickly learn the rest.

Actually it's more like 85%. But you get the idea. B-)

Re:One word: Skill, Talent and Knowledge

[Anonymous Coward]

I found a bug! Skill, Talent and Knowledge are four words.

Two factors

[SoulRider]

arrogance and lack of communication, from anyone in the development chain. Of course in any field of endeavor these two factors only make life more difficult, they just seem to manifest themselves in software development more often.

vendors & gnome are to blame.

[LordMyren]

because you pussy fucking bitches eat the dog crap the vendors sell you. no one has gotten off their ass and made the infrastructure, we rely on re-implementing the awful cruft vendors excrete where we should be making distributed information spaces ripe for modification and mashup. the only ones defining rules and pushing outwards are vendors, and linux is left focused on pussy 1993 shit like making a nice pretty desktop and clean installers. fuck that shit, thats catching up, its old hack low level crap thats irrelevant to the information war taking place (xgl aside) and most importantly it will never play a single step in creating better technology. its pure implementation, no tool advancing. right now, the vendors are defining the verbose vicious hideous standards that will dictate the rules and ethics of our online existance. i blame all of you and your worthless desktop projects for not turning the internet into a viable medium. we need the compelling rich mashup based extensively hyperlinked peer to peer environment before coding becomes an act worthy of being done. right now we're coding in sludgepools, isolated little worlds that pull a couple libraries together to dredge up a single interface for a single user. its trash and its shite and it sucks because the environment sucks, the environment is literally 18 inches in front of you and no where else. code needs to get the fuck out of the constrained shit hole of the application and become mobile. across the network, code has a purpose. some day it will all be self evident, that the code sucked because the environment sucked. get the fuck off the desktop, start building something pervasive, and stop fucking wasting everyones time. before the vendors fill our gene pool with any more feces. i blame you all. you started an asymetric competition and then competed directly against the lowest common denominator available. the desktop is a worthless god damned endeavour.

go read Data Trash. figure out whats being said. and tell me we're not just the bitch of the corporations. AND THEN, after a little edumacation and contet, you can consider flamebaiting me.

Development fails because developers are ignored

[ph1ll]

Well said.

But while stereotypes persist that programmers have no people skills, you forget that many business people don't either.

Just ask yourself: how many effective, people-oriented bosses have I ever had? If your answer is "not many", you're not alone.

I've been software engineering for over a decade. These are a few observations I'd like to share with managers:

1. Hire good people. If you don't, you're hosed before you start. If you can't tell the good people from the bad, you're in the wrong business.
2. Software engineering often fails because software engineers are ignored. In a badly run company, nobody even asks developers which is the best approach. Imagine if civil engineering were like this...
3. "Software is too expensive to do cheaply". Sure, you can get 20 engineers with 2 years experience apiece. But it's more effective to get 4 engineers with 10 years experience.
4. Let the team (largely) manage themselves. Open source projects work just fine without management - or, at least they do no worse than managed software...
5. Don't do things to keep morale high; instead, cease doing things that keep morale low. We engineers just want to do good work in a nice environment. It goes back to hiring good people and then leaving them well alone.
6. Give the team a hand in decision making. The team will be more incentivised to hit targets they set than the ones you set. Afterall, the only incentive they have for hitting your impossible deadlines is furthering your career.
7. Automated regression tests! There are still managers who don't know what these are/how important they are. Along with all other good processes, automated regression tests reduce stress.

reality

[anwyn]

Software is not created by software developers. Software evolves under selection presure. Software developers do the dirty work of implementing this presure, but they exist in a social milieu.

Managers prefer a different description. They like to say that software is created by software developers under a process that they control.

This is false.

But it makes managers, their pathetic plans, and their meaningless schedules seem more important.

This deception has a cost. It makes software development even more difficult that it already is. This is one of many advantages that open source development has over commercial software development.

Re:Gap between computer science and person problem

[Anonymous Coward]

"we need to start working at higher-level, human friendly concepts"

sweet, we could have computers automate all kinds of functions for us. Hey we could even build that kind of intelligence into security systems to help us fight the terrorists... I think we should call it Skynet.

On the other hand, I kind of like filing cabinets.

Nothing is unforeseen

[mangu]

Actually, they can build bridges and skyscrapers nearly on time and close to budget. You can't have both. There are always unforeseen circumstances that cause either, or both, to rise a bit

... Any engineering project that comes both under budget and ahead of time is highly suspect. Someone skimped along the way or lied about the timeframe or costs

Or maybe they just took into account the fact that accidents happen? Ask anyone in the insurance business how it works. Accidents happen at a predictable rate, you cannot predict one single accident, but you can predict how many accidents will happen in a large project during the years it takes to build a bridge.

Re:Nothing is unforeseen

[PitaBred]

How many accidents will happen when building a LEO sky cable system, using untested materials, techniques and machinery?

Re:Nothing is unforeseen

[Fulcrum of Evil]

I don't know, but it will be a lot of fun finding out.

Cats and penguins...

[Savage-Rabbit]

Why is herding cats so hard? Running an OSS project is a lot like herding cats. It makes one wonder why the Linux mascot is a Penguin? After all penguins are reportedly fairly easy to herd.... kind of like Windows users....

Re:Cats and penguins...

[thrillseeker]

It makes one wonder why the Linux mascot is a Penguin?

Indeed - it should have been this - it so appropos in so many ways.

Re:Cats and penguins...

[TemporalBeing]

It makes one wonder why the Linux mascot is a Penguin?

Check out Linus' book - Just for Fun. He tells why in there...though I'm sure it's probably in the really ancient parts of the LKM archives too.

Users

[filesiteguy]

Users make software development hard.

And Managers.

Users and Managers make software development hard.

And Vendors.

Users and Managers and Vendors make software development hard.

Other than that it is easy. I am actually able to read this article two weeks into a major implementation....

http://www.perfectreign.com/?q=node/42 ...ahhh, sleep - who needs it!

you are correct

[Anonymous Coward]

"I think one of the biggest problems in our industry is accountability."...that'll change overnight the first time some judge rules that software that is sold must have a normal consumer warranty attached to it. Every other consumer product out there has at least a legal implied warranty-software?

It's either treated like art, or engineering, if it is engineering and sold-it needs a warranty. If that stops new features for a few years all over the planet and forces serious code review and best practices changes and marketing descions changes-, than so be it,better now than later. The day that software got patents, it should have had warranties forced on it from the government.

I'm still waiting for some big business who gets hosed from the exploit dujour or some bug where it costs them serious folding money turning to their lawyers and going ENOUGH, sue the hell out of them and base it on lack of warranty, see what happens.

You are only going to avoid it for some time period, you aren't going to avoid it forever. It's only gonna take one rich well known and powerful company which isn't an IT company but relies on it to do this. And because of that, given the amount of total failures out there with software, and the amount of companies who aren't software companies..it's going to happen and that judge is going to rule in favor of the plaintiffs without any doubt.

    Ya'all screwed the pooch insisting on patents. As long as it was just in the art class, copyrights, there's no case, but with patents? Ha! And you can threaten it's impossible and a web browser will cost one million dollars per copy and all sorts of dire things-and it won't matter. there are 6 billion folks and growing on the planet, they all want good paying and non-physically exhausting jobs if they can get them, and if some folks won't or can't do a job that still has a lot of demand-someone else *will do that job* and cash the check.

It's time the "get out of any responsibility, neener, neener" free EULA card got smashed to pieces. And it will happen.

Two big elephants in the room

[Jerf]

I could babble on about this at length... in fact I'm planning on it on my website later this year... but there are two points worth making.

Fred Brooks in his masterpiece The Mythical Man Month draws a distinction between an essential thing and an accidental thing. Something is essential if it can not be removed; accidental is the antonym.

Most people, when answering the question of why software development sucks, answer with a long list of accidental points. Most of them are true, and most of them are worth thinking about. But it's worth considering the essential reasons why software development is hard, too. I think the two biggest essential problems are:

• Our expectations rise more quickly than our capabilities. In many important ways software development is getting easier, but we always want more. Towards the end of last year, over a couple of months with no more than two full-time man-weeks devoted to the task, I squeezed out a weblog server platform. It has all the features you'd expect from a weblog platform in 2001, a healthy dollop of later features, and the beginnings of some unique features that are why I wanted a new platform in the first place. I did enough programming in 1997 to know what it would have taken to match the capabilities of this new weblog platform. Rather than two man-weeks, I'm probably looking at twenty man-years, what with the caching and SQL and other fun stuff I get nearly for free. It's easier for me because I got to build on one of the current trendy web frameworks, which itself builds on a lot of software that wasn't around ten years ago.
  
  But no matter how our software advances, our expectations advance faster, so it seems like we aren't moving forward or are even moving backwards. We aren't. We aren't moving forward as quickly as I'd like and I think life for programmers is about to get very, very "interesting" in the Chinese-curse sense (multi-core is going to rock our world, and I think a lot of people may never really make the transition...), but we are making progress.
• Nobody wants to pay for good software. Given the choice of "Fast, cheap, correct", a choice that isn't going anywhere in the near future, people consistently choose fast and cheap. By "people", I mean everybody: Developers, customers, the accountants in charge, users, you name the group, that's their decision. I'm not prepared to say that's even the wrong choice, because of the magic of compound interest "fast and cheap" can be the most rational choice. But nevertheless, this makes it effectively impossible to have "correct" software, which is what people are really bemoaning when they complain about the state of software engineering.
  
  Many people say that all we have to do is apply the practices of other engineering disciplines to computer programming. This Slashdot discussion is no exception. But what these people fail to address is who is going to pay for these techniques, which uniformly involving doing "more" of something, usually a lot more, with more expensive people with more training and "more" of a lot of other things (training, personnel, time, testing, etc.).

This isn't by any means the whole story; we've still got a lot of accidental things in the way. But these essential problems guarantee that many of the complainers aren't going to get any happier anytime soon, because even if we completely fix all the accidental problems, these essential problems will still be enough to trigger further complaints.

Today's applications are like 1850s houses.

[Kadin2048]

Of all the Renaissance painters that ever painted, only a few became forever famous.

I think this might be taking the art side a little far.

With a painting, the surface appearance is the end product. If it looks good, it is good.

This is distinctly not the case with software. Not only does the application (or whatever) have to look good on casual inspection, but have to be built to work well, including under conditions that might not have been thought of in the beginning.

I think architecture as a metaphor for software development is actually pretty good; architects have to combine artistic judgment and technical skill in order to produce a building that's both aesthetically appealing -- true works of art, in some cases -- but also structurally sound, designed according to accepted standards and principles. It's not enough, in most cases, to just have either one. Nobody wants an ugly building (well, most people don't), and nobody wants a building that's just a facade without substance.

Software "architecture" is where house-building was a few hundred years ago. When you wanted a house built, you just went and got a few people that you thought would be good at building houses, or had good reputations for building them, and told them what you wanted. They might or might not have any formal training, and the only 'standards' were what they knew from past experience would probably work.

I suspect that in a century or two, software will be not unlike house-building is today. While every house today is different, depending on the owner's requirements, there are a set of well-understood and basically accepted standards for putting them together. That doesn't mean that some builders aren't better than others, or that some don't cut corners, or that there's not room for artistry and physical beauty in the final product, but just that in theory, they all meet minimum standards for structural integrity and other key factors. In time, I think software design will probably follow. Every piece of custom software will be different, because they each solve different problems, but getting to that solution will involve a combination of creativity and the application of existing standards, built up from the accepted body of knowledge of what works and what doesn't.

Re:Today's applications are like 1850s houses.

[Grishnakh]

I suspect that in a century or two, software will be not unlike house-building is today.

So you're saying that in 100-200 years, almost all software will be developed by large companies who charge high prices, but hire illegal immigrants with little or no training to do the building? Worse, the software won't be written correctly, and won't be checked because 1) the company doesn't care and 2) the government software inspector is paid off by the company to look the other way instead of doing a proper inspection to make sure the software is "to code"? And while the software may look cool at first, it'll stop working in a few years after numerous parts have broken? And if all that wasn't bad enough, the software developers will come back to work after-hours and steal parts of the application and resell them on the black market?

No thanks.

Re:Today's applications are like 1850s houses.

[Glonoinha]

Shit man you don't have to wait a century for that scenario - that's what we have TODAY.

Re:Today's applications are like 1850s houses.

[FirstTimeCaller]

So you're saying that in 100-200 years, almost all software will be developed by large companies... software developers will come back to work after-hours and steal parts of the application and resell them on the black market?

Yeah. That sounds about right.

Re:Today's applications are like 1850s houses.

[Grishnakh]

Oh, I forgot something:

While the government software inspectors turn a blind eye to shoddy workmanship and even gross errors made by the "professionals", when nonprofessionals try to write their own software, these same inspectors refuse to pass it, saying it's "not up to code" because it doesn't meet various insane regulations. You also have to pay a huge fee to get access to the codes, even though you're bound by law to meet these codes if you choose to write your own software.

Re:Today's applications are like 1850s houses.

[idiat]

But 1850's houses are *better* than modern houses, you do know that don't you? The sad thing is I actually suspect your analogy still holds however, there is little evidence software is getting better quality over time, simply broader in scope.

Joel On Software is a good site if you want to understand some of the issues in software engineering, in particular this article springs to mind.

http://www.joelonsoftware.com/articles/Development Abstraction.html

Ob Swordfish

[xant]

Eh, I'm not even gonna say it. You saw the movie.

Re:Ob Swordfish

[alienmole]

Nah, that movie was unwatchable. But I watched the relevant clip!

Software Engineering != Engineering

[Pedrito]

Everyone keeps on with the bridge building metaphors as if software development/engineering is actually engineering. It isn't. We're not engineers. Software hasn't been around long enough to be engineering, it's barely a science and largely an art. That's the problem right there in a nutshell.

There are really a number of related issues, but when it comes down to it, it's not engineering. How much code will it take to do XYZ? I don't know. I can guess and give a rough estimate. How much cement and how much steel will it take to build bridge ABC? A good bridge engineer could tell you with a great deal of accuracy without building the bridge first.

Software Engineering is such a bad name for it. Knuth had it right in the early days: The Art of Computer Programming.

if it wasn't hard, everyone would do it...

[hguorbray]

Oh wait everybody does do it now.....

and that's part of the problem.

-if we step back to the '90s the growth of the internet and increased usability of PC-based tools combined with the .com boom resulted in all sorts of people becoming involved with software development who did not come through the traditional Academia, DOD and Fortune 500 DP departments avenues to IT and therefore did not learn formal software project best practices -particularly the ideas of QA, testing, signing off on approved, agreed to and frozen specs, and Acceptance testing/signoff.

Obviously some of the methodologies have changed and streamlined since the rise of the internet and I'm not dissing everyone who came into the industry through non-traditional routes, as I did, but there are a lot of people in the biz who are unaware of or unwilling to follow organized software project practices (or their corporate culture does not support these practices ie. the customer is always right, even when they're wrong, clueless non-technical managers, etc).

I also realize that the formal requirements of a Javascript applet on a webpage is going to be considerably less than those of an Fortune 50's payroll application. But it seems that there are too many people who are used to the Javascript-level of project formality who continue to work at that same ad hoc level even when working on something much more complicated and risky like the aforementioned patroll app which requires formal specs, schedule, acceptance criteria etc.

The resultant jumble today is that two identical projects at 2 similar companies will go completely different directions and potentially have completely different outcomes based on things like the personal prefernces or knowledge of the architect, or the lack of an architect and the whim of the Software Lead or Project Manager. Then the QA and Test departments and everyone else down the line will also have their unique take on what is needful and what they consider to be a priority and what is correct functionality.

This underscores the thing that I am wrestling with at my current company: Best Practices

Every company should have and follow them and the different softwware subcategories should also have them, but as with most good ideas, they are mostly honored in the breach as it takes extra effort and consensus to implement them.

On the other hand there is so much diversity in software development these days that there is definitely no 'one size fits all' Best Practices solution either.

-I'm just sayin'

What makes Software development so hard?

[wonkavader]

No specs. That's what does it. Period. No design in the first place.

Its possible to iterate forward by making chunks that do parts, but in that case you need the customer to understand that. They usually don't.

Failing that you need specs. ANd the customers never have them.

You just admitted you don't know what you're doing

[syousef]

I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development.

People in charge of decision making who don't know how to build software, don't understand what impact their decision has on the overall project, and don't allow counter-arguments to be made properly because they don't understand the problem in the first place is probably THE number one reason software projects are difficult and come out with less than ideal solutions. The poster should never have been in charge. He should have had competent technical staff who ALSO understand the business under him who he trusted and had authority to make decisions.

Other things include:

1) Poor industry standard libraries/APIs and tools that require the developer to think about work arounds and fulfilling requirements to successfully use the API/tool instead of focusing on the business problem. Just look at the mess that a J2EE project is with several different languages: Java, JSP, Jabascript, HTML, CSS and libraries: Hibernate, Struts, EJBs. By the way from what I've heard .NET is not much better. Then there's the tools which typically require a whole bunch of error prone glue code that does nothing for the business problem but is required to use the APIs. Then there's sub-standard debugging tools that crash or slow down the system.

2) Technology moves so quickly that developers and architects barely have time to learn one technology before another is thrust upon them and touted as an improvement that will solve all their previous problems (conveniently sweeping under the rug any new problems created because hey the peddler is out to sell their product to your team to use).

3) Specs that change significantly mid way through the project. Usually these changes don't just add something new but actually change the fundamental way the project worked. Oh and we need it yesterday. How many bridges do you know that need to be made 30% bigger mid way through the project, and not just by extending on? The business needs to plan what they want out of the software much better and try better to anticipate changes that may be required.

4) The developers are often good at coding and "talking to machines" not at talking to people. This seems to be steadily improving but its still true that a lot of developers don't have social skills. Often this leads to the test team and/or business becoming hostile or seeing them as "propeller heads". I've seen this get REALLY bad on one project to the point that the test and business staff were openly abusing the developers who didn't know how to take some stand. Developers can get frustrated or burnt out if treated badly regardless of their abilities or lack their of.

5) Changes to trivial look and feel elements when there are more critical flaws in the product. Yes I understand polish is important but you don't polish a half built car for fuck sake.

Family Guy perspective

[Anonymous Coward]

> Why is conducting an orchestra so hard?

Peter Griffin: .... "Then I tried conducting an orchestra." (cut to Peter in front of an orchestra) "Should I-... Should I conduct with my penis?"

Oddly enough, this is insightful if you think about egoless programming and egoless management.

Conducting an orchestra is a lot easier than conducting a software project precisely because in the orchestra, when is playing, people subordinate their egos to "the greater good of the music". In software projects, ego enters all over the place. Programmers want to work on pet projects. Managers want the appearance of progress more than they want actual progress. Users want to short cut the change control process to force their features in above what others want.

ls

[camperdave]

Wish somebody had told that to the first guy who coded ls.

I agree. I don't know how many times I've needed to get a list of just the subdirectories in a directory (like dir /a:d in MS-DOS) and have gone through the ls man page. ls just doesn't do it.

Re:ls

[Marxist Hacker 42]

That's one I know by heart. Unix: ls -r -ad. VERY useful if you know part of the file name to apply as a filter.

Re:ls

[camperdave]

Sorry, ls -r -ad does not list the subdirectories in a directory.

Try:
mkdir test
cd test
mkdir a a1 b c
touch a2 b1 c1

Now, give me an ls command that will list the subdirectories. ie:
ls <some options>
a a1 b c

Your command ls -r -at gives
.
not a a1 b c

Re:ls

[Anonymous Coward]

ls -d */
ls -l | grep "^d"
find . -type d -maxdepth 1 -mindepth 1

Re:ls

[Anonymous Coward]

tree -diL 1

Re:ls

[camperdave]

ls -d */
This is the closest ls actually gets, however it puts trailing / characters on everything.

ls -l | grep "^d"
grep is doing the work here, not ls

find . -type d -maxdepth 1 -mindepth 1
This isn't even an ls command.

ls cannot list directories. It has a million and one things it can do, but it cannot perform one simple, but highly useful function.

Looks like he doesn't know Knuth

[plopez]

FTA:
The idea is that if we're going to turn the creation of software into a true science, we need to first have principles. We need to know the fundamentals and formulas by which software behaves. What are the laws and principles we can count on in creating it?

I think Knuth has been working on that.

But in general, the reason software is hard is that the problem domain is hard. You can't solve anything without understanding the problem. This is why hiring code monkeys causes so many problems. THey may make nice GUIs and know how to build a web based wizard 'out of the box', but end up knowing nothing about the problem at hand.

Other observations:
1) He gets it write when he talks about things always changing. Part of the reason projects like Vista get delayed is that they take so long is that the requirements change. If you can't do it in 6-9 months, don't do it.

2) Nice to see him reference Brooks.

3) Another nice qoute:
The only real reason that a team gets together and writes a new piece of software is that they want to do something new, something another product doesn't do, whether it's to add a feature, or to be compatible in a different way with another system.

For years now I have said: "Software development is R&D". If the software has been built then use it. If you must do scratch development, then understand that it will be open ended; and all attempts to schedule, budget and predict will fail. It is *not* an industrial process, though we tend to treat it as such.

4) My observation is that much of 'Software Development' is actually social. You are working with users, managers, investors and programmers and trying to get everyone to agree on something. It gets political *fast*. So a good SD team must not only bee good at learning new problems domains and coding up solutions, but also must be good at human relationsships. And the closer the development team is to the end user, the easier this is.

The Kolmogorov Complexity of requirements changes

[presidenteloco]

(and design changes and implementation changes) is not well enough understood as
a factor that makes code development expensive and in some cases infeasible.

Relative Kolmogorov Complexity is, roughly speaking, a quantitative measure of the
informational (descriptional) difference between a bitstring (e.g. program) A
and another bitstring (e.g. program) B.

What we really need to understand is the complexity of the difference in the expected
input to output patterns of a program, after someone specifies or makes a certain change
to the program.

Often, a requirements change that sounds to a layperson as very simple actually results
in a very large informaitonal complexity of the difference in the program and intermediate
data structures needed to achieve the change, compared to the previous program and data
structures.

Usually, programmers have some inkling of this complexity of change, and
managers/executives/customers do not.

For a software program to be well-architected and well designed means, among other
things, that it is explicitly designed (via appropriate patterns and abstractions)
to make many hypothetical future changes to the behaviour of the program be achievable
with low Relative Kolmogorov Complexity of the software change.
The program's form must also be constantly minded by a software architect so that it
retains this low-complexity adaptability property.

The opposite of this is known technically as a "hairball" codebase, and unfortunately
is what the overwhelming majority of programs turn into in very short order after
a small number of changes from the original conception (initial requirements understanding)
are implemented.

Formally, a hairball codebase has almost no changes in its external behaviour (appearance
or logic of the output or process) that can be made without an infeasibly high relative
Kolmogorov Complexity. The "hairball" program structure is encrusted with special-case "warts" that
lead to geometrically multiplying complexity of the program's logic, and consequent multiplying
complexity of changes to the program.

Oh I suppose it is important to mention that the relative Kolmogorov Complexity of a software
change will be highly correlated with the cost and likely bug introduction rate of that change.

Until this (admittedly very) theoretical notion is understood by the authority figures
(PHBs, managers, executives, and customers)
who make the executive decisions controlling the conduct of software projects, disaster will
almost always loom.

The PHBs and customers need to ACTUALLY LISTEN to the software architects who say we should
spend some time and money up front crafting a maintainable, adaptable program, and we should
spend some time and money throughout the lifetime of the program to keep it that way. The
architects, grokking as they do at least the essence of the "RKC of change" problem, are actually wisely
trying to SAVE money (and maximize the chances of project success, in the medium and long term)
but are usually incorrectly perceived as being geeks with "no sense of business" if they
are foolish enough to bring these issues up in meetings with PHBs.

As Ray Ozzie says, "complexity kills", and the "relative Kolmogorov complexity of software change"
theory (aka "hairball entanglement theory") should tell us exactly when, why, and how much.

What a Surprise

[arjay-tea]

"I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development."

Gee, what a surprise. A lot of people in charge of software development projects don't know much about software development. Do you think ... naw, couldn't be.

Speech Synthesis Software

[camperdave]

art is about "saying" something through your craft which I don't think is really the goal of software development.

...Unless, of course, you are developing speech synthesis software. In that case saying something is the entire point of what you're crafting.

Lemons problem

[Beryllium Sphere(tm)]

The usual (imperfect) example of this class of problem is used cars. The seller knows the condition of the car. The buyer knows it's either OK or a lemon. The buyer doesn't know which, doesn't want to get burned, and therefore won't pay the price of an OK car. The seller doesn't want to sell OK cars at lemon prices. The market fills with lemons. Dealers in used cars get a bad reputation as a class.

Cars are a bad example because the buyer can pay $65 to a mechanic and get a pre-purchase inspection. Software buyers don't have that option. Imagine paying for an audit of the Firefox code base. If the software is closed source, then the pre-purchase inspection goes from prohibitively expensive to outright impossible.

So buyers can't force the suppliers to provide good software, because the buyer's don't know what it is. Especially since the people approving the purchases know even less about the software than the users do. Software suppliers can stay in business shipping junk like $YOURFAVORITEEXAMPLE. Suppliers who take time to do a good job go out of business competing with hack shops. It's not coincidence that some of the best software is non-commercial.

An attitude problem

[cunnilingus]

I've worked with various companies at the past and one of the main reasons why software projects fail is lack of attitude of people involved in the process. Even brightest analysts, developers, managers etc give up fighting over time, resources or trying to change the flow and in one point they say "yeah, whatever".
Then we have simple human problem - when they need to work as a group, they think that they've already failed (that manager is a dick, project will fail again. Yeah, whatever) and then we do not even try to improve situation, meaning - we have lost battle without even fight.
Of course, it is very difficult to manage humans, but at least execs could try hiring more HR pros, instead of counting money (which have not been earned yet).
Everyone talks about customer needs, satisfaction and motivation of talents that work in the company, but basically no one gives a shit for real, so maybe they are doom to fail and it is not such a bad thing anyway.

The real point is...

[Anonymous Coward]

...that software development needs to have a team of programmers who are talented and competent, and a project leader who knows his people and can effectively lead and manage both the team and the project... and a corporate culture which doesn't treat its people distrustfully like a bunch of indentured servants, then they can write good software and get it done in reasonable amount of time. Getting and retaining a team of very good talented and experienced programmers is very difficult and expensive however since most development firms are cheapskates and not only do not pay enough, but treat their people like dirt, binding them under all kinds of bullshit NDA and non-compete contracts, poor work environment with noisy cubicles or "bullpen" office areas, not empowering them with latest technology workstations and tools, etc. The software development company that teats its programmers golden is the rare exception... the exact opposite "sweatshop" like I described above is generally the norm, as I've seen it in my 20+ years in the biz, hence you usually get kids right out of school as entry-level positions doing the bulk of the development. The best cream of the crop programmers have mostly all quit programming years ago and now are all sitting in management positions and programming no longer.

A good orchestra conductor who is in front of a bunch of rank beginner inexperienced musicians will not be able to make very good sounding music. You get what you pay for.

Re:The real point is...

[Metasquares]

The week before a concert is absolutely brutal, especially if you're rehearsing with a band/orchestra (but even if you're playing solo piano and rehearsing on your own). You're treated fairly well at other times (most importantly, after the performance), but you can sure feel like dirt during all of the rehearsals.

I guess it's like being treated with respect as a programmer, except that you still have deathmarches.

Your analogy is good, though - hire amateur musicians, and you're not going to get a good outcome. If the conductor knows next to nothing about music, you're not going to get a good outcome. If the instruments the orchestra is using are not tuned well, you're not going to get a good outcome. If you rehearse a piano part for weeks and the conductor suddenly asks you to play the oboe instead five minutes before the concert, you're not going to get a good outcome...

Re:The real point is...

[RedDevilCG]

"The software development company that teats its programmers golden is the rare exception..."

No kidding! I have never heard of a software development company that let's the golden teats loose on the programmers! I would code in Turbo Pascal with that job perk!

Re:The real point is...

[Anonymous Coward]

I would code in Turbo Pascal with that job perk!

Let's not get carried away, now... ;-)

Re:The real point is...

[Anonymous Coward]

Your analogy is good, though - hire amateur musicians, and you're not going to get a good outcome.

I respectfully disagree, but I still think the analogy is good. Hire amateur musicians in the true sense of the word (i.e., people who love playing music), and you can get quite a good outcome. I have played with many professional musicians who will spend the smallest amount of time on a certain project that they can get away with. In fact, I found out that many professionals pride themselves on their ability to quickly scan the music for what absolutely needs to be right (or the audience will find out they're putzing around) and what can be "safely" dropped because only the "experts" will notice anyway.

The same goes for many professional software developers, especially the ones you contract for a specific job. They know that it is only the UI which the boss will ever look at, so they'll spend a lot of time on eye candy and fake functionality. The boss will not ask them whether they're using smart pointers instead of valgrinding until there are no more visible leaks, whether they are using standards-compliant protocols instead of whipping up something themselves which will only work with their own (also whipped-up) server, etc.

If I were allowed only a single question when selecting my team members, I'd pick professional developers and ask them what they're working on in their spare (i.e., unpaid) time. I have plenty of colleagues who'd answer "I spend enough time programming at work, there's no way I'd crawl behind a computer in my spare time." Those are not the types of people who'll get your project done.

Re:The real point is...

[Maltheus]

Cream of the crop developers do not go into management. They set up their own consulting firms. I've yet to meet a decent developer who had any aspirations for management. It may pay a little more, but only in exchange for embracing all that you hated as a developer.

Fred Brooks answered this 20 years ago

[jam42]

Joel on Software loves to remind us of this article whenever some company claims to have vaporware that will allow "anybody" to program computers:

"No Silver Bullet: Essence and Accidents of Software Engineering"

by Frederick P. Brooks, Jr. (author of "The Mythical Man Month")

http://www-inst.eecs.berkeley.edu/~maratb/readings /NoSilverBullet.html

"Computer", Vol. 20, No. 4 (April 1987) pp. 10-19.

I blame the requirements

[Xernon]

Software development is hard for many reasons. I agree with earlier comments that a fundamental issue is that our ability to dream up software requirements easily exceeds our ability to build it. I recently wrote an article discussing how trying to meet the various functional and non-functional requirements makes producing good software so hard. Check it out: Producing Good Software is Hard.

Re:SW dev is hard because big CO's gain power/cont

[SimBuddha]

The point I am making is that instead of making things more reliable and refactoring out useless complexity and "features", market dominators benefit from complexity which they use as a competitive weapon. They also leverage the profits of one product line/service to take over others. I have worked for 2 different large SW companies for many years that each were market leaders and yet were driven out of a product sector by a larger domination oriented software/system vendor who used this exact technique. I have lived it and witnessed the process. If it were funny, I would be laughing.

If the dominant commercial OS of today could provide half of the guarantees of antique OS's of the past, I would be happier. The reality is we have a poorly designed foundation on top of which software has to sit. The whole thing leads to a system and software that corrupts itself and crashes and is effectively designed to wear out. Imagine software that wears out, a perfect way to ensure future revenue and sales of new software.

If I hadn't written a complex vertical application that is still running all aspects of a complex medium size business without incident for over 20 years, I'd say that software cannot be reliable or just fully serve its purpose like an appliance for years without maintenance or improvement. But I know it can...

I have worked with Linus years ago and respect him and his goal, which I understand as just providing an alternative. Linux is, well basically unix, like BSD... It isn't something new nor is there any agenda to create a complete, yet fully functional "new" environment that has dramatically reduced complexity while offering new levels of power and capability (like Flash for example). Linux is good enough for a lot of people to do useful things with.

What bugs me most is that young people who might consider programming are not provided a welcoming environment that most can succeed in using for school projects. Use of the computer is limited to word processing and graphics. The 100$ Laptop is a multimedia book for 3rd world kids that I wonder about. How about kids in the USA? Lets see, they should use MS Dev Studio? or Logo... Laugh. How about smalltalk or a similar environment.

Software is being made to serve vendors need to sell something they know how to make, not to serve peoples need to understand, model and solve problems most effectively. Vendors make what is easy, not what is worthwhile. Exceptions to this are rare and usually expensive. (like Reason(tm) music synthesis for example)

Thanks for listening to my perspective. YMMV

SimBuddha

WTF?

[slashflood]

Pudge
Pudge's projects

Maybe it's you, who's arrogant.

Re:WTF?

[HomelessInLaJolla]

I doubt it. I haven't made any claims about hard work, sacrifice, or superhuman intelligence in some random homeless guy's journal just to show off how much better I am. If Pudge didn't want the attention he shouldn't have been trolling in my journal.

Why are you pointing me to his projects page or a small biography of him? You're simply reinforcing my suspicion that his success, at least in part, is due to other people doing (at least some of) the heavy lifting for him.

Every great man stands on the shoulders of giants... and all that.

What makes it difficult

[Anonymous Coward]

What makes software development so difficult is:

* Capturing accurate requirements from users. Users are rarely able to articulate what they want in IT terms, so you need a translator... and things are occasionally lost in the translation, or the user says "well, that's not too important..."

* Development takes time. Lots of time. Beta tests, regression tests, etc... Even before it's in production (and I won't even go near the getting it in production part right now). As an iterative process - programmers are used to writing, tweeking, writing, tweeking, writing, tweeking until it's just right...

The problem is that a USER has to be involved in some of the tweeking and testing, and they get burned out on the whole process rather quickly... they tune out...

* While the developement is happening - the business continues to run and evolve and *shit happens* which means requirements change to deal with the shit... This has a ripple effect on other development processes... and that requires more testing, etc...

* Because requirements change on the fly - things take longer than expected. When they take longer, they cost more. When they cost more - there's a meeting. And we all know how worthless 99.9% of meetings are...

The way you manage it is by taking things in chunks... Plan out the chunks on a GANTT chart - give everyone a timeline. Then if there's a change - you can plug 'er into the GANTT chart and look at the effects on the timeline and let the (l)users decide if they want to accept that or not...

Set milestones - make a big deal about getting to the milestones... Keeps everyone in the loop on what's happening with the project...

Lock the requirements in stone - (see change above) unless it's a "gotta have". If it's a nice to have, it waits until the next release. Gotta have = gotta have lots of signatures, all the way on up to the people PAYING for the development... if they gotta have it, they gotta pay for it. If they don't wanna pay for what they gotta have, they can't have it. It's that simple.

Oh yeah - caffeine... anything the programmers want with caffeine - no problem. Get them food. Get them caffeine. Get them training... Treat them right - they do right by you too... they're nuts and fight with nerfs and frisbees in the aisles, but they'll get it worked out...

Here's a template for you

[ColdWetDog]

Input requirements:

-- whatever the user wants to key in

Output requirements:

-- the answer they needed

OK, the rest is for you guys to figure out. See you tomorrow.

Too many fad diets...

[Kazoo the Clown]

From Yourdon to Xtreme, every few years comes along another fad diet purportedly designed to "fix" the "software development problem." In fact, these things are really designed to sell books.

Managers, under pressure to make things better and who don't have any idea how, naturally gravitate to these fad diets and reorganize their departments to attempt to conform and gain the advantage these diets promise. This will often have the effect of making the problem worse, pissing off developers, adding additional constraints that impede development by attempting to pigeonhole their round-peg developers into the square-holes of the "perfect" software organizational structure.

The problem is, just like food diets, there does not exist a one-size-fits-all solution to the problem. However, that does not deter the interest and saleability of a one-size-fits-all solution, even one that does not actually work as advertised-- and an author can always explain away the failures by an "incorrect" application of the solution for one reason or another.

There's at least one factor though, that must be in place in a succesful project, at least according to the manager's definition of "successful" which is usually more related to "on time" than "feature complete"-- the developers must produce, manage and own their own schedules. A developer working to a schedule externally imposed will feel too little responsibility to meet it, while a developer working to their own schedule will be personally invested in it enough to feel the need to work harder when necessary to retain their own credibility.

A far better motivator is, "you said it would take you three weeks," than, "if you don't get it done in three weeks your fired," or even, "if you get it done in three weeks you'll get a bonus." While the second approach may seem to work in the short run, it will often have the side effect of increasing turnover via the reduction of morale when the imposed schedules are unrealistic (as they often are). And the third approach may do the same, depending on the effects the pressures have on the developer and to what extent money outweighs them. But the first approach involves a developer's own integrity in the motivation, who only has themself to blame if the schedule was overly optimistic.

Be cautious however, of attempting to convice a developer that he can do it faster than their initial estimate by suggesting that some optimistic scenarios are likely. Quite the contrary, a good manager should insure that the more junior developer has considered some possible unforseen events, in order to assist the less-experienced developer in developing realistic schedules for himself. And a cautious manager will actually add additional padding to a developer's schedule to make up for the unforseen that even the experienced developer may not have considered, further enhancing the likelyhood that the ultimate target release will be met. I know some managers for example, that will routinely take a developer's schedule and multiply it by 2-- and I can say it makes it a great place to work with little turnover-- developers feel that there is usually enough time available to do a good job, are invested in the process and proud of the (often) quality results.

Some thoughts

[cdrguru]

Writing software today is in some ways comparable to sword making in about 800AD. Lots of people want to do it, some actually know how and fewer still know the right way. Doing wrong can get people (and projects) killed, but still there are a lot of folks doing it that just barely know how.

Things didn't really get better until the whole process was "industralized", probably closer to 1400AD.

Software development beyond just mathematical processes is only perhaps 50 years old. Software development as any real sort of "engineering" is maybe 20 years old, if that. Call back in 100 years and maybe, just maybe, there will be as much progress as there was between 800AD and 1400AD in sword making.

Captain obvious

[plopez]

Herding cats is hard because you are using the wrong management technique. You herd cattle (and sheep and goats and pigs etc.), you do *not* herd cats. Cats, you put them in the general area of the mice and let them do what they are good at. Micromanagement of cats is a losing proposition.

Re:Captain obvious

[Anonymous Coward]

Which is why that dead Greek said, "give me a remote controlled mouse, and I will move the earth."

Re:Captain obvious

[zettabyte]

I just emailed myself your post. Very nice.

I'd give you a thumbs up but this is an old school forum.

Awesome list

[Anonymous Coward]

Especially #5 and #6, so true. I'd add one more correlary to #1:

(8) Developers that don't understand business. Just like the MBAs can be arrogant bastards with unrealistic expectations, you also have developers that might be very talented coders, but at the same time egotistical fools that completely disregard the needs of the customer. Joel Spolsky does a very good job at documenting that type.

not everyone is a rocket scientist

[TobiasS]

- mediocre developers that think they are architects/computer scientists
- design by commitee
- vague accountability structures

Re:SW dev is hard because big CO's gain power/cont

[micrometer2003]

I think there is some truth to this along with renaming, repackaging, reselling, and retraining (of past capability) to keep cash flowing into the future. Example: Lotus123 (and the shareware knockoff AsEasyAs) were faster, more accurate and capable of doing more sophisticated computations (e.g. multiple regression analysis) than Excel. The GUI revolution had much to do with it. Same with email. At 1200 baud in character mode, they flew by faster than I could read them. Now, with a popular provider's latest beta version with Flash, it can take as much as a minute per just to delete them unread! Coupled with simple but very useful programming languages back then, it was too empowering. Users would become totally independent of MIS and big vendors. And now there are so many non-productive distractions that rank and file employees need to be heavily monitored at their workstations. Another need to be filled! How this scenario ever got sold to upper management is quite a mystery to me.

How to learn software design

[nih]

Obviously learn the basics of programming, then read Code Complete, you will quickly realise that this book is a revelation in coding terms, not just coding design, i'm not sure if it's worth reading Code Complete until you have a reasonable level of coding knowledge, then follow the suggested reading on pages 860-863, and a few others at the ends of the chapters, eg: Object oriented design heuristics by Arthur J. Riel, which i'm reading atm and looks good so far. Above all just keep reading, gl:)

Crisis? What Software Crisis?

[DavidHumus]

Maybe fifteen years or so ago, the "software crisis" was still a much-talked-about topic. Today, with the situation as bad or worse, it isn't such a hot topic - we're used to it.
Or, according the Wikipedia, it was at least partially addressed by the implementation of various processes and methodologies http://en.wikipedia.org/wiki/Software_crisis.
If you've ever had to deal with the magnitude of overhead introduced by many of these "solutions", you know what bull-hockey that is.

There was a study in Communications of the ACM several years ago (sorry, can't find the reference) about user interfaces. In the course of the study, the experimenters had people use interfaces that were deliberately poorly designed. They didn't get a single comment on how difficult it was to use this software - people have gotten used to crappy, clumsy software.

I dont understand the question......

[MrBandersnatch]

Software development is now incredibly *EASY*. I mean we have tools such as C#, VB, .Net, Java, Perl, PHP, Python, COM, CORBA, SQL, J2EE, IIS, APACHE, XML, XSLT, HTML, XHTML, SOAP, XML-RPC, JBOSS, ZOPE, CSS, AJAX, Javascript, XQuery, XPath, UML, Patterns, SCRUMM, WMF, CardSpace, Passport, Windows, Linux, OS-X, WME, Direct-X, OpenGL, SDL, Eclipse, SVG....I mean, all this stuff software development still cant be "hard" now can it?

Nothing new in the entire discussion here....

[BarnabyWilde]

...it has already been thrashed out with the same old comments.

Waste of time to re-hash the same-old same-old.

BWilde

software isn't hard

[timmarhy]

The BUSINESS of software is hard. if i had design specific's that never changed mid project, managers who let me do my work and didn't make other worldly promises to clients, if coders were actually apprechiated and this nonsense idea that coding is an unskilled job went away.... then software developement would be a great job and we'd have great software. unfortunately business does all it can to ruin software for itself. they demand speed over quality then wonder why they get shit?

The Peter Principle

[Randym]

This has a name. It is called the Peter Principle.

Very good analogy...

[Belial6]

That is a very good analogy, and I would add that it is very common for city official to expect and intentionally ad 'hacky' solutions to city planning.

Just ask any female programmer

[0xdeadbeef]

Barbie says it's all the math involved.

No Silver Bullet

[KillerCow]

What Makes Software Development So Hard?

Asked and answered in 1987, and still spot-on. No Silver Bullet: Essence and Accidents of Software Engineering

See also: The Iceberg Secret, Revealed.

Too much creativity left to the peons

[CPE1704TKS]

I've been thinking about this for a while, and my answer will probably rustle a few feathers will all the developers in the crowd. I know I don't like the answer since I am a developer as well, but I believe it is correct.

The simple fact is that too much of the software development is left to the peons, ie. the developers. The skillset of the developers are totally random, as is the style, their expertise, etc. It introduces too much variability to the software development process.

Look at civil engineering projects. They are able to create buildings, bridges, roads, etc, and for the most part are much better understood than software development projects. But the peons, ie. the construction workers, do not dictate anything. There is one way of bolting steel together, one way of mixing concrete, etc. The only person who has any say is the chief architect.

However, in software, a lot of the design decisions are left to the developer.

In order to introduce predictability to the software development cycle, unfortunately, all variability must be removed. Developers must not have the freedom to code whichever way they want to, but have a simple, standardized way of accomplishing similar tasks throughout the codebase. This completely removes creativity and enhances predictability and leave all creatvity to the chief architect for the project.

Obviously, I don't like this answer because it's the creativity portion of programing that I enjoy the most, but I think it is for the most part correct. With the people I work with, their skill levels vary from good to horrible. Software development requires a certain level of skill sets that most people don't have and too many people who shouldn't be coders are coders. These are the people that introduce bugs, etc.

Re:Too much creativity left to the peons

[toganet]

I tend to agree with you on this, although a bit reluctantly. I also think a large reason why this part of the problem persists is the general laziness of senior developers. Speaking from personal experience interacting with senior developers and so-called 'software architects', coders suck at communication and leadership skills. Whether it's ego or shyness, they don't seem to be able to do much beyond just coding.

Where I work the end result of this is that the 8 or 10 senior developers all have discrete areas of expertise, rarely interact even with each other outside of a rowdy lunch or two, and cannot effectively back each other up when one is out of town. In addition, junior developers struggle to get up to speed on our crazy chewing gum-and-duct-tape "platform", and our attrition rate is about 50%.

I'd love to work someplace where the senior developers actually design the application, provide detailed instructions to their subordinates, and consistent coding practices are followed -- but it sounds like those places are few and far between.

Re:Too much creativity left to the peons

[mollymoo]

The simple fact is that too much of the software development is left to the peons, ie. the developers. The skillset of the developers are totally random, as is the style, their expertise, etc. It introduces too much variability to the software development process.

Look at civil engineering projects. They are able to create buildings, bridges, roads, etc, and for the most part are much better understood than software development projects. But the peons, ie. the construction workers, do not dictate anything. There is one way of bolting steel together, one way of mixing concrete, etc. The only person who has any say is the chief architect.

The countruction workers in the software industry have no say whatsoever. The construction workers in software are things like preprocessors, compilers and linkers. The blueprint is the code, the building is the executable. Code is really just a more formal, detailed spec. A chief architect may design the look of the whole building and the main structural components, but they aren't the ones who specify how thick a girder to use to support the walkway on the 7th floor, the size of bolts to use and how much torque to tighten them with; that's some more junior architect or engineer - the equivalent of non-senior programmer.

Read

[alanmeyer]

The Mythical Man Month tells us a lot about large scale projects. Written decades ago, it still holds true today.

Some key points:
  + Large scale projects that require a large resource pool will lose a significant % of man hours due to communication overhead
  + Adding resources to projects will cause the project to slow down before it will speed up

Re:Development fails because developers are ignore

[Anonymous Coward]

Sure, a lot of business people have no people skills.

But since we're all about talking truth here, let's not forget to mention that a lot of software engineers don't have programming skills either.

I'd say loudmouthed egos outnumber talent in both camps.

But you know, that's all good! Because if everyone were as smart and talented as we wish they were, then we would be the dumb ones that everyone is complaining about! ;)

Re:Gap between computer science and person problem

[Anonymous Coward]

"So, if well tell Cyc that Duke is a Dog, it can conclude that Duke has a tail and eats food. If we tell it that Duke lives in Paris, it knows that Duke lives in France."

$duke = new Dog('Terrier'); //duke is a dog
$duke->setHome($earth->country['France']->city['Pa ris']); //who lives in Paris
$duke->eat('Alpo'); //yum.

Yay for OOP.

Besides, how would Cyc know you meant Paris, France and not Paris, Tennessee? Or Paris, Missouri? Or Paris, Illinois? The ambiguity inherent in natural language is probably one of the biggest reasons we're not programming in Plain English.

Root Causes

[kaoshin]

The Worse is Better design philosophy, idiocy, selfishness and greed are factors leading to the difficulty of software design.

Everything large scale is hard

[mldqj]

Large software projects, books, PhD thesis ... Even if you know how to do every single part, it is hard to manage consistency because of the sheer size.

Re:Gap between computer science and person problem

[lawpoop]

What you're missing is that an ontology system like cyc has hundreds of thousands of such assertions, and has reasoning capabilities of basic abstract concepts such as 'things', 'temporal things', 'spatial things', 'intangible', etc.

Check out this diagram. Cyc has 1.4 million assertions. How long will it take you to finish your program? How long to troubleshoot? ;)

Cyc has a language for making assertions. It has no ambiguity. In my example of the IM interface, cyc could determine ambiguity and ask clarifying questions.

Why is software so hard?

[Anonymous Coward]

Lack of Rigor. But then, it's not appropriate to compare a bridge to a piece of code in 90-99% of the applications. Bridges HAVE to work every time - hard to reboot a bridge. Software, on the other hand, can be rebooted at need.

Notice that fuel injection units, pacemakers, fly-by-wire systems, ... tend to work way better than, say, word processors? I think people put the necessary effort into software creation, and for essentially everything we use, that's not much.

here's a thought

[briancnorton]

I see this all the time. Programmers are a lot like the computers the program. The good part about them is that hey do exactly hat they are told. The bad part is that they do exactly what they are told. Paradox? I think not. Programmers assume that requirements are perfect, and that their vision meets that requirement.

I think that the time has come for "programmers" to become workers. You want a new system for managing telephone calls? Let the programmer have a day of training and some OTJ experience answering calls using the old systems. There really is no substitute for first hand knowledge. It's probably also time for the death of the general "programmer" in favor the the more specialized niche programmer, everybody specializes in one type of software or another.

"Mathematical Limits to Software Estimation "

[betelgeuse68]

One of the better reads on this topic:

http://www.idiom.com/~zilla/Work/Softestim/softest im.html

Here's my view, based on my own life experiences as a one time software developer. Software development is partially an art form. No matter how well studied people are, writing awesome software requires passion and having some qualities that are like being an artisan. Just an artisan with great vision can turn a piece of stone into "David" it takes analagously speaking, similar qualities in creating a system of code that are resilient to change but at the same time extensible yet with maintainable.

The reality is, for many people, software development becomes a means to pay bills, nothing more. Which is fine, many people do this. However unlike the task of perhaps excavating a 10 foot by 100 trench for construction purposes, software is littered with intangibles including bad specs, unmotivated software developers, new paradigms (AJAX), etc., etc.

At least if you're working a construction crew, things are much clearer and you as a foreman can insist on people "stepping up" if needed. These measurements are simply not possible with software. Despite all the panaceas that has been given labels over the years, hard numbers just are not possible. And if you at least concoct a metric system that seems to work, you might have staff that quits and at that point, all those metrics are gone. All developers are *not* created equal. Fact.

There was a day when I used to follow the ANSI committee on C++ and knew the language incredibly well. I wrote some of my best code ten years ago including C++ frameworks that others had occasions to rely on. Sadly, I did *not* observe this attention to detail with my peers save for a couple of them. I had occasion to observe a couple of developers once wanting to rewrite my code since they had never used the Standard Template Library (STL) and had never encountered keywords such as "mutable" or my use of the "new" C++ (at the time) cast notation. Never mind that the code worked and did what it was supposed to and a peer review of this code (two seasoned and driven C++ developers) did not recommend changes.

The greatest "sin" I observed during my years of software development were individuals who programmed by rote, i.e. cut and past programming and simply did not have the motivation, inclination or passion(?) to learn the very programming language that constituted their "bread and butter." Expecting a largely unread person to suddenly write like Shakespeare (or columinist at a major newspaper) doesn't just magically happen. The most articulate people of the English language are always well read. They've at some level "paid their dues" engaging your mental faculties that many people never bother with. And just like with many things in life, practice, practice, practice.

Things perhaps have gotten better in the software development realm if only because there are less platforms to worry about now. By and large everyone is creating web applications and if you are not, then you are probably serving some vertical market (I doubt you are creating a WinAmp competitor) or work at large companies - Microsoft, Apple, IBM, Cisco, etc. Writing a Cocoa application (Mac OS X) means coding in Objective C but how many of you can make the claim you code in it regularly? Point made.

As my career advanced and I worked with web developers I saw many people not at all inclined to pick up new material. The one thing that web developers were incredibly bad at is induction - taking two principles or abstractions and creating a third. Most of them seemed to learn one way of doing things and applying this by rote over and over - cut and paste programming at its worst.

And since web apps are par for the course at many institutions now... well you can imagine the results.

-M

Because silly...

[ed1park]

Software Development is about solving problems given a limited amount of resources and time. And as with any "project", software or not, things get harder to manage as they get larger.

Go read Mythical Man Month by Frederick P Brooks
http://en.wikipedia.org/wiki/The_Mythical_Man-Mont h

Training & under estimating the difficulty

[aGuyNamedJoe]

I think part of the problem is training. Too often programmers are taught "How to use a computer to do X" (and often " ... in language Y") instead of "How to do X". The focus is on the tool(s). If we taught carpentry similarly, and the Saw was the primary tool, the training would be "How do cut siding with a Saw", "Building Frameworks with a saw", "driving nails with a saw", "Roofing with saws!", etc.

Computer science seems to focus on the computer. Engineering more often focuses on problem solving and/or design, with a emphasis on the type of engineering the department specializes in. I think it would be interesting to see if there's a difference in projects where the primary degree is Computer Engineering vs Computer Science vs Electrical Engineering. Also, many of CompSci departments seem to have cut way back on math and other subjects that focus on abstract thinking. I don't want to condemn all CompSci departments of course, but having taught and helped in curriculum development in Comp Sci, I'm not completely blowing smoke here -- one has to put up a fight to keep broader courses in the curriculum

However, even if that's completely off base, and I'll allow it could be, there's also the fact that people tend to severly underestimate the difficulty of designing a system. We'll happily decide to do something in software we'd never consider doing in hardware. Why? Does the system design get easier when the construction material is completely insubstantial and there's essentially no intermediate level at which parts of the design can be simulated or tested? If a system is to be implemented in electronics, there's a lot of science that can be applied at various levels (differential equations, boolean algebra, component simulators, etc). If it's going to be in software, we don't need that, the programmer can do it all in her head!

We know hardware design is difficult. For some reason, we think software design is easy.

(I'm sorry, perhaps I'd best take my medication...)

Moo

[Chacham]

Because programmers refuse to design, and designers refuse to program.

If only we'd just work together...

Management makes it so hard

[Orion Blastar]

always making promises to consumers that cannot possibly be kept. Always making promises to developers that cannot be kept. Running a negative work environment and heaping stress on the developers in hopes that they will work harder and faster and also calling them names and yelling at them does not help but only makes things a lot worse instead.

Compare this to the F/OSS development that doesn't have Classical Managers screwing things up, etc. Is it any wonder that most F/OSS projects are better than the closed ones?

so many inadequate similies ...

[constantnormal]

... here's one more:

In every complex activity, there are limits to the envelope formed by the boundaries of complexity, time available, and changes per unit time. Like juggling 3, then 6, then 21, then 317 bowling pins while ice dancing to reggae.

What makes software so difficult is that the profit motive drives people to operate on the other side of that envelope. That, and managers are too fscking stoopid to understand that those boundaries exist, preferring those time-honored management techniques of allowing customers to massage the design long after construction is well under way, and throwing bodies at the resulting late project.

It's not so difficult to understand.

its hard because its being done wrong.

[3seas]

Like doing advanced math with roman numerals (no translation).

But then the hindu-arabic decimal system came along with its (how can nothing have value) zero place holder and the common man was able to do more than replace the roman numeral elite accountants, they were able to do more math.

The same thing applies to software development, a need for an easier way.

hence, Abstraction Physics.

http://threeseas.net/abstraction_physics.html

any one (of the elite) wanna help?

Wrong Answer

[Anonymous Coward]

Engineering is the application of materials sciences.

The problem with your theory is that it's wrong in its premise, because:

"Engineering is the design, analysis, and/or construction of works for practical purposes.".

I'm an engineer, and Wikipedia's definition is perfectly correct. For instance, many important areas of engineering deal in energy, which is not a material and is not studied in materials sciences.

The real problem is

[Anonymous Coward]

"Why is conducting an orchestra so hard?"

The real problem is the tools and lack of maturity in building "atoms" or building blocks of software code. This is the real fundamental problem that and many programmers are divorced from. Then there is the serious hardcore mathematics and actually DEFINING the hell the problem... is and it's solution, no one takes the necessary time to coalesce the raw data, refine it, and make conclusions from it. Everyone's in a bloody rush.

Next is time constraints, time constraints are a REAL problem in my opinion. You can't plan for what you don't know about or what you can't even begin to define and measure, especially when it has cascading effects through an enormous system know one can truly understand fully. This is why MEASUREMENT is so critical in the software industry, there should be whole sections of industry working together and DEDICATED to analyzing the scope of different problems and how large systems can get so they can better estimate whether their chasing fantasies or not.

The next is serial vs parallel programming, many programs still run in a serial manner and we have yet to really wrap our minds around how a computer works (running algorithms) and building algorithms that can actually make easy to use parts of programs, to build bigger 'machines' (programs).

All programming is is an exercise in engineering mathematical 'components' called software. The fact is much software development is 'on the fly' engineering. There should be a serious industry dedicated to just making the basic 'parts' of programs, and testing these parts by actually making programs with them.

Many programming languages do not cut it in the slightest... the fact that there are so many different languages tells you a lot about the fundamentals of computer science: That even the fundamental tools are still not there yet to build robust software.

Feature Drift and Meta-tizing

[Tablizer]

In my opinion feature drift is the biggest problem. CRUD (edit/search/report) screens would be almost a commodity science if it was not for all the added tweaks that people want. Saying "no" is not good politics, but necessary if you don't want to make a mess. I don't want to say "no" myself; It is a career slapper.

Another problem is poor designers who don't know when to seek help. I've seen cases where every combination of factors was turned into a different trend-monitoring screen. They tried to "meta-tize" it once so that the combinations were table-driven or dictionary-driven, but the programmer they tried couldn't handle meta programming (dispite being otherwise good at regular programming). Thus, they abandoned meta techniques and hand-wrote every combination to meet a looming deadline. You need meta experts for meta work. Instead they paid 10 programmers 8 months of work to produce each combo. (I was one of those, but my lobbying for a meta technique was rejected because meta-ing failed once before I got there.)

Re:One word: Skill, Talent and Knowledge

[istartedi]

Nah, bring in the right management team, and the problem is solved. The only reason the number 45,872nd seller on Amazon isn't number 1 is because the author wasn't being managed by an experienced team of managers familiar with Unified Authorship Model and Extreme Storytelling. Oh, and the right marketing team could have made it fly off the shelf. And what were they thinking writing this thing in English? Everybody knows it's too easy to make mistakes in English. Esperanto would have fixed that. I think the guy was lazy too. If they had barged into his garret and asked him for status reports every half hour, the book wouldn't have missed the deadline by six months. What a shame.

Increasing LOC -- increasing failure points

[Money for Nothin']

The bigger the program, the more interplay of potentially-failing factors: more variables, more method calls, more potentially insufficiently-tested classes/libs/components pulled-in, more likelihood of multithreading (raising the probability of the problems that come w/ that, e.g. race conditions, deadlocks, etc.), more likelihood of networking functionality (which of course depends critically on the reliability of other systems), etc. etc..

Calculating the risk of failure in software is one giant conditional probability problem, in which each very small piece (say, storing a 4-byte value as an integer in memory) may have been coded and tested ruthlessly for decades. But the software those operations form are, in every case, brand-new within the user's domain, [1] else, it would likely be purchased instead; the more code is being executed, the more opportunities there are for failure...

So say you write an app and you know it has 2 major failure points, A and B. P(successful_execution | !A and !B) = 1 - (P(A) + P(B)). But now add a third failure point, C. P(successful_execution | !A and !B and !C) = 1 - (P(A) + P(B) + P(C)). The probability of a successful execution can do no better than remain at the same level with each additional failure point -- that is, with each additional line of code (or class, or module, or however you choose to measure and define potential failure points).

In short, as complexity rises, so do the number of potential failure points, and thus the probability of failure. And because most projects are from-scratch -- even if they've been written before for another organization, the developers won't have perfect memory of that solution -- they are all-new creations, each time, assuring that those failure points have not been ironed-out before (again, minus whatever those developers manage to remember at their new job)...

[1] You may have written an awesome logging function for a megacorp once, but jumping over to a startup, they won't have it, so you'll have to write it again (unless the previous company open-sourced it, but that's highly unlikely).

2x3.8

[jbengt]

The 2x4 is 1-1/2"x3-1/2"+/-.

Re:Gap between computer science and person problem

[Tablizer]

Current programs seems to be exclusively a digital re-creation of paper-based forms and filing cabinets. It's a sheet of paper with a bunch of field:value pairs, and reports are the resulting logical operations you can do with all of that data. This is *basically* the relational database. I think we are hitting the limit of the field:value model of reality. I think there are other models, such as virtual realities like online worlds, knowledge systems like opencyc, etc.

I disagree. Relational databases can represent just about anything any other logic system can, including Cyc. I do agree that RDBMS vendors sometimes put in limitations or difficultes to gain performance, but that is an implementation issue, not a problem with relational theory itself. I've seen Expert Systems written in dBASE that did what $900 AI ES's could do. (dBASE was only partially relational, I should note.)

The biggest problem with tried AI is that the users don't understand it. Yes, it *can* be powerful, but it is too abstract and "odd" for most users such that programmers are needed anyhow to translate biz rules into machinable stuff, and all the usual problems of programmer-versus-user interaction/translation comes right back just as before.

There still is no Golden Hammer or Silver Bullet.

Re:Gap between computer science and person problem

[Tablizer]

Yay for OOP.

You don't need OOP for that. Plus, we want the *users* to put the rules in, not Java programmers. And, OO is lousy with many-to-many relationships. It turns into a big gob of object references (pointers) like a modern-day Go-To filled program. Similar messes in the mid 60's are what motivated Dr. Codd to invent relational.

Insightful vs. Funny

[smccto]

Looking at the ratio of "insightful" vs. "funny" postings here... it's obviously because programmers just don't have a very good sense of humor!

Because it is.

[etnu]

Why doesn't anyone ask why Brain Surgery is hard? Why doesn't anyone ask why writing good music is hard? Why doesn't anyone ask why starting a successful business is hard? Why doesn't anyone ask why designing an effective sales campaign is hard? Software development is hard because it's complicated, just like all of the above. It takes dedication, time, and, most of all, a lot of intelligence. Software development frustrates the business community because they want it to be manual labor. In manual labor, employees have no value because they can be replaced by anyone at any time. With skilled labor (such as software development), it isn't that easy. The standard for companies is to offer lower pay and attract worse engineers, which yields worse and worse code. In recent years, we've seen companies looking for lower-cost countries, hoping that would save them. It turns out that there was a huge shortage of good engineers in those countries, too! Throwing more people at the problem doesn't help. Companies need to hire better people, and to get better people, they need to pay them more. Your product managers should not be making 2-3 times as much as your engineers. If that's the case, you're going to have shitty engineers, and shitty products, too. You wouldn't trust your child's life with an unproven, unskilled, low-cost surgeon, why would you trust your business' life to an unproven, unskilled, low-cost engineer? Stupid, stupid, stupid.

Because you code in error-prone ways

[einhverfr]

Instead of:

if (x == 0)

try the following:

if (0 == x)

This way if you do:

if (0 = x)

you will get an error.

Re:Because you code in error-prone ways

[zrq]

This way if you do:
if (0 = x)
you will get an error.

I use this technique all the time. Very useful way to avoid bugs, particularly with C++ pointers or Java references.

Tricky to remember at first, but after a while it become automatic to type

if (null != x) { ... }

Miss out the ! and you get a compiler warning rather than a hard to trace runtime error.

It's design not engineering.

[Anonymous Coward]

So there really is such a thing as a software engineer.

a foolish consistency ...

[strangedays]

software necessitates invention

do foolish process and management

while abort retry fail >>>

Re:Development fails because developers are ignore

[Anonymous Coward]

In a badly run company, nobody even asks developers which is the best approach. Imagine if civil engineering were like this...

Well, I do live in Indianapolis...

I blame the languages...

[grumbel]

I think one of the core problems that makes developing software hard these days are the limitations of the programming languages we used, more specifically their limitation to express *what* one wants to do instead of *how* to do it. Simple example: If I want to read a file in C I have to fopen() the file, fread() my bytes out of it and finally fclose() the handle, maybe even dynamically allocate the buffer with malloc() and stuff. By writing that I however don't express what I want, but instead I express exactly how it should be done.

As soon as I want to do the same thing differently, say read a compressed file via zlib, it all becomes a mess, since now I am stuck with a munch of different read/open/close function that look pretty much the same as the ones before, but are all incompatible. Next step now is to abstract it via something like struct IOStuff{ ReadFuncPtr read; OpenFuncPtr open; ...} to have an interface that works with both gzread and fread, this however only works so long till I have to interact with some for foreign code that has its own IOStuff structure, it does the same as mine, but again, a little bit different and 100% incompatible, the whole wrappering starts all over again. A few days down the road and I am stuck with wrappers stacked on wrappers, that do pretty *nothing* beside working around limitations of the language. And that happens a lot in programming.

The solution to this would be a language that gets much closer to what I want to do and much less into how I want it to be done. In Haskel for example I can just write 'readFile "foobar.txt"', there is no opening, no closing and most important the file is *not* read in completly as it would be in your average scripting language (Python, Ruby, etc.), instead the reading of the file is delayed till you actually start to access the returned string.
Now I am not saying that Haskel is the solution, but functional languages seem to get a lot closer to allowing to specify the 'what' instead of the 'how' then imperative languages.

Another important aspect is that such a language to be practical needed to allow one to specify the 'how' as well, thus allowing manual optimizations in the places where they are needed, many how the higher level languages so far fail to do that.

Todays programming is just to much based on duct-taping to work smooth and easily and I am not sure that will ever change until we kind of restart from scratch.

Re:Development fails because developers are ignore

[Tablizer]

But while stereotypes persist that programmers have no people skills, you forget that many business people don't either.

My observation is that the "suits" have better people skills than the techies. However, they often stop using them, especially around techies. Thus, techies don't know how to be nice; suits choose not to be nice. The flip side is that techies tend to be more honest with you. Suits enjoy tricking people.

Re:One word: Skill, Talent and Knowledge

[Tablizer]

Anyone can write a book. But, writing a book that's *GOOD* and that people will want to read is an entirely different matter. Computer software is no different.

It was a dark and stormy framework. However, I found secret hooks in the dingy rusty corners of the internet that would allow me to bypass the ugly leathery underbelly of this belching framework beast. My ex-girlfriend told me where to find the hooks in exchange for a little favor. Well, she's not really my ex. You see, on my last C# contract, I found a bug in the compiler and both her and I spent all night trying to...

Lack of Efficient Methods of Planning

[yosofun]

Communicating a nascent idea is hard enough, but most aren't taught (and few learn) how to efficiently plan their project. Things get haphazard, and code soon becomes dense and unmanageable, as the team attempts to meet deadlines without spending the time to develop solid structure.

Re:Lack of Efficient Methods of Planning

[procurementguy]

Exactly. It's all about designing and spec'ing a system out first.

Any big project should have the majority of the time designing and detailing the specfications; very little actual programming time and the rest testing. The programming part is easy. Detailing and Designing the system is where you seperate the professionals from the amateurs.

Politics not Engineering and Experience

[mccoma]

The sad part is that we look at this like an engineering problem. For people doing business related software, it really isn't. Mostly, we are trying to formalize a process that has "guidelines" and "best practices", but no true hard rules. There is no laws of physics in the business world. Obviously, there are some rules (e.g. accounting), but nothing that governs the whole process.

So, every piece of business software becomes an exercise in politics. We all hate politics. The mindset required to really like politics tends not to be prevalent in people who like programming. Politics are imprecise and have wiggle room that doesn't translate well into software.

The second part to this is that we are a young profession. Not even a hundred years of programming. We are craftsman, not engineers despite some people's job titles.

Wrong approach...

[slashmais]

You are wrong: "programming" isn't hard, what is "hard" is finding that unique algorithm that will precisely solve the problem on hand, once you have that, implementing becomes a breeze.

It all boils down to a search for algorithms.

Read up on the following to get an idea of why the "hardness" really exists: Rice's Theorem, the Halting Problem, the Totality problem

This guy must be some kind of dumbass

[edunbar93]

why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises.

You must be new to this world. In my world, every project that builds some non-trivial thing misses deadlines, blows budgets, and breaks promises. See also: Government, Engineering, and Construction.

Example: I just moved into a townhouse that I put a down payment on roughly 18 months ago. I was given at least 4 deadlines by which they were supposed to be finished. Each one blew by with a mighty whooshing sound. The price I paid miraculously did not go up.

Example: The Government of British Columbia built a fleet of fast catamaran ferries about 5 years ago. Except that they weren't fast, they weren't reliable, and they were astoundingly over budget. They were eventually sold for pennies on the dollar.

Hey, didn't we already figure this one out?

[cowboycarl]

Pick one:

a) cheap
b) fast
c) good

Software design is just like...

[ignavus]

Software design is just like building bridges ...

"I want a bridge from A to B ... but it needs to have a swimming pool in it. And an airport. You must be able to take off or cross over, depending on whether the vehicle is green or non-green. Except on Tuesdays, because on Tuesdays we actually need a boat ramp, not a bridge. But the swimming pool stays. But only the diving end.

We'll let you know how wide it will have to be halfway through construction. Well, actually, we are not sure that we really need to cross over that body of water. We might just be using the bridge as a car park. Our staff always need to have somewhere to park. Well, half the time. But Accounts would like to use part of the bridge to store their old end-of year reconciliations. Perhaps we could use those to prop up one end of the bridge, to save costs and materials."

Users often don't know what they want. Don't know what their own processes are. And often lack any vision about how to change them. Getting the user to be clear about what they need, and to sign off, is hard. And meanwhile they think you are the one that is holding up the project, because they have no idea how useless their "specifications" are. And you have to be nice to them. Patience is a programmer's virtue, if they are dealing directly with the client (our programmers do).

This is already the reason

[tcmak]

It is hard because there are people still raising this question.

Software development is not hard at all. Writing a hello world program is very simple.

The hard part is to meet client satisfaction, given the intangible nature of software.

Requirements in business system change quickly too.

Re:This is already the reason

[freedom_india]

Have you EVER handled a mid-size or a major software project in real life? Iam not talking about your college projects. Am talking about a T+0 system or a T+2 System which is just an internal Netting application.

Looks like obviously you haven't progressed beyond Hello World. Either you are a kid yet, or smoking something....

A simple 286 UCP project takes atleast 4 months to deliver from design to delivery ASSUMING: Requirements don;t change more than 20% of original specs, deployment environment is well known, technology is NOT esoteric, and the design is actually DoAble. Plus other factors like a well-knit team led by a good lead who helps, goads, cajoles and punishes to deliver. Oh BTW i forgot it would take 7 noobs or 3 experienced guys to deliver it on schedule.

After 11 years in IT, iam barely able to make deadlines especially on such simple projects even after following Tom DeMarco's "Debug-The-Design" approach which reduces debug time by about 30%.

Hard??? Grow up kid. Get a life, before you can post in this serious discussion.

Re:This is already the reason

[tcmak]

Make up your mind, and read carefully.

No matter how large the project you've handled, it is about meeting client expectations. Timeline is one.

If you still can't understanding, let me putting it more simple for you: It is not hard to develop a program. But it is hard to write a program meeting client expectations.

It is really a pity for you not to realise the root of the problems despite of the long years working in the industry.

I believe you really need to think carefully before you can post in this serious discussion.

Re:Gap between computer science and person problem

[bobetov]

I can't believe this is getting the ranking it is. Ok, where to start...

Currently with computers we are trying to abstract problems such as banking and business into simple logic puzzles. I think that's too much of a simplification. I think we need to create a virtual world full of basic human-percieved concepts, such as time, weather, humans, animals, etc. and create programs by manipulating those basic ideas and objects.

What you are describing is a domain specific language for the entire world. It would be the work of generations to construct such a language - a language that correctly modeled the human intuition about how the world works. How do we model gravity, friction, refraction, fluid dynamics, etc, etc to such a degree of fidelity that we can use that model to do work? We don't. We can't. This is the kind of wonky pipe-dream that infests the management levels of businesses and the ivory towers of academia. It's a great example of why software development is hard - people like you think you understand the problem, and convince people in power to do stupid things.

You, sir, are no software developer.

Computers *are* simple logical machines. We must render down complex human needs into "simple logic puzzles" because that is all computers do. That's it. At its heart, every computer made is a simple logic machine. EVERYTHING ends up ground down into the same fine paste of boolean logic and simple math. That is the tool, full stop.

Now imagine, instead of dealing with animals and where they live, it has a bunch of assertions about generally accepting accounting principles. One day, you might be able to just sit down and talk with an ontological system via email or IM, and say, "We got a check from client A for $575, another check for $440." and then the computer balances the books with all the other accounting principles it 'knows'.

This is just another variation in the strong AI pipe-dream. "Generally accept[ed] accounting principles"? What a laugh! Even in domains as staid and static as accounting, there are thousands of nuances to every decision. When and how to file expenses. How to categorize items. Which depreciation method to use, given dozens of unique conditions, to maximize tax benefits. Do you really think we can come up with a system that gets these choices right 99.999% of the time? Basically, you're waving your hands, and saying "given the proper libraries and world simulation software, development is easy." You're missing the point that developing those items would be a dozen orders of magnitude harder than just writing the software the way we do it today, with your much derided "logic puzzles".

Most people are wrong :)

[TheLink]

Actually IMO he and tons of people have got it wrong and that's why Software Dev is hard and software still sucks.

This is because most people don't understand the big difference between making software and say making a skyscraper (or bridge or airplane).

1) With software, the _prototype_ _BLUEPRINT_ actually compiles and runs and is usually sold as is (version 1.0 ;) ).

2) Too many people keep thinking that the programming part is like the _construction_ phase of a skyscraper, and that it should be planned and managed that way.

But they are WRONG, programming is more like the _DESIGN_ phase of a skyscraper: coming up with drafts then a detailed design, the the final blueprint where the position of every significant item is fixed.

The actual construction phase of the skyscraper is more like a _compile_ of the final version of the source code - the part where you go "make all".

Blueprint = source code. Skyscraper in use = executable being run.

3) The other important difference with software is:
The design cost of a software project is much more than the compiling cost.
The design cost of a skyscraper project is usually far less than the construction cost.

The owners of skyscraper projects are usually willing to spend a fair bit more on the design if _necessary_. No problem spending a million more to make sure the skyscraper "just works" - because it's still a fraction of the cost of building a new skyscraper.

But for the owners of software projects _each_ design and redesign adds significantly to the cost of the project (in time and money), and it doesn't matter whether it's the prototype, or the "final" version you are designing. Redesign = project costs 2 x more.

Constructing one thousand copies of the same "skyscraper" is trivial and cheap in software but not for civil engineering.

Designing one thousand _different_ "skyscrapers" is not trivial or cheap in software or in civil engineering.

And the advantage of designing skyscrapers is it is easier to _realize_ and explain to the bosses that they are asking you to break the laws of physics - "sorry you can't fit an extra 3 elevators there, this is how much space one takes, this is how much space there is".

Seems to me that most "Management", "Software Engineering" and "Project Management" people[1] don't understand these differences, or don't want to accept these differences.

And that's why most software sucks and will continue to suck.

[1] A project management trainer once asked a class I was in for example suggestions on how to speed up the building phase of a software project - he said for a civil eng project you'd add more machinery and people e.g. bulldozers, construction workers etc. So I suggested that the equivalent was more and faster CPUs, and he didn't like that.

I doubt the civil engineering bunch will keep adding fresh grads to speed up the design phase of a new building.

Should be solved in 20 yrs

[shashark]

In 20 yrs, Software will build Software. Humans will longer be required to code, test, and deploy. We will have software build software - and humans will just input the specs. No c#, no Java, no dev platforms, no operating system issues.

Believe me, its only a question of few years. Writing code will be as extinct as punching cards.

Re:Should be solved in 20 yrs

[Anonymous Coward]

"In 20 yrs, Software will build Software... - and humans will just input the specs... Writing code will be as extinct as punching cards."

Is it that time again already? This goal has been proposed many times in the history of computing. However, automatically generated code is notoriously hard to analyze. That makes it difficult to debug, maintain, modify, or test for robustness or even completeness. As a result, the method of specifying the desired result has to be very tight. This is why we don't see more widespread use of Genetic Algorithms, Neural Networks, etc.

To get software you can understand and trust, the compromise that is repeatedly implemented is higher and higher levels of computer languages, each level being an additional abstraction layer removed from the hardware. Machine language -> Assembly -> C -> Libraries -> Framworks/Toolkits/APIs -> Scripting languages -> Drag-and-drop "automated task" dialog boxes -> Powerpoint presentation -> functional spec -> requirements document -> customer request -> user complaining about broken software.

Some of those steps I just mentioned use code modules that were written in a lower level. Other steps are less clear in the definition of "written": is a compiled executable still hand-written, or was it software created from a higher-level spec known as C or Perl or PHP?

Oh come on!

[Viceroy Potatohead]

Really it's not that hard.... Exclusively use global variables, people, global variables! Software practically writes itself at that point.

Also, don't let other departments steel your interfaces. You wrote them, damnit, and they're your secrets!

On a more serious note, I'm more interested in who knows how to make it easy. From my experience, writing non-trivial software is an inherently difficult process, but perhaps there is something nobody is telling me...

Re:...because planning isn't doing and we want to

[dubl-u]

Likewise, with software development (and really most computer systems development), everyone wants to rush out and start coding as fast as possible...but if they'd made sure there were no assumptions, defined their problem specifically, and spent way more time than they wanted to developing an architecture, the coding afterwards would just be data entry for the most part.

I think for many interesting problems, this is effectively impossible. That's because the future environment into which your software must fit is, for many projects, unknowable. Even if you really could collect every current available fact and carefully work out every logical implication, that wouldn't be enough because there are teams of other people just as smart as you working hard to beat you any way they can.

In the end, I've just accepted that requirements will change all the time. Every week our team says, "Given everything that we have figured out so far, what's the most useful thing to build this week?"

To you I'm sure that sounds crazy, but let me add that I agree with your next points: everything should be tested, we should invest vigorously in good architecture, in highly reusable code, and the product should be kept spotless and taut, like a well-run sailing ship. I think the only place we differ is in that you'd like to put most of that effort up front, while I'd like to do it continously.

At least for my environment, that seems more sustainable. Instead of fighting people for time up front, I start coding as soon as they feel like they can name one week of work that they're sure will be useful. But when they ask me to hack something together, to cut corners that we'll clean up in that mythical land called "later", that's when I fight them tooth and nail. Every week I give them some features they want built the best way I know how.

IT development

[threesixty]

The reason software development is so hard simple. How many software projects have you worked on that were more or less exactly the same as the last one? Probably none. If you ever do work on a new project that is exactly the same as the last one it's usually the easiest thing in the world. Because you know EXACTLY what to expect and what's needed to make sure it all works. The very nature of IT projects is that your always doing new projects that have never been done before. Why write software that already exists? On top of this your working with a tool set that's constantly changing. Every 3 - 5 yrs programming languages change so much you might as well start from scratch. Since I started working in development I have used Progress, VB 3-6, .NET 1.1/2, ASP, ASP.NET 2 etc... All significantly different, which means your not really building on your knowledge. So a developer with 10 yrs experience does'nt neccessarily know more than one with 3 yrs experience in term of using the tools to do the job. If the same situation applied to real world engineering, hammers, nails, drills would be completey different every 5 yrs and every house built would be totally different in shape, materials etc.. than the last one built. In fact if you ever watch a documentary on buildings where radical designs are used, you'll find they have more or less the same problems software projects have. The buildings are never on time, they might have to restart half way through. The new tool they bought which hasnt been tested properly doesnt do the job etc.. The same problems as software projects have. In short, software development by its very nature does not build upon it's previous expriences. To gain competitive advantage IT has to be an ever changing process, in terms of tools used and technology implemented. Therefore IT development is a science but not an exact one and IT engineering is virtually non existent because engineering is about consistency, something IT will never have.

The best realistic explanation

[12357bd]

about software development i've ever saw was (and still is!) an almost 30 years old comic? poster, see http://people.bath.ac.uk/ccsjst/gifs/swing.jpg/, or google images for 'What the user wanted' to see other versions.

Re:The best realistic explanation

[12357bd]

Sorry, bad url, the good one is http://people.bath.ac.uk/ccsjst/gifs/swing.jpg

It's easy

[Heembo]

Software engineering is not any more difficult than other engineering disciplines. What makes software "hard" is that 1) Engineering methods are not rigorously used 2) Not enough people review the work of coders during the SDL My .02 cents!

People DO die

[Digital_Quartz]

> Perhaps if people died when software was not right like it does with bridges, things might change.

http://en.wikipedia.org/wiki/Therac-25

Re:People DO die

[cmat]

I'm going from memory here a bit (I read a few papers on these accidents a while back), but I recall that in these series of accidents, the manufacturer was not intentionally trying to screw the clients (hospitals directly, patients indirectly). Most of the mistakes were failures in the development/design process, as well as the support processes. This is NOT to say that ACEL was not responsible (because they sure as hell were imho), but puling this out to prove the "managment made us choose the bad solution!" meme is misleading at best.

Re:People DO die

[Tablizer]

Medical devices don't count. They should be engineered and tested more like bridges, and have extensive code reviews and QA. I would not want some late-hour VBA script running a cancer laser on my ass.

freedom versus the wicked witch ?

[planetfinder]

"I thought, I don't really know that much about software development."

Thats right Dorothy, you and Toto aren't in Kansas anymore. There's this evil thing called
software engineering ( sometimes referred to by prominent freedom coders as "the wicked witch" ).

Is there a correlation between the level of formal software engineering education and the belief that software
should be free ?

Amateur vs. pro

[Anonymous Brave Guy]

I think I agree with the underlying ideas of what you're saying, but I disagree with the comments about amateurs.

I don't play in an orchestra, but I do have a lot of experience in the world of dance. The current world champions in the style I dance turned professional recently, after winning the world amateur championships. In their first competition as professionals, I think they came fourth, i.e., although they were training as amateurs, they were comparable to the fourth best couple in the world by professional standards. Clearly this is the peak of achievement, something most of us can only aspire to, but the same pattern is repeated at lesser levels as well, with people reaching the top ranks of amateur competition in their country, turning pro, and almost immediately establishing a high ranking in pro competitions as well.

Perhaps more telling is that when organising events for my local club and looking for a couple to do a show, we've often found that top amateur couples give a more entertaining performance than some of the pros. Technically, they're not always quite as sharp (though this is by no means certain either way), but the passion and performance are all there, while sometimes it feels like a pro couple are just going through the motions. I've always put this down to the fact that the amateurs are doing it for love, not money. Every show is still a special occasion to them and not just earning enough cash to pay the rent, and they take pride in their performance and enjoy entertaining people for its own sake.

That love-or-money distinction applies to a great many fields, IME, and software is no exception. As numerous freeware, shareware, and open source apps have demonstrated over the years, amateur developers can be every bit as good as professionals, if they work effectively and (very importantly in this case) they collaborate well. IME, it's usually the latter that lets down larger-scale amateur software compared to professional stuff written by teams who are all working in the same place with professional management co-ordinating them.

Re:Amateur vs. pro

[Metasquares]

My post was ambiguous. I meant amateurs in the sense of unskilled beginners, not in the sense of those doing programming for the love of programming.

Re:...because planning isn't doing and we want to

[jofny]

Actually, we're almost completely in agreement. Even if you rebaseline your understanding of the project (and what's important to build first) weekly (Ive done it daily for some things), you're still planning before you code and making sure you completely understand your requirements before you code. The two go (as you rightly point out) hand in hand.

The only thing that I think really needs to be done up front that is much much harder to change in the middle of the project is the basic operating architecture of your project. In many cases, that's going to be the messaging bus and other similar components. Hell, most (??? my experience, don't know if that's fact) complex projects don't even have a robust messaging/services layer architected in at all. If you do that right up front, it makes making changes, adding new features, and scaling the project much easier down the road.

If you don't have that at all, your project will be much more painful.

But how to test it?

[cascadingstylesheet]

One day, you might be able to just sit down and talk with an ontological system via email or IM, and say, "We got a check from client A for $575, another check for $440." and then the computer balances the books with all the other accounting principles it 'knows'.

We'd better test this well, to keep our customers and the IRS happy.

Maybe we can check its results against some formalized rules. I wonder if we can automate that somehow? We could call it our "test program", or "program" for short ...

It's Denial, Baby!

[mooredynasty]

The powers that be never seem to want to know or pay the true cost of software projects. If more initial go/no-go decisions were based on reality there would be fewer project failures.

Re:Gap between computer science and person problem

[arevos]

It was summed up nicely a few years ago in a criticism of open-source software someone on the internet -- someone was complaining that the developers of GNUcash were dealing with memory leaks. I think they were using c or c++, and the complainer was saying they should migrate to java or python or something. If you're trying to make a computer program to solve problems, it's a waste of time to be solving computer problems. You should be solving the pre-existing human problem.

Ignoring problems won't make them go away. At some point you have to deal with memory management, either manually, as in plain C, or automatically, as in Java. It's not a waste of time solving 'computer' problems, because if you do not solve them, your program won't work. How do you think your ontological system will manage memory - by magic?

Currently with computers we are trying to abstract problems such as banking and business into simple logic puzzles. I think that's too much of a simplification. I think we need to create a virtual world full of basic human-percieved concepts, such as time, weather, humans, animals, etc. and create programs by manipulating those basic ideas and objects.

Easier said than done. Researchers have been experimenting with strong AI for decades, and have yet to produce any system that can operate in the same way you describe. Sure, you can string together simple logical concepts, but that's not how people think (apart from programmers). Natural language is littered with complex inferences, and designed to be processed by a great number of parallel analogue processors. Computers in 2006 lack the processing power, memory and architecture to operate an effective general interface with such a system.

The main culprit is

[BlindRobin]

having to deal with too many overwrought tools with too many extensions and partially compatible layers of overly extended somewhat coherent but not entirely consistent object collections attempting to isolate functionality and succeeding in broadcasting vulnerabilities and erratic performance.

Physics of Software

[Anonymous Coward]

He says there's no physics of software -- no rules.
He's wrong.

• The structure of the human organization pretty much prefigures the structure of the code.
• The longer you code, the bigger it gets. Predict code size by looking at coders multiplied by months. After you're past startup numbers.
• Communication is n-squared difficult.
• Higher level languages produce higher level results.
• Teams that work together on a project will work together better when and if the same team is used on the next project.
• If you build what the customer says they want, you may be building the wrong thing.

There's more, some of it undiscovered.

Not "what" but "who"...

[Anonymous Coward]

...and the answer is Microsoft, who "improwes" existing standards to be incompatible with the rest of the industry.
Another reason may be the whole sw-patent business...

What is SOX?

[jkauzlar]

Haven't heard this acronym before...

Re:What is SOX?

[ThrobbingGristle]

Sarbanes Oxley.

http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act

The IT Impacts section is what has made corporate IT/development environments crazy with
paper work and sign offs and arbitrary access limitations.

As if there wasn't enough of that before...

Cost.

[neimon]

You can....

1) Apply middling investments to compromised specifications with weak analysis, high turnover and poor deployment planning and pay a bucketload of money in the backend well in excess of any money you "saved" and STILL have a failure-in-place.
2) Do the job right, which will cost so much that everyone will realize that, gee, computing as we know it is actually quite expensive and you don't get that much of an ROI on large, complex, all-encompassing systems, especially after you count the fact it takes years to do a project right and the target moves.

How about

3a) Design a lightweight standard for software designed internally. Don't get caught up with the process becomming the deliverable. Do it in a month.
3b) Work on small, light, throw-awayable modules that do specific tactical functions and connect to each other through the lightweight framework in (3b). Never spend more than 6 months on a project. Don't be afraid to abandon projects that aren't working out because there's no point in feeding a dead horse.
3c) Engage engage engage the user population so they understand that IT is a craft, that it deserves respect, and that you're trying to help them.

The problem always comes down to how many orders of magnitude a given project is larger than human-scale. If you get beyond something that, say, 7 people can understand, you're screwed.

Compilers considered harmful.

[wikinerd]

I consider compilers and interpreters harmful. Programmers write a line of code and test it to see if it works without understanding what happens inside the computer. This "write - observe - fix" cycle gets repeated many times until the most visible bugs are fixed, but nothing guarantees that the system does not contains hard-to-detect bugs deep internally. We need more software engineers, but most people who use this title in companies are either normal programmers or systems analysts, or project managers. When was the last time you used non-deterministic finite state automata, Z notation, or even basic flowcharts at your company? Never used such methods? That's why your software sucks and you need a year to write simple applications that could be written in a month if you used a proper methodology and good methods.

Re:Gap between computer science and person problem

[lawpoop]

"Ignoring problems won't make them go away. At some point you have to deal with memory management, either manually, as in plain C, or automatically, as in Java. It's not a waste of time solving 'computer' problems, because if you do not solve them, your program won't work. How do you think your ontological system will manage memory - by magic?"

How does Java's or Python's garbage collection work -- magic? No, what happened was that people solved the problem once already, in a re-useable way, so that we don't have keep solving them every time we write a program. The idea is to make progress, to make things easier. If you're writing an accounting program, you should be worrying about accounting problems, not memory management. Or do you think all programs should be written in assembly?

I'm not saying that Java's memory management has reached Nirvana and there aren't any more improvements to be made. I'm just saying that not every programmer should be working on it, especially when you're trying to solve other problems. Sure, memory issues will show up in any programming project you undertake, but that doesn't mean that every program should be optimized to that level, or that every programmer should be able to work at that level.

It's like saying a figure animator at Pixar needs to know how to push memory around in a stack, and that will help him when he is animating a running lion on a computer. No, it won't. We make higher-level programming tools so he doesn't *have* to worry about registers and stacks. Tha animator needs to study physiology and anatomy to be a better animator on a computer, not boolean logic.

He doesn't need to learn about auto-repair to help him drive to work in the morning. If his car has trouble, call AAA and let him spend his valuable time learning to be a better artist. We have a complex, diversified society so that we can have experts that push the boundaries and advance our culture, whether it is in computer science or animation.

Like I said, there will always be a place for low-level programming, boolean logic, OS development, etc. But I think we think of programming too often as computer science, rather than as a way to create tools to allow people to solve problems in other areas of life. The perspective we seem to use is "How can we translate problems from other areas into computer science problems" rather than "How can we make programs that are more helpful in solving other people's problems?" Introducing an accountant to memory management is *not* progress.

Basically all I'm saying is that for most programming jobs, we are still working at too low of a level. We seem to think that real programmers need to work at the memory-manipulating level, and if you're not doing that, you're not doing real programming. Well, not every smart, talented person working in programming needs to be doing memory management.

Re:Gap between computer science and person problem

[arevos]

How does Java's or Python's garbage collection work -- magic? No, what happened was that people solved the problem once already, in a re-useable way, so that we don't have keep solving them every time we write a program. The idea is to make progress, to make things easier. If you're writing an accounting program, you should be worrying about accounting problems, not memory management. Or do you think all programs should be written in assembly? I agree with you in general, you just gave the impression that it didn't matter whether GNUcash was programmed in C or whether in Python or Java. It sounded as if you dismissed such issues as irrelevant, when I'd have thought that programming GNUcash in, say, Python, sounds a reasonable idea.

Basically all I'm saying is that for most programming jobs, we are still working at too low of a level. We seem to think that real programmers need to work at the memory-manipulating level, and if you're not doing that, you're not doing real programming. Well, not every smart, talented person working in programming needs to be doing memory management. Again, I agree with you that low level programming is unnecessary for many tasks. However, it's a mistake to think that programming in a high level language is necessarily easier than programming in a low level language, any more than a 3D artist working for Pixar has an easier time than the programmer who is developing the animation package the artist will use. Whilst high level languages allow you to achieve simple things with less effort than in a low level language, they also pave the way for added complexities.

For instance, contrast assembly to a language like Java. You can create a dynamically sized list of numbers easily in Java, whilst doing the same thing in assembly would be a lot of work. On the other hand, the problems that a Java programmer has to deal with are usually more complex than those an assembly programmer would need to face, and Java is also a relatively more complex language. It seems to me as if programming won't get any easier in future; the problems will just get harder.

Furthermore, just because a language is higher level, doesn't make it easier to understand or comprehend. Compare these two code snippets:

In Java:

void main(string[] args)
{
    int a = 1;
    int b = 1;
 
    System.out.println(a);
    System.out.println(b);
 
    for (int i = 2; i < 10; i++)
    {
        int x = b;
        b = a + b;
        a = b;
        System.out.println(b);
    }
}

In Haskell:

fibonacci = 1 : 1 : zipWith (+) fibonacci (tail fibonacci)
main = mapM (putStrLn . show) (take 10 fibonacci)

Whilst Haskell is clearly the more powerful and expressive language, it also relies on concepts that aren't immediately obvious. It seems to me that there's no guarantee that programming in the future will be any simpler than it is now.

There are many parts to the problem

[rfc1394]

I believe our biggest problem is the lack of adequate abstractions to be able to describe software in such a way as to reduce the amount of grunt work needed in the development of applications. We have not-very good tools and poor languages to describe what we want to accomplish. Plus, the people developing software are often not well trained, and the concepts are difficult, and often the customer doesn't even know what he wants, and may not even know how to articulate what he wants. And even if he knows, and knows how to tell what he wants, he may not know how to get the developers on-line to understand his vision. (Every writer who has ever sold a book to Hollywood and saw the resulting movie, complains about how the screenwriter and director "botched" the translation and "bastardized" his or her story, so the idea of "concept" not equaling "implementation" isn't new.)

The point which can be made as far as software is concerned is that the higher the level of abstraction the greater the productivity the person is capable of producing. A programmer in a language like Cobol is going to be two to three times as productive as one in assembly language. A software designer using a system like, say, Ruby on Rails might be able to do ten times as much work as someone writing an application using, say, C or C++, presuming the target task is the same. I am using this as an example, I do not know if something like Ruby on Rails actually can give a full order of magnitude increase in productivity. But I wouldn't be surprised.

The only problem is that if you take abstraction to its ultimate conclusion, you get a language like APL which, except for some niche applications, failed dismally because it became too difficult to work with. But you could do some amazing things with it. The "big thing" in APL was to write a "one liner," an attempt to write a complete working application in one line of code. And I have seen some unbelievable stuff done with it. Things that conceivably might take dozens or hundreds of lines of code in a lesser programming language really could be done in *one line* of APL. I've seen it done. It kind of convinced me I'd never be able to do that kind of work, I don't have the background.

But to have that kind of capability requires a really powerful programming language, and/or excellent subroutine libraries to support it. As well as good people to write code using that language. It is often said of APL that it is a "write only" language in that some people would develop cute tricks but if you ever had to do maintenance it would be simpler and easier to start over.

But if you are even just competent in APL, the level of abstraction of the language is so high that your productivity will easily be at least an order of magnitude - maybe even two orders - higher than someone of the same capability working in any other programming language. And a real "superstar" could do things that might not even be possible even in a considerably longer period of time in anything else.

Our minds determine the tools we have. And the tools we have, which basically consist, as I have stated, of not-very-good code editors, poor programming languages, and inadequate run-time support, don't make it surprising that we have such problems in the development of software. But people are thinking about ways of improving it.

There is a guy named Louis Savain who has been doing some work from time-to-time on his idea: the creation of software modules using the concept of a hardware abstraction, limiting the number of interactions and cross-actions by limiting how side effects can occur. He was of the opinion that he had a great system for reducing error, increasing productivity and reliability of software. I pointed out to him that what he has is an *idea*. Unless and until he actually has something implemented all he has is a promise which may be useful but is at this time unproven.

Basically he wants to create for software what the transistor and the printed circuit did for

Because it's not a legitimate discipline

[Anonymous Coward]

Software eng is hard because it's right up there with Political Science and night soil carrier in that there is not real theoretical basis, no established codes of conduct or construction, no transferable teachable skills and so on.

Software Engrs as defined today should be banned from writing anything other than compilers and operating systems. Especially they should NOT be allowed to build any application whatsoever. Applications should be mandatorily developed by specially trained users of the application domain itself.

Anonimass

Divide and conquer

[Per Abrahamsen]

Software development is not hard. There is really just a single technique you have to learn, and it predates software development with a few millennia. It is divide and conquer.

When you have a large problem, divide it into smaller problems that can be dealt with separately. All the other techniques and tools (OOP, structured programming, extreme programming, top-down, bottom-up, rapid prototyping, waterfall model, etc) are just specializations of this. It sounds like you quickly got bogged down by the details of programming languages and tools, happens a lot to younger programmers. Just remember that these details are just details, and not the real issue.

The only slightly tricky part in software development is how to divide the problem so that each subproblem can studied in isolation, or at least with a minimum of information about the other subproblems.

Conducting styles

[gidds]

It depends very much on the conductor. (Okay, my experience is more in choral rather than orchestral music, but I think the principle's the same.)

I've known conductors who did all the work in rehearsal, and in performance did just enough to keep people together. I've known conductors who have to work really really hard in performance to keep inexperienced and unskilled people doing roughly the right things at the right time. I've known inspirational conductors who can bring out meaning in a piece that was never there before, just from subtle changes in stress and timing. I've known conductors who can completely change the mood and style of a piece to match the flow of the programme or the mood of an audience.

Of course, it very much depends upon how well the performers know the music, how well they perform, and how well they know the conductor -- and of course, how much attention they're paying to him or her!

In performance, some conductors concentrate on low-level issues: exact timing, articulation, pitch. Some think more about overall balance, texture, and mood. Some think mainly about the melody, and mostly leave the rest to take care of itself.

And no one of these is right or wrong. It depends so much on the performers, the music, the occasion, and the conductor's own interests and preferences.

Conducting styles vary, too. Even when simply beating time, some conductors put the beat at the end of the downstroke, or halfway through, or during the following upstroke. (Some conductors don't beat strict time at all, just wave their arms in their own time to give a vague idea of when to speed up or slow down.) Some add expression to their beating time; others use the other hand or even both hands for that. And so on.

As to orchestras ignoring the conductor, I've heard tales of orchestras who decide after a rehearsal or two that the conductor is useless, and so the performance is secretly conducted by the lead violinist!

(Relating this all back to software project management is left as an exercise for the player^H^H^H^H^H^Hreader.)

We need to be more like home contractors

[bill_mcgonigle]

Building a complex application from scratch is little different than building a mansion using no COTS parts.

I know a contractor who's favorite line is, "yep, that'd be about a thousand bucks".

"Can you just move the stairs six inches to the left"?
"Yep, that'd be about a thousand bucks."

See?, works great!

Many software developers tend to say, "OK, I can have that for you by 5", because their estimates tend to be the first point in time with a non-zero probability of being true. So, time management is another recurring issue in this circle, and inevitably intertwined.

"Can you move this button over here and make it blue?"
"Yep, that'd be about a thousand bucks."

Feels better, no?