Slashdot Mirror
Why Software is Hard
GoCanes writes "Salon's Scott Rosenberg explains why even small-scale programming projects can take years to complete, one programmer is often better than two, and the meaning of 'Rosenberg's Law.' After almost 50 years, the state of the art is still pretty darn bad. His point is that as long as you're trying to do something that has already been done, then you have an adequate frame of reference to estimate how long it will take/cost. But if software is at all interesting, it's because no one else has done it before."
Programming websites that let you actually view a page without requiring a cookie is obviously hard for the folks at Salon.
OMG!!! Ponies!!!
An approach like this would probably have been helpful in FBI's failed $100 million debacle the Virtual Case File system
It should be pointed out that the author is not a software engineer and really does not know what he is talking about. Professional developers who want to understand what makes software development difficult should read some of the textbooks on software engineering that he quotes.
Wrong tools: C and C++ language. Overcomplicated APIs: Win32, HTML, drivers, JAVA, etc... Lack of standards enforcement: HTML disaster. All this makes it easy to derail any project.
I design buildings for a living, and I've dabbled in programming, and I think architecture and software development have a whole lot in common.
Your step one in "building a house" can go through all 6 of the steps that you have listed for software development. We get hired by clients, sometimes they have a good idea what they want, sometimes they don't. Sometimes what they want is feasible, sometimes it isn't. It's not unusual for even smaller projects to drag on for years, because the client keeps changing his/her mind. Many projects that cross our desks will never be built.
Many projects are not the traditional design phase ->building phase. They often overlap, and it's pretty messy.
I could go on for paragraphs with the similarities that I see between software design and architecture, but I'll save that for another post.
One time I threw a brick at a duck.
I think that once someone improves the situation of software architecture and programming languages so that programmers don't have to mess with ad-hoc hacks but instead write the logic that they want to implement, then software will cease to suck.
The main problem is Operating Systems architecture and Programming Languages.
Due to lack of time, I will only list a few of the Operating Systems problems that weren't solved after more than 30 years of OS development:
This would also get rid of the unnecessary bootup/shutdown sequence all programs are currently dealing with.
Instead, a capability-security model should be used (not POSIX capabilities, but EROS/KeyKos type ones), which is much simpler to use, verify and much more powerful and fine-grained. This would also facilitate secure movement of components between computers - which could be done automatically by the OS to improve performance. More on that on a later post.
One thing I've noticed about companies is that they try to treat programmers like factory workers. Expect each one to be interchangeable and jump in anywhere on the "assembly line" at any place at any time for any piece of code. However, programming takes understanding, and complex programming takes complex understanding. Even a good programmer fixing a bug may need to analyze surrounding code for several hours before changing a single line.
Unlike most engineering projects that are completed and done, most programming is a living growing process that is constantly changed modified and improved.
That implies that there is a need for specialisation and clear boundries, to assign "ownership" or "territory" over certain parts of code. A programmer who understands it and gets the last say on how it's changed and have clear non-arbitrary rules for changing that "territory". Like in open source projects. If you want a kernel fix, you submit it to the proper maintainers, or make your own fork, but no corporate bureaucrat comes along and micromanage how the code is merged and managed.
People love to ask why we can't build software like we build bridges. Well, we've been building bridges for what? Several thousand years now? I bet if we look back at the first 50 years of bridge building (which is about how long we've been building software, give or take) we'd see similar mistakes being made to how we are trying to build software. In a thousand years or so, I'm sure we'll have enough accumulated knowledge of how to build software that these silly questions of why we can't build software easily now will seem childish at best.
Utter bullshit. I use UML for not only analysis, but design, programming and working on things in daily life. It's a matter of understanding the techniques. I've designed four cooperative wire transfer subsystems using it myself.
"we need to listen to what Brooks said... more specifically their knowledge and experience."
Basically what I said.
"Solid developers will produce solid software."
Ibid.
Most cooking projects don't take more than 10 man-hours, but pretty much every programming project does. And, furthermore, mostly when the chef makes a mistake it's obvious to her.
:)
Neither condition hold for programming. It's for this reason that I think that, in general, *two* programmers can program faster than one. At least, me and my partner can program code that's more bug-free together than we can when we program separate projects, and that makes a difference. If the project is sufficiently large - i.e. takes longer than about 10 hours, the cost of communication between two people is less than the cost of switching.
While we're at it, I think that there's another misconceptions in this interview.
programmers are programmers because they like to code -- given a choice between learning someone else's code and just sitting down and writing their own, they will always do the latter
Two of the five developers at my little software company are programmers because they like to figure things out. So we almost always figure someone else's code out before we do anything ourselves. There are varying degrees of this in a lot of the developers we've got there. I would say that none of us will write anything ourselves unless it saves us a considerable period of time.
But even more, if you had a relative who was always wondering, "What is it that you do all day?" you could hand my book to that relative and say, This is what my work is really like.
No. I couldn't. My experience as a developer is nothing like what he's described. And he didn't talk about the phenomenon of unknowns that I've noticed - for every project I do, if I estimate how long the known things will take, dealing with unknowns will generally take 60% longer (so multiple time estimates by 3 is generally correct). He didn't talk at all about testing.
Almost everything he talked about are things that I thought would be true when I started but that have ended up more or less untrue. Discipline coding makes a difference. Automated unit testing catches most problems, and regression testing finds almost all the rest, and not everybody does these things.
Mod me down and I will become more powerful than you can possibly imagine!
Actually, every instructor I've had works in the industry. Not *DID WORK*....but *WORKS*. Classes are at night. It's in Austin, so there are plenty of studios to pull from. I've had instructors that have worked on games from all eras and genres. Some of the companies that represents: Sony and SOE, Midway, NCSoft, and Microsoft. Plenty who have started their own studios after having worked at bigger ones, too.
http://www.austincc.edu/techcert/Video_Games.html
It's not a degree program (yet), but I'm not too worried about that since I already have a CS degree. For me, it's more about having fun, learning some new stuff, and making good contacts for when I'm ready to jump into the industry.
Check out the list of names on the Advisory Board and the list of Instructors. There are some influential names on that list.
Layne
Well another problem is in building a house there is a definate line draw between architect and contractor/builder. You generally know whos fault it is when something goes wrong with a building. With software you often have part time contractors who think they are architects, and noone really knows better.
Software engineering is NOT manufacturing
I'd argue that most software projects are far from "engineering".
Here we are, using a construction kit (modern OO language) that is 100% predictable and designable, yet beta software and patch cycle after patch cycle are the norm. You couldn't do that when you build a sky scraper - they get it right the first time. It's the level of design, prototyping, QA and intrinsic belt-and-braces attention to detail that is missing in most software projects.
That said, I don't think many software projects would be given enough money to achieve this anyway, because the consequences usually aren't important enough (no-one's life is on the line).
Try doing advanced math using the roman numeral system (no translation).
The analogy is that programming is today being approached in a manner that is far more limiting then it needs to be.
There are those who claim programming is nothing more than mathmatical algorithims, but it is more as programmers create higher level abstractions to deal with lower level abstractions faster. Sure it can be all boiled down to mathmatical algorithims and even down to binary or machine language but it is the higher levle of abstraction where software is today created.
The science of software has failed or been distracted from the genuine objective of identifying and defining abstraction physics. For it is Abstraction that is the essence of programming, and there most certainly is a physics that applies to our creation and use of abstractions.
We create abstractions in order to simplify or automate complexity of lower level abstractions, down to binary.
The failure is that of not recognizing what we all constantly do, what action constants we apply in our creation and use of abstractions.
Its like doing chemistry before we came up with the understanding to create the table of elements. We didn't understand the underlying mechanics. But once we understood these underlying mechanics, we created chemical megaplants.
Though we would not create software megaplants, in understanding abstraction physics, we would do what was accomplished with teh conversion of math from roman numerals to the hindu arbic decimal system. We'd make programming easy enough that the adverage user would do alot more for themselves, just as the general population was able to not only do math for themselves in teh conversion of symbols used (roman numerals to decial) but were able to do more advanced math then the roman numeral elite accountants were able to do.
Of course the problem is in conversion, as it took 300 years for the conversion to happen. It took 350 years for Galelio to be exonerated....ask the catholic church why... and know why the industry of programming, and regardless of what side of the fence you are on (proprietary or open source), presents resistance to the needed change.
Programming is hard, because the industry wants it to be. So to keep the elitism, social status and pay scale.
Of course social demands weigh in on the change happening. As computers today could not have been created using the roman numeral system of math. It won't be hundreds of year for this change to happen, as we already can't keep up using the lessor/harder route.
Abstraction Physics
I sometimes wonder if it's a questions of people who naturally see things as images, versus people who don't.
Anyway, I'm with the grandparent. A small class or state diagram can be useful to me, but I get lost very quickly in a big or detailed one. And when I go into details, I soon find that what I want to say is easier to express in text, where I am not limited to the few languages permitted by the UML.
.....There is no single answer to why software is hard ......
/. shouldn't be too harsh on MS and their trials and tribulations with VISTA.
Maybe the biggest reason is that writing software is an art rather than a science, more akin to writing a symphony or poem. There are PRECISE mathematical formulas and design rules which an engineer of physical things, such as a bridge, for example, can apply. Software, being non-physical, is fundamentally different from any physical device or machine which can be 'engineered' by precise mathematical procedures. Perhaps the very term "software engineer" is a misnomer in the same way that applying the term "engineer" would be to a musician.
The hardest sort of software to write is that which is intended to interoperate with people. Computers are precise, mathematical, deterministic machines, whereas people are not. No amount of "maturing" of the software industry will ever change the fact that writing most software is not and never will be an exact science. Folks here at
All theory is gray
Mostly, in my experience, programmers aren't trained much at all.
There is an old saying: if you think education is expensive, try ignorance. Well, we have been, and early results are pretty much in line with predictions.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
This thread is full of analogy after analogy.
I think if you want to have a discussion amongst people who develop a software you have to ditch the analogies, because none apply. The reasons software development takes longer that you might think, or the reasons software is difficult to create, sometimes doesn't give the expected return, sometimes is buggy, etc, doesn't have anything to do with cars or buildings or spacecraft.
// MD_Update(&m,buf,j);
Couldn't agree more. And the reason is I know I'm one of those untrained people. I'm also probably the only one among my peers that thinks we have no business calling ourselves software engineers.
Judging by a bunch of people I've met and worked with, schools fail to teach basic problem-solving skills, professionalism, or indeed, even the fundamental tools of software engineering. Instead they focus on teaching a specific language or tool, and the students never rise above that. Hence we get a lot of Blub programmers.
For quite some time I've been thinking that we do stuff ass backwards: we go to school, take a stab at learning something useful and then go to work where very little of the teachings are applicable. I wonder how things would be if students first got to work in real-life conditions and then got to decide what their learning focus should be.
Software development is hard because of a misconception that knowing how to program in a particular programing language makes someone a good developer. Just because someone can learn a foreign language (i.e. an American who's native language is English can learn to read and write in French) but it does not mean that person will be able to write a good novel in that language.
I'm currently working towards my Masters in computer science and although I don't intend to ever work as a software developer programing concepts make up the majority of the curriculum. The majority of the graduate program, in computer science, is made up of international students (the university is in the US) of whom most hail from India. After working with these students for some time now I have learned that they obtained their undergraduate degrees from India where computer science is taught in theory. This means that they got their degree without ever touching a computer. Combine that with the fact that they have never owned a computer themselves makes me doubt the quality of their education. For those of you that know some C++ you will understand their level of knowledge when I say that they enter the program without understanding the concept of pointers (dynamically allocated memory instead of compile time arrays), structures, or object oriented programing concepts. Luckily many of these students do not make it past the remedial courses of the graduate program but most of them switch to the IT program under the business college of the university. Unfortunately I am sure many of these students go on to be programmers in their country where there is a big demand due to the trend to outsource coding projects to their country.
I have spoken with other students in similar programs at different universities and this seems to be a widespread scenario. So with people that barely understand the language they are programing in are being asked to write programs for consumer use. It seems that very little time goes into educating students on how to program. Here I am using the word "program" to mean given a problem utilize one's analytical skills and artistic ability (yes programing takes artistic ability) to conceive a solution and write it in a programing language in such a way that the language compiler and linker produces an executable program. With this in mind you add the fact that virtually no effort goes into teaching people how to move beyond this basic level of understanding of specific programing languages to a point where any competent programmer would consider them to be fluent in the language they are developing in. Would anyone buy a book written by someone that is not fluent in the language that the book is written in? I think not.
Beyond the fact that the majority of programs are being written by people who are not competent, by any standard, is the fact that programming is not just an application of knowledge of a specific language but an abstract concept that is created from the individuals understanding of the problem and their ability to conceive a solution entirely from within their own mind. This is where the artistic ability comes in. If you give 100 people a problem and ask for a solution in the form of a program you will get 100 different solutions. Out of those 100 people only a handful of the solutions will meet the standards of what could be considered efficient coding.
Consider that the programmers that I am speaking of here are people that entered into a post graduate computer science program. These people do not make up the majority of programmers out there. What makes up the body of the worlds programmers are people that have not attempted to progress to this level. Most enter the workforce after receiving their undergraduate degree and many have not received any higher education at all. It is my belief that aside from what can be taught to an individual programming requires an inherit ability within that individual for them to be able to produce what I would call quality code.
Encryption: I may not agree with what you say, but I will defend your right to encrypt it...
The article's title gives an indication of this (as some other comments have pointed out): it talks about "programming" but not "software engineering" as a whole.
Still many companies hire people with not enough computer science knowledge, for performing software engineering tasks. You can do this, but the results cannot be good (at least in the long term).
I think this is because software is usually successful, in the short term. It apparently solves the problem and the customer gets satisfied. Therefore, why "losing" time and money making documents (where experience gets archived) or performing a good design?
If you create software, how often do you (your organization) apply these concepts?:
- Life cycle of a project,
- Gantt and Pert diagrams,
- Risk management,
- Ishikawa diagrams,
- Code complexity,
- Software quality assurance,
- Coding style standards,
- Documentation standards,
- Software patterns...
Have you ever wondered why Linux is still failing to widely conquer business marketplace?
Satisfaction - both for companies and indivual programmers - should be switched from being creative to getting a good job done. We still have much to learn.
1) Social-economic processes impose changing biases to the current paradigms of software development. This leads to a biggest reason of code not being reused and ever-changing development environments.
2) As usual in the bussines world, the driving force is only the first derivative of the cost. Local minima can be quite stable. Morover the local minima are deterimed in a way wich seldom integrates over a long time (e.g. support).
What i mean is: Todays product lifecycle is *assumed to be* shorter than ten years ago. Tell to somebody today that there will be a phase of a few months with no visible results in terms of the final product will yield a different response that ten years ago. Sadly this means that porting code (which, if you do it right, takes most of the times longer than developing "something" with a "state of the art" Development environment) to be usable on a new platform is underrated.
However, old code very often has no semantical problems any more and all these "small bugs", like implementing a special semantic for a certain parameter value or realizing that a certain database lookup will fail under certain circumstances because of some kind strange constellation happening seldom but on a semantical level, are fixed.
Hate to break it to you, but you should learn to read the entire comment and not just the first sentence. Just in case it is too difficult for you to understand what is said here:
This touches on all avenues of life and business, the best people for the job are not hired, because friends of the friends are hired and those people more often than not are not going to do the job right. - see? Had you read the rest of the comment your comment would have been totally unnecessary, and so would have been this explanation.
Cheers
You can't handle the truth.