Slashdot Mirror
A Unified Theory of Software Evolution
jso888 writes "Salon has a nice article today on Meir Lehman's work on how software evolves and is developed. Lehman's investigation of the IBM OS/360 development process became the foundation for Brooks' Law: "Adding manpower to a late software project makes it later." He is hopeful that his work will make software development less of an art and more of an engineering science."
Software doesn't evolve by chance, folks, it is DESIGNED by its CREATORS.
/. is a site for SERIOUS INTELLECTUAL DISCUSSION.
Please check your crackpot theories and psuedo-science at the door.
Thank you.
dinner: it's what's for beer
very few projects should have more than 10 programmers (if any).
You realize you just suggested that very few software projects should have any programmers. How is the project going to get completed without anyone working on it?
While "Brook's law" might be a law, it's only useful in retrospect. Most software projects have no idea how far behind they really are. So basically, you can always add manpower, you're really only half way through anyways...
My boss seems to think that having a lot of meetings about it will do the trick.
"When I first wrote about this topic, nobody took a blind bit of notice."
No, sir, I did and many collegues who were also interested in good timely work. We lent your books to each other with the notion "that's something you should read".
Great to hear that you are still alife and enjoying to give programmers and their managers something to look at and something worth to read and think about.
Youngsters, better pay respect to this old software camel with the hole in the sole of his shoe (and probably also in his all-too British pullover), or I DDOS your toilet!
"Unless IBM programmers had suddenly figured out a way to write error-free code -- an unlikely assumption -- Lehman made a dire prediction: OS/360 was heading over a cliff. IBM, in stressing growth over source-code maintenance, would soon be in need of a successor operating system."
Which means that commerical systems don't so much evolve as stub their growth paths out and switch direction or spawn new generations because embedded complexity has killed off the feasibility of maintaining it. In other words, all new releases are the cause of and ultimately an attempt to escape from, the chimera that is overly complex code. In commercial terms this should be astounding. We're paying to gronk up our own because we erroneously believe the NEXT version will be something radically new and elegant which of course it can't be.
New Version "x+1.y" is simply an ejection seat.
I'm not attempting to flamebait here, just submitting an observation. It seems to me that many of the complexity issues can be overcome by designing better languages. I've never stopped scratching my head over the perseverance of old languages like C++ and FORTRAN. Sure, they are extremely useful in the hands of experienced folks, but they need to die. They were good solutions to problems decades ago, but so much has been learned since then and the constraints of sparse computer resources and CPU speed have moved a lot.
From the article:
Michael Godfrey, a University of Waterloo scientist, is equally hesitant but still finds the Lehman approach useful. In 2000, Godfrey and a fellow Waterloo researcher, Qiang Tu, released a study showing that several open-source software programs, including the Linux kernel and fetchmail, were growing at geometric rates, breaking the inverse squared barrier constraining most traditionally built programs. Although the discovery validated arguments within the software development community that large system development is best handled in an open-source manner, Godfrey says he is currently looking for ways to refine the quantitative approach to make it more meaningful.
It would have been interesting had they delved deeper into this finding. Yeah, I know, the true believers in open source all feel superior (we are, aren't we?), but exploring the reasons why it works would be interesting.
Is it the large-scale peer-review process? Is it that we occasionally rewrite parts (filesystems, VMM, etc)? Something else?
Although the performance audit showed that IBM researchers were churning out code at a steady rate, Lehman found the level of debugging activity per individual software module to be decreasing at an equal rate; in other words, programmers were spending less and less time fixing problems in the code. Unless IBM programmers had suddenly figured out a way to write error-free code -- an unlikely assumption -- Lehman made a dire prediction: OS/360 was heading over a cliff. IBM, in stressing growth over source-code maintenance, would soon be in need of a successor operating system.
Except that the "[dire] need of a successor operating system" isn't so dire at all: the world's richest man didn't get where he got by writing code that didn't need to be replaced by a successor operating system, did he? The whole premise is to produce something that works now, and when it stops working later, you sell a later version. Heck, just a couple of months ago, Billy announced that 92.3% of the calendar year would focus on new code, leaving the rest for the old.
What's smarter, coding the Microsoft way, or coding a server that's been up since before Windows NT was released, without a patch in 7 years, handling half a megabit of data both upstream and down, every second of every day forever. Where's the revenue?
~r~
Note: the 92.3% figure might only be for the year 2002, with later years being still closer to 100%.
Someone else called this oversimplification. It all depends on the project.
_ __
The number of programmers needed on a project depends upon the number of software modules in said project. Each programmer working on their module and with the other programmers and project managers for the sake of integration and communication between modules talking to each other. I am not a project manager so I do not have the magic formula but there needs to be some serious research in the IT industry of how many programmers are needed per project for the number of independent sections or modules of software being created.
Then and only then will you have a situation of better utilized output over a large group of programming talent.
10 may me too few programmers for some huge program and too many for plenty of other projects.
_______________________________________________
ACK
If you've got the requirements well enough defined in terms of previous work that you can estimate accurately, then most likely all you've got to do is cut and paste the old code anyhow...
I've been trained in that stuff. It's wonderful in theory. In practice? All the metrics only work if you are doing the same stuff you've done before. If you are doing something new, then they don't work. Which is why few people actually use them.
Looks good on a resume, though.
Best Slashdot Co
I worked on a system running OS/360, up to Release 23 or so, when IBM 'retired' it.
We installed new Releases about once every 6 months. IBM also had 'patches' available for about 19,000 known bugs.
These patches were not incorporated into the latest release because each of them, if installed, broke some other aspect of the OS.
We, and every other site, only installed those patches needed to work around problems that the particular site encountered. And you always hoped that today's patch would not break something else that your users needed.
Not only is this not true, it's impossible to do this in practice. If you do this, you'll find that you still blow a lot of time on design, development takes longer, because your design is unnecessarily abstract, and your design proves inadequate for something that you need to implement further down the road. Requirements change, and this has consequences for the design. The best one can hope for is that the basic architecture is robust enough that it doesn't require a complete upheaval.
What is necessary is a method for changing design gracefully. "Refactoring" is the best source I've seen that addresses this. Basically, you change methodically, and you test.
To a Lisp hacker, XML is S-expressions in drag.
Back in the early 1980s I headed up a small team that developed 'industrial strength' applications.
Our firm licenced this software to major manufacturing firms with a Money Back Guarantee. As in, "If you are not satisfied, for any reason, we will either fix the problem or give you back your money. Your choice." We were never asked for a refund.
It was semi-open source. You could have the source any time you wanted, but asking for the source voided your warranty, since problems in your data might have been caused by your own temporary code changes.
Funny thing. I've had that on my resume for many years, but no prospective employer has ever asked how I did it.
No one has hired me specifically to help them produce similar quality code. Much of the time their reaction to my resume is, 'but you don't know c++' (or their other favorite). I know enough about c++ to know that I want to stay away from that second generation language for all but the most specialized situations.
I have also been told, on numerous occasions, that I'm not qualified to lead a particular project because I lack experience mannaging the large team that will be needed. I've never gained that experience because I've never needed a large team to accomplish anything.
As an MBA, as well as being an application designer & a coder, I know that large teams do have a place -- mostly where you have a blank cheque and are earning a percentage of the total billing. (:-)
A quick warning... I consider myself a relative newborn in the world of software development. I present these opinions under the consideration that my opinions can change at any moment. =]
A lot of the dire predictions of software atrophy and such are a result of applying the wrong methodology to a project. Yes there are uses for Software engineering, but I think this approach is overkill for even large scale projects. Check out Software Craftsmanship: The New Imperative for a different perspective. A perspective I think is in need of serious consideration. The gist is returning to the days of master craftsman and apprenticeships. This focuses a bit more on the learning aspect than actual development methodologies, but you can always go to The Pragmatic Programmer to fill in that gap.
"As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress."
This is where "refactoring" (see Fowler's Refactoring) really shines. I find it difficult to believe that refining the software base is not progress. An initial revision where the code functions by its contract (if your into designing by contract), then you refactor the body of the function/method for speed / elegance. Then you can run your unit tests on the function / method to test that the refactoring session did not break any of the design contracts (whew).
I think they may be trying to restate the broken window theory (see Pragmatic Programmer), were a broken window (or bug) in a building (or system) leads to delapidation elsewhere in the building (or system).
And then there are the agile methods, including XP. I think these answer a lot of the limitations and issues with Software Engineering practices. Interacting with clients (having a client there during each iteration) gives you the benefit of almost real-time feedback so that you can update your user stories on the fly, etc.
Without rambling on any farther, my point is not too spend too much time looking for a specific unified theory. Read up about all the ideas, methods, and theories. Take the best parts from each, then crank the knob all the way up (if I may borrow that from XP =] ). Don't let anyone tell you there is a science to software development that is easy to reproduce, and that you are just a link in the overall chain. You practice and perform a craft. Enjoy it!
>How is the project going to get completed without anyone working on it?
.po, and .in files, and see what you get : GNU/Hello !
:)
Read my lips: E-VO-LU-TION
Example:
Start with "printf("Hello World\n");" and leave it in a warm, wet place for a few months, feed it with some
I have a strong belief that's what they did with Mozilla
-- don't discount flying pigs until you have good air defense
The basic issue is changing requirements. A contractor building high-rise apartments does not have to worry about the customer coming around when it's half built to look at it and say, "You know, I think I want a hospital instead." Programmers quite often have to deal with customers that are just about that confused -- they can't begin figuring out what they really want until they see what they asked for on the screen.
More than that, software vendors routinely write a program, release it, then add features so they can sell it again. It's as if the builder has finished the apartment building, and now they want a factory tacked onto the north side and a Wendy's onto the east. Next year, add a hospital wing to the west. Repeat once a year for 10 years and you get one hell of a mess, but how else would M$ keep a continuing revenue stream from the same OS and Office programs?
From the article:
Is fetchmail complex enough that it needs to be growing geometrically? I mean yeah, fetchmail does a lot, and I do know what "geometric" means. Still, I doubt the world of email is changing fast enough that you'd want to choose that as your example of out-of-control software maintenance.
[Insert obligatory ESR goading.]
What is necessary is a method for changing design gracefully. "Refactoring" is the best source I've seen that addresses this. Basically, you change methodically, and you test.
.NET, but wouldn't it be nice if you could get up-to-date architecture diagram syncronization with code directly from a source versioning tool (ie. cvs, SourceSafe)?
I was talking about this with a friend the other day. Wouldn't it be nice if a senior software 'architect' could maintain a unit-level view AND current code at the same time? That way his busy programmers could refactor all they wanted, as long as they didn't overstep their unit bounds but at the same time improve the product. The architect could look at the project at different levels of abstraction (units, subunits) to make sure the programmers aren't getting off track.
Probably the hardest thing about using the iterative or refactoring methodology is knowing what your architecture looks like at any given time. You design a great, flexible architecture for the first iteration, but after several rewrites you may not know where you are in terms of the big picture. Surely a tool that spits out UML-like diagrams of the current code would be very useful to spot architecture flaws introduced during the refactoring process. Effective use of design patterns may also help. Is it impossible?
I've seen some work done by Rational in terms of code generation with
----- rL