Slashdot Mirror
A Unified Theory of Software Evolution
jso888 writes "Salon has a nice article today on Meir Lehman's work on how software evolves and is developed. Lehman's investigation of the IBM OS/360 development process became the foundation for Brooks' Law: "Adding manpower to a late software project makes it later." He is hopeful that his work will make software development less of an art and more of an engineering science."
Software doesn't evolve by chance, folks, it is DESIGNED by its CREATORS.
/. is a site for SERIOUS INTELLECTUAL DISCUSSION.
Please check your crackpot theories and psuedo-science at the door.
Thank you.
dinner: it's what's for beer
This can be simplified: "Adding manpower to a software project makes it later."
There's rarely that many programmers needed for a given task anyway. You need a project leader and lots of monkeys to test it... very few projects should have more than 10 programmers (if any).
//TheToon
Perhaps we should call it Software Engineering.
Quick patent it before some else discovers its been in use for the last 30 years
While "Brook's law" might be a law, it's only useful in retrospect. Most software projects have no idea how far behind they really are. So basically, you can always add manpower, you're really only half way through anyways...
Mythical Man Month???
Where's the guy with the .sig "it takes nine months to bear a child, no matter how many women you assign to the task" when you need him?!?!?!?!?
First it's : "You'll have to forgive me," apologizes Lehman at one point, sifting through a pile of research papers on a nearby shelf. "Since I lost my secretary, I can't seem to find anything." . And in the last sentence of the article it says: ...a man in search of both fulfillment and a little revenge"
Poor, poor man; he'll never find it I'm afraid!
from www.dictionary.com
evolution
n.
A gradual process in which something changes into a different and usually more complex or better form. See Synonyms at development.
a. The process of developing.
b. Gradual development.
Seems like quite an appropriate term for 90% of the software projects I have ever worked on.
the site is slashdoted already... does anyone have a mirror?
Fabio - Sumare/Sao Paulo/Brazil/South America/Earth/Solar System/Milky Way/Universe
http://www.morroida.com.br
i refreshed it and it appeared, if anyone finds any problem this is the content:
April 8, 2002 | The office of Meir "Manny" Lehman is a cozy one. Located on the outer edge of the Imperial College of Technology campus in South Kensington, London, it offers room for all the basic amenities: a desk, two chairs, a Macintosh G4 and a telephone. Still, for a computer scientist nearing the end of a circuitous 50-year career, the coziness can be a bit confining.
"You'll have to forgive me," apologizes Lehman at one point, sifting through a pile of research papers on a nearby shelf. "Since I lost my secretary, I can't seem to find anything."
The pile, a collection of recently published papers investigating the topic of software evolution, a topic Lehman helped inaugurate back in the 1970s, is something of a taunting tribute. Written by professional colleagues at other universities, each paper cites Lehman's original 1969 IBM report documenting the evolutionary characteristics of the mainframe operating system, OS/360, or his later 1985 book "Program Evolution: Processes of Software Change," which expands the study to other programs. While the pile's growing size offers proof that Lehman and his ideas are finally catching on, it also documents the growing number of researchers with whom Lehman, a man with dwindling office space and even less in the way of support, must now compete.
"And to think," says Lehman, letting out a dry laugh. "When I first wrote about this topic, nobody took a blind bit of notice."
Software evolution, i.e. the process by which programs change shape, adapt to the marketplace and inherit characteristics from preexisting programs, has become a subject of serious academic study in recent years. Partial thanks for this goes to Lehman and other pioneering researchers. Major thanks, however, goes to the increasing strategic value of software itself. As large-scale programs such as Windows and Solaris expand well into the range of 30 to 50 million lines of code, successful project managers have learned to devote as much time to combing the tangles out of legacy code as to adding new code. Simply put, in a decade that saw the average PC microchip performance increase a hundredfold, software's inability to scale at even linear rates has gone from dirty little secret to industry-wide embarrassment.
"Software has not followed a curve like Moore's Law," says University of Michigan computer scientist John Holland, noting the struggles of most large-scale software programs during a 2000 conference on the future of technology. "In order to make progress here it is not simply a matter of brute force. It is a matter of getting some kind of relevant theory that tells us where to look."
For Lehman, the place to look is within the software development process itself, a system Lehman views as feedback-driven and biased toward increasing complexity. Figure out how to control the various feedback loops -- i.e. market demand, internal debugging and individual developer whim -- and you can stave off crippling over-complexity for longer periods of time. What's more, you might even get a sense of the underlying dynamics driving the system.
Lehman dates his first research on the topic of software evolution back to 1968. That was the year Lehman, then working as a researcher at IBM's Yorktown Heights facility, received an assignment to investigate IBM's internal software development process. Managers at rival Bell Labs had been crowing about per-developer productivity, and IBM managers, feeling competitive, wanted proof that IBM developers were generating just as many lines of code per man-year as their AT&T counterparts.
Lehman looked at the development of OS/360, IBM's flagship operating system at the time. Although the performance audit showed that IBM researchers were churning out code at a steady rate, Lehman found the level of debugging activity per individual software module to be decreasing at an equal rate; in other words, programmers were spending less and less time fixing problems in the code. Unless IBM programmers had suddenly figured out a way to write error-free code -- an unlikely assumption -- Lehman made a dire prediction: OS/360 was heading over a cliff. IBM, in stressing growth over source-code maintenance, would soon be in need of a successor operating system.
Although IBM executives largely ignored the report, Lehman's prediction was soon borne out. By 1971, developers had encountered complexity problems while attempting to install virtual memory into the operating system, problems which eventually forced the company to split the OS/360 code base into two, more easily manageable offshoots. The linear growth curve that seemed so steady in the 1960s suddenly looked like the trail of a test missile spiraling earthward.
Lehman's report would eventually earn a small measure of fame when University of North Carolina professor and former OS/360 project manager Frederick P. Brooks excoriated the IBM approach to software management in his 1975 book "The Mythical Man Month." Using Lehman's observations as a foundation for his own "Brooks Law" tenet -- "adding manpower to a late software project makes it later" -- Brooks argued that all software programs are ultimately doomed to succumb to their own internal inertia.
"Less and less effort is spent on fixing original design flaws; more and more is spent on fixing flaws introduced by earlier fixes," wrote Brooks. "As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress."
By 1975, Lehman, with the help of fellow researcher Laszlo Belady, was well on the way to formulating his own set of laws. A quarter century after their creation, the laws read like a mixture of old developer wisdom and common textbook physics. Take, for example, Lehman's "Second Law" of software evolution, a software reworking of the Second Law of Thermodynamics.
"The entropy of a system increases with time unless specific work is executed to maintain or reduce it."
Such statements put Lehman, who would leave IBM to take a professorship at Imperial College, into uncharted waters as a computer scientist. Halfway between the formalists, old-line academics who saw all programs as mathematical proofs in disguise, and the realists, professional programmers who saw software as a form of intellectual duct tape, Lehman would spend the '70s and '80s arguing for a hybrid point of view: Software development can be predictable if researchers were willing to approach it at a systems level.
"As I like to say, software evolution is the fruit fly of artificial systems evolution," Lehman says. "The things we learn here we can reapply to other studies: weapon systems evolution, growth of cities, that sort of thing."
That Lehman conspicuously leaves out biological systems is just one reason why his profile has slipped over the last decade. At a time when lay authors and fellow researchers feel comfortable invoking the name of Charles Darwin when discussing software technology, Lehman holds back. "The gap between biological evolution and artificial systems evolution is just too enormous to expect to link the two," he says.
Nevertheless, Lehman aspires to the same level of intellectual impact. While he was in retirement during the early 1990s, his early ideas jelled into one big idea: What if somebody were to formulate a central theory of software evolution akin to Darwin's theory of natural selection? In 1993, Lehman took an emeritus position at Imperial College and began work on the FEAST Hypothesis. Short for Feedback, Evolution and Software Technology, FEAST fine-tunes the definition of evolvable software programs, differentiating between "S-type" and "E-type": S-type or specification-based programs and algorithms being built to handle an immutable task, and "E-type" programs being built to handle evolving tasks. Focusing his theory on the larger realm of E-type programs, Lehman has since expanded his original three software laws to eight.
Included within the new set of laws are the Law of Continuing Growth ("The functional capability of E-type systems must be continually increased to maintain user satisfaction over the system lifetime") and the Law of Declining Quality ("The quality of E-type systems will appear to be declining unless they are rigorously adapted, as required, to take into account changes in the operational environment"). For added measure, Lehman has also thrown in the Principle of Software Uncertainty, which states, "The real world outcome of any E-type software execution is inherently uncertain with the precise area of uncertainty also unknowable."
While the new statements still read like glossed-over truisms, Lehman says the goal is to get the universal ideas on paper in the hopes that they might lead researchers to a deeper truth. After all, saying "objects fall down instead of up" was a truism until Sir Isaac Newton explained why.
"Whenever I talk, people start off with blank faces," Lehman admits. "They say, 'But you haven't told us anything we didn't already know.' To that I say, there's nothing to be ashamed of in coming up with the obvious, especially when nobody else is coming up with it."
For extra ammo, Lehman also has expanded the graphs and data from his original studies in the 1970s. Taken together, they show most large software programs growing at an inverse square rate -- think of your typical Moore's Law growth curve rotated 180 degrees -- before succumbing to over-complexity.
Whether the curves serve as anything more than a conversation-starter is still up for debate. Chris Landauer, a computer scientist at the Aerospace Corporation and a fellow guest speaker with Lehman at a February conference on software evolution at the University of Hertfordshire, was impressed by the Lehman pitch.
"He has real data from real projects, and they show real phenomena," Landauer says. "I've seen other sets of numbers, but these guys have something that might actually work."
At the same time, however, Landauer wonders if the explanation for similar growth trajectories across different systems isn't "sociological." In other words, do programmers, by nature, prefer to add new code rather than substitute or repair existing code? Landauer also worries about whether the use of any statistic in an environment as creative as software development leads to automatic red herrings. "I mean, how long does it take a person to come up with a good idea?" Landauer asks. "The answer is we just don't know."
Michael Godfrey, a University of Waterloo scientist, is equally hesitant but still finds the Lehman approach useful. In 2000, Godfrey and a fellow Waterloo researcher, Qiang Tu, released a study showing that several open-source software programs, including the Linux kernel and fetchmail, were growing at geometric rates, breaking the inverse squared barrier constraining most traditionally built programs. Although the discovery validated arguments within the software development community that large system development is best handled in an open-source manner, Godfrey says he is currently looking for ways to refine the quantitative approach to make it more meaningful.
"It's as if you're trying to talk about the architecture of a building by talking about the number of screws and two-by-fours used to build it," he says. "We don't have any idea of what measurement means in terms of software."
Godfrey cites the work of another Waterloo colleague, Rick Holt, as promising. Holt has come up with a browser tool for studying the degree of variation and relationship between separate offshoots of the original body of source code. Dubbed Beagle, the tool is named after the ship upon which Charles Darwin served as a naturalist from 1831 to 1836.
Like Landauer, Godfrey expresses concern that a full theory of software evolution might be too "fuzzy" for most engineering-minded programmers. Still, he credits Lehman for opening the software field to newer, more intriguing lines of inquiry. "It's the gestalt 'Aha' of his work that I find more interesting than the numbers," Godfrey says.
For Lehman, the lack of a scientific foundation to the software-engineering field is all the more reason to keep digging. Fellow researchers can quibble over the value of judging software in terms of total lines of code, but until they come up with better metrics or better theories to explain the data, software engineering will always be one down in the funding and credibility department. A former department head, Lehman recalls the budgetary battles and still chafes over the slights incurred. Now, as he sits in a cramped office, trying to recruit new corporate benefactors and a new research staff, he must deal once again with those who label software development a modern day form of alchemy -- i.e. all experiment but no predictable result.
"In software engineering there is no theory," says Lehman, echoing Holland. "It's all arm flapping and intuition. I believe that a theory of software evolution could eventually translate into a theory of software engineering. Either that or it will come very close. It will lay the foundation for a wider theory of software evolution."
When that day comes, Lehman says, software engineers will finally be able to muscle aside their civil, mechanical and electrical engineering counterparts and take a place at the grown-ups' table. As for getting bigger offices, well, he sees that as a function of showing the large-scale corporations that fund university research how to better control software feedback cycles so their programs stay healthier longer. Until then, the search for a theory has rendered Lehman less of a Darwin and more of an Ahab -- a man in search of both fulfillment and a little revenge.
Fabio - Sumare/Sao Paulo/Brazil/South America/Earth/Solar System/Milky Way/Universe
http://www.morroida.com.br
"When I first wrote about this topic, nobody took a blind bit of notice."
No, sir, I did and many collegues who were also interested in good timely work. We lent your books to each other with the notion "that's something you should read".
Great to hear that you are still alife and enjoying to give programmers and their managers something to look at and something worth to read and think about.
Youngsters, better pay respect to this old software camel with the hole in the sole of his shoe (and probably also in his all-too British pullover), or I DDOS your toilet!
"Unless IBM programmers had suddenly figured out a way to write error-free code -- an unlikely assumption -- Lehman made a dire prediction: OS/360 was heading over a cliff. IBM, in stressing growth over source-code maintenance, would soon be in need of a successor operating system."
Which means that commerical systems don't so much evolve as stub their growth paths out and switch direction or spawn new generations because embedded complexity has killed off the feasibility of maintaining it. In other words, all new releases are the cause of and ultimately an attempt to escape from, the chimera that is overly complex code. In commercial terms this should be astounding. We're paying to gronk up our own because we erroneously believe the NEXT version will be something radically new and elegant which of course it can't be.
New Version "x+1.y" is simply an ejection seat.
"Lehman's investigation of the IBM OS/360 development process became the foundation for Brooks' Law: 'Adding manpower to a late software project makes it later.'"
Yet since ancient times, those alive within the Tao of Programming have known this to be true.
I'm not attempting to flamebait here, just submitting an observation. It seems to me that many of the complexity issues can be overcome by designing better languages. I've never stopped scratching my head over the perseverance of old languages like C++ and FORTRAN. Sure, they are extremely useful in the hands of experienced folks, but they need to die. They were good solutions to problems decades ago, but so much has been learned since then and the constraints of sparse computer resources and CPU speed have moved a lot.
I have always seen programming as something magical but thats probably more because of lack of knowledge than because it really is magic. Maybe its time to straighten the myth behind big software projects and get a bit more grip on how it works and why. The intent is noble and given current state of software capabilities and performance in comparison to hardware its about time. Imagine having an os that had evolved like your hardware? ......
Think *SCREEEEAAAAMMMMM* woshing numbers like nothing you have EVER seen.
HTTP/1.1 400
a standard printed book value of time estimates for projects. the auto repair industry has standard estimates for certain repairs, why doesn't the software repair industry. i know they're worlds apart, but it sure would help out a little to be able to pull out a little book and say, well, you need a gui interface consisting of 15 screens to maintain 20MB of data, it's going to be 10,000 hours for developing, testing and documenting. if you want to cut the documentation, we can do that, but you're really slitting your throat there.
From the article:
Michael Godfrey, a University of Waterloo scientist, is equally hesitant but still finds the Lehman approach useful. In 2000, Godfrey and a fellow Waterloo researcher, Qiang Tu, released a study showing that several open-source software programs, including the Linux kernel and fetchmail, were growing at geometric rates, breaking the inverse squared barrier constraining most traditionally built programs. Although the discovery validated arguments within the software development community that large system development is best handled in an open-source manner, Godfrey says he is currently looking for ways to refine the quantitative approach to make it more meaningful.
It would have been interesting had they delved deeper into this finding. Yeah, I know, the true believers in open source all feel superior (we are, aren't we?), but exploring the reasons why it works would be interesting.
Is it the large-scale peer-review process? Is it that we occasionally rewrite parts (filesystems, VMM, etc)? Something else?
It is a Beastie Boys reference!
Good article; I think the description of the sociological basis of the "laws" is correct. My experience suggests that the slowest development paths are those that cross other people's areas.
i ant*)
(And yes, I know about XP's "All code is shared.")
As for the maintenance, it's my normal experience, but the prohects I've been involved in may be atypical. (*cough*Canadian*cough*telecommunications*cough*g
We spend a *lot* of time reworking old code to (a) fix obscure bugs, many of which are slow leaks shown up by weeks serving live traffic (b) adapt the code to support new releases of underlying hardware product and (c) adding new features to satisfy users.
Although the performance audit showed that IBM researchers were churning out code at a steady rate, Lehman found the level of debugging activity per individual software module to be decreasing at an equal rate; in other words, programmers were spending less and less time fixing problems in the code. Unless IBM programmers had suddenly figured out a way to write error-free code -- an unlikely assumption -- Lehman made a dire prediction: OS/360 was heading over a cliff. IBM, in stressing growth over source-code maintenance, would soon be in need of a successor operating system.
Except that the "[dire] need of a successor operating system" isn't so dire at all: the world's richest man didn't get where he got by writing code that didn't need to be replaced by a successor operating system, did he? The whole premise is to produce something that works now, and when it stops working later, you sell a later version. Heck, just a couple of months ago, Billy announced that 92.3% of the calendar year would focus on new code, leaving the rest for the old.
What's smarter, coding the Microsoft way, or coding a server that's been up since before Windows NT was released, without a patch in 7 years, handling half a megabit of data both upstream and down, every second of every day forever. Where's the revenue?
~r~
Note: the 92.3% figure might only be for the year 2002, with later years being still closer to 100%.
You didn't get it that he is simply doing what he wants to do, doing it good and trying to motivate others to do good reliable work also?
Mastery is when you know (and not only guess), that you can only search.
Illusion is when you think you got it.
From the article:
"In software engineering there is no theory,"
I don't buy that... at least not completely. I would say something more like, "In software engineering, theory is extremely underutilized."
I believe there are many instances of engineered software, but not necessarily high-profile stuff. A lot of DoD conscripted code may never the the civilian light of day, but there are procedures and documentation requirements that, flawed or not, enforce certain practices. Can we call that "theory"? Anyhow, defense suppliers can afford the extra development time, 'cause the government is forking over big bucks for the code to right.
For the mainstream (read desktop) apps, where all the money is, the time to market and feature pressures will continue to suppress even the best "unified theory" of software development.
Treating software as a system is an interesting and slightly frightening idea. I think that idea is valid and, once you're willing to give that concept a chance, you have to agree that entropy applies. If our "system" becomes more disordered over time as we add to it, we, as software designers, have to re-evaluate our focus. I see cases in my own experience where this happens. In the middle of a SQL to Oracle conversion right now, a mess of a project is becoming even more of a mess.
The theory is also a good case for templatizing and making very generic our software. If, over time, we only have to change our data (files), and not the software itself, this should help to control how disordered our software becomes.
The biggest flaw in the theory discussed comes when we consider real-world software design. Marketing types and customers don't care about software theory and they don't care if our software is becoming more disordered. Just try to tell your boss that you can't add a feature a particular customer wants because you don't want to contribute to the rate of entropy of the system...
C++ isn't that old. C, yes, but C++ is one of the newer languages.
Best Slashdot Co
Manny Lehman is credited with coining the expression "Software Engineering". About 1968, I think. See also the website of the company he founded Imperial Software Technology .
"Whenever I talk, people start off with blank faces," Lehman admits. "They say, 'But you haven't told us anything we didn't already know.'
:)
99% of the consultants do this for a living. Get used to it.
-- No sig today
10,000 people working on Windows XP?
No wonder they have so many problems! The should have a smaller team of say, 20 or 25 people
(ugh)
C++ isn't that old. C, yes, but C++ is one of the newer languages.
Yes... C++ extended the old, crusty C language and brought it right into the 1980's.
-CausticPuppy "Of all the people I know, you're certainly one of them." -Somebody I don't know
mark_lybarger asks:
This already exists. Estimates based on Function Points essentially give you this.
Ne mæg werig mod wyrde wiðstondan, ne se hreo hyge helpe gefremman.
I was interested in the fact that some researchers have only recently come to the conclusion that software is written by people.
It is questionable how useful purely statistical methods are in these situation.
One thing I would be interested in knowing is how staff turnover effects development. For matainable software to be possible a consistent approach must be maintained on adding new functionality, this usually requires deep understanding of alrge code base, and if your programmers keep changing, the newbies may not follow the rules.
Choose your allies carefully, it is highly unlikely you will be held accountable for the actions of your enemies
There's a lot of piss poor code out there because there are a lot of piss poor programmers out there -- people who should not be in this industry, people who took a couple of classes in VB and think that qualifies them for the title of "Programmer." And they can still bullshit their way past hiring managers with their shiny buzzwords.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
I worked on a system running OS/360, up to Release 23 or so, when IBM 'retired' it.
We installed new Releases about once every 6 months. IBM also had 'patches' available for about 19,000 known bugs.
These patches were not incorporated into the latest release because each of them, if installed, broke some other aspect of the OS.
We, and every other site, only installed those patches needed to work around problems that the particular site encountered. And you always hoped that today's patch would not break something else that your users needed.
- The functional capability of the OS too, since new hardware keeps coming out
and the Law of Declining Quality ("The quality of E-type systems will appear to be declining unless they are rigorously adapted, as required, to take into account changes in the operational environment").
Exactly what is happening to windows? And why Linux is so successful -> Open Source like fetchmail et al being more linear in their development, all users get a stab at getting the environment right.
But users who aren't prepared to do any work to make things better in their environment for their PC are always going to lose. But it's the same as those people who make their desks tidy and optimise them for work, and those that don't. The difference on your virtual desktop is that you can't easily hope someone else will tidy it for you...:)
Conversion Rate Optimisation French / English consultant
it just goes to show that 99% of the work in creating software is in the design.
you have to try to map out not only what you will need but what you might need in the future.
yes, it's a near impossible task but it's the only way to avoid automatically commiting yourself to an endless cycle of patches and hacks.
the good part is, if you can plan the project well enough then the actual coding becomes nearly trivial.
the problem arises when the boss says 'i don't care about scalability or flexibility, i just want code now' and i have to try to explaining that i'm trying to save his ass 8 months down the line when clients (and not to mention, the boss himself) bombard us with feature requests, etc.
OS/360 was actually heading over a cliff. The various pieces of software did not work when they were put together. The OS was delivered years late and massively over budget. Many IBM 360's (costing six figures back when $1 was worth something) were delivered and then spent years simply running emulators for the old machines they replaced, because the native software wasn't ready.
Yes, there were lots of things they could have done -- like define a subset of the original committee-designed bloated specification, get that working, then start adding features. But the manager (Fred Brooks) didn't know that, yet, and didn't even know the project was in trouble until it was impossible to deliver anything at all on deadline. Afterwards, he wrote a book, The Mythical Man-Month, which has become a standard text for large-project management. But he learned how by doing it wrong, more massively than anyone ever had before...
So it is accurate to say that C++ has only been standardized recently. But unless you're comparing C++ to Fortran/Simula/Algol, it is just wrong to call it "new".
The Daily Build
FORTRAN is definitely an old language but old does not necessarily mean bad. It still has place in many scientific and engineering circles because it has been thoroughly debugged and tested. Would your trust your local nuclear power plant software if it was Perl? Python? Java? C++? I wouldn't. There is a reason that most code for calculating nuclear plant fuel assembly concentration and positon is written in FORTRAN--and it's not because of a love of old, perceivedly sluggish, languages.
Besides which, just because we have more hardware to throw at a problem does not mean that that is the optimal approach. That very point of view is the reason that computer hardware gets ever faster but the operating systems and software don't. It's a symptom of the Microsoft school of coding: throw more hardware at a problem and that will fix it. 640k^h^h^h^h256M of memory is all you'll ever need.
I don't think either point is valid. If you want to argue based on security, memory issues, etc., fine. Age of a language and the fact that it runs efficiently are really bad reasons to attack it, however.
What is your Slash Rating?
Creating common APIs allows seperate development projects to proceed at their own pace. You don't need OO for this, but it helps.
I think one of the reasons that Linux has been so successful is because Linus decided long ago to take a modular approach to designing his monolithic kernel.
-josh
I've used plenty of late-version software that is arguably worse than the previous version...I believe that's called de-evolution :)
What is your Slash Rating?
To a Lisp hacker, XML is S-expressions in drag.
Back in the early 1980s I headed up a small team that developed 'industrial strength' applications.
Our firm licenced this software to major manufacturing firms with a Money Back Guarantee. As in, "If you are not satisfied, for any reason, we will either fix the problem or give you back your money. Your choice." We were never asked for a refund.
It was semi-open source. You could have the source any time you wanted, but asking for the source voided your warranty, since problems in your data might have been caused by your own temporary code changes.
Funny thing. I've had that on my resume for many years, but no prospective employer has ever asked how I did it.
No one has hired me specifically to help them produce similar quality code. Much of the time their reaction to my resume is, 'but you don't know c++' (or their other favorite). I know enough about c++ to know that I want to stay away from that second generation language for all but the most specialized situations.
I have also been told, on numerous occasions, that I'm not qualified to lead a particular project because I lack experience mannaging the large team that will be needed. I've never gained that experience because I've never needed a large team to accomplish anything.
As an MBA, as well as being an application designer & a coder, I know that large teams do have a place -- mostly where you have a blank cheque and are earning a percentage of the total billing. (:-)
A quick warning... I consider myself a relative newborn in the world of software development. I present these opinions under the consideration that my opinions can change at any moment. =]
A lot of the dire predictions of software atrophy and such are a result of applying the wrong methodology to a project. Yes there are uses for Software engineering, but I think this approach is overkill for even large scale projects. Check out Software Craftsmanship: The New Imperative for a different perspective. A perspective I think is in need of serious consideration. The gist is returning to the days of master craftsman and apprenticeships. This focuses a bit more on the learning aspect than actual development methodologies, but you can always go to The Pragmatic Programmer to fill in that gap.
"As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress."
This is where "refactoring" (see Fowler's Refactoring) really shines. I find it difficult to believe that refining the software base is not progress. An initial revision where the code functions by its contract (if your into designing by contract), then you refactor the body of the function/method for speed / elegance. Then you can run your unit tests on the function / method to test that the refactoring session did not break any of the design contracts (whew).
I think they may be trying to restate the broken window theory (see Pragmatic Programmer), were a broken window (or bug) in a building (or system) leads to delapidation elsewhere in the building (or system).
And then there are the agile methods, including XP. I think these answer a lot of the limitations and issues with Software Engineering practices. Interacting with clients (having a client there during each iteration) gives you the benefit of almost real-time feedback so that you can update your user stories on the fly, etc.
Without rambling on any farther, my point is not too spend too much time looking for a specific unified theory. Read up about all the ideas, methods, and theories. Take the best parts from each, then crank the knob all the way up (if I may borrow that from XP =] ). Don't let anyone tell you there is a science to software development that is easy to reproduce, and that you are just a link in the overall chain. You practice and perform a craft. Enjoy it!
Take, for example, Lehman's "Second Law" of software evolution, a software reworking of the Second Law of Thermodynamics.
"The entropy of a system increases with time unless specific work is executed to maintain or reduce it."
As evidenced by the back of my Subaru.
Samsung took back my unlocked bootloader because Google wants me to rent movies. They're both evil.
I can't speak for everyone, nor would I try to, but I find joining the phrases "quote scripture" and "tell the truth about homosexuals, minorities, etc" a bit queer.
You post anonymously. Take your cue from your heroes of scripture. Did they hide in corners, whispering about how the Romans were being mean to them? No, they spoke openly, risking death. I hardly think you'd risk anything so severe for posting your Christian thoughts on a "news for nerds" web site.
Its simply true; there is no other language that even comes close to filling all the roles of C++. Most of the languages people advocate for taking a certain niche from C++ are implemented in C++.
Its a very difficult language to learn, and hard to use properly. It has lots of syntax, and many idiosyncrasies. Yet it yields you control of the machine in the manner of C, adding in alot of the niceties of high level languages for those who know how to use them.
You might argue that its less error prone for certian programmers to use a more specialized and high level language for certain tasks. You might make a good case that C++ should not be someones first learned language (I say learn assembly, then C, then C++, then some high level lang).
What you cannot say is that C++ should be ditched. It is filling a vast role in real-world programming, where nothing else can compete.
>> And that's not even getting into the fact that there are still (a lot of) asm programmers out there
What, they're a dying breed?
Well, that is one of the core concepts of extreme programming. Small teams are a fundamental part of the methodology.
--the verb
We had a case were a system no longer proved ameniable to feature addition or continual improvement to match the changing operational and customer requirements. In the end the benefits of refactoring the codebase to match the changing production requirements were more costly than to rewrite the system using more modern libraries, methodologies and frameworks. It got rewritten and the old system phased out.
It wasnt a case of "fixing" inherently broken software, it worked perfectly well, just the operational flow it supported changed due to new customers and more efficient management procedures.
Incidentally we have found with each major rewrite of that system ( there has been two ) there has been an immediate growth spurt in customers. I am not sure if it is because it looks like something new, or that the software better matches the operational requirements or because of increasing feature addition. Either way the last two rewrites have been paid for almost immediately by the addition customers the new software has brought in.
mocom--
The recurring theme that its the programmers fault, not the language, is entirely tired and completely wrong. You have to maximize the productivity of the average programmer. Sure you can snidely conclude that they are stupid and just not man enough for C++, but that isn't going to get your product out the door any faster or reduce the error rate.
...is that it points to an old Salon piece about Larry Wall and the creation of Perl.
Thanks for helping me to improve my English.
A scale ? It's only worth anything, if you have something to put into. What this is one brings with him is often allready fixed - on which side of the scale you put is your decision.
Same holds true for friend, foe or colleague.
access to machine registers and memory
architecture specific machine instructions
transfer of execution to an arbitrary address
coerce object refs to addresses and back
invoke OS services
This doesn't mean that you can't write GC in Java! IBM implemented a JVM and GC system entirely in Java, called Jalapeno. To do this, they created a Java class called "Magic" that had empty methods for these services which any Java compiler could build. Then, the internal Jalapeno VM compiler would recognize calls to the Magic class, verify that what they are compiling is a valid part of the JVM and inline appropriate machine code where these calls occur.
Now, all GC systems can be written in reference to this Magic class and porting the VM is simply a matter of generating appropriate machine code for these half-dozen methods. And you get all the security of Java's automatic memory management model!
Check the ACM's OOPSLA Conference Proceedings, 1999, Implementing Jalapeno in Java or www.research.ibm.com/jalapeno for the paper.
I'll take a shot...
Think Panama Canal, Hoover Dam, the Great Pyramids, walking on the Moon, the tower of Babel... Throughout history, humans in large groups who come together, find a way to work toward a common goal, and actually work without a what's-in-it-for-me attitude accomplish wonders.
OK, yes, many of the accomplishments I mentioned had paid workers, but in most cases there was a cause (racing to the Moon out of patriotism, avoiding a death sentence from Pharaoh... etc.)
In all of these efforts, the sense of being part of something greater was there.
Just as termites can rally around a queen and build giant mounds complete with air conditioning (really, Google-search it), humans in large numbers have altered the Earth's geography and climate. It's only natural that humans in large numbers working toward a common goal (be it a sense of belonging, or just to beat MS) could successfully put together something as complex and large-scale as an OS.
... Not that I'd categorize Linux as a wonder of the world. :)
ESR took a better shot at explaining it in his book, The Cathedral and the Bazaar.
/
.
/
.
The views expressed in this post are my own and aren't connected to the views and opinions of my employers.
Why does making software engineering more of an engineering science necessarily make it less of an art? Can't it be both? Shouldn't it?
Plus the fact that the author obviously would not intend such a similarity as it would be destructive to his/her/it's very viewpoint.
The author intended exactly this similarity to be funny. Someone mocked your mistake to bring this to your attention (personally, I'd have written, "Gee, you got the punchline, go find the clue'). You then repeated your mistake by not recognizing this and then explaining your obvious error still holding the misconception that you had made a clever discovery of found humor.
I'm spelling this out because you apparently don't have the faintest comprehension of wit.
Please shut up, and improve the signal:noise ratio on slashdot.
... and he sucked at Software Engineering :)
:)
In fact he was one of our worst lecturers. He lectured a little of software engineering and a lot on general management. This is after Imperial had sunk squillions into Manny's failed software company, so go figure
The article is actually quite good so maybe he has improved with age, but I still don't have too much respect for him.
In the software industry, it is not uncommon for the developers to be far more intelligent and knowledgable than their managers. This creates more than resentment, there is a fundamental conflict. The manager is setting the deadlines, and yet the developer is the one who knows that all deadlines are bullshit.
This is the reason most software sucks.
humans? oh no, I thought it was all written by an Intelligent Designer.....
I hardly think you'd risk anything so severe for posting your Christian thoughts on a "news for nerds" web site.
You're new here, aren't you? Christians are often savagely ridiculed on Slashdot and are often exposed to some of the most atrocious blasphemy possible. I have had my e-mail address given to a pornography address list, others that I know have had threatening phone calls made to their home and in one case a threat made against a whole family. There is a very real risk and there is a very real reality, and that is that it is not safe to be a man of faith in a society of decadence.
Spoken like a seller of software. And this is why commercial software is junk.
But the people who use software want it to keep on working. And 95% of all programers don't work for the software vendors, they work for software users. And we'd like to be able to take pride in our work, and not create junk.
Thus is the 'traditional' software vendor business model doomed. They sell junk, we don't want junk, so we'll build our own and exchange it. Vendor of junk goes out of business, good riddence.
From the article:
Is fetchmail complex enough that it needs to be growing geometrically? I mean yeah, fetchmail does a lot, and I do know what "geometric" means. Still, I doubt the world of email is changing fast enough that you'd want to choose that as your example of out-of-control software maintenance.
[Insert obligatory ESR goading.]
Somehow the last two words form a concept that at least from observation, doesn't really exist.
Then again I'm only 2 years out in the "commercial world"...what do I know?
The study by Mr. Godfrey and Mr. Tu can be found at http://plg.uwaterloo.ca/~migod/papers/iwpse01.pdf . (4 pages in a PDF file).
And yet you hide behind anonymity. His point was not that there is no danger. His point was that if you want to make a difference, take a stand.
Surely the martyrs who risked death had quite a bit more to fear than porn spam and phone calls.
I program in Java and I think it's a great language for most things. I also know C++ and I think it's usually very ugly in comparison, but there are a lot of things that C and C++ are just better for. Drivers and Operating systems for example. Java can do them, but you can tell Java wasn't designed to fiddle with the machine directly. It feels clunky. So my question is, what would you suggest for the machine level stuff? I think that if you need to do something easy and cross-platform, you should use java, because it's easy to make stuff pretty, but when you need C++, it comes in awfully handy.
Since you seem to have so many ideas on what a language should do (as long as it isn't English), perhaps you could design the ultimate language?
What is your Slash Rating?
You're absolutely right about this. I'm another semi-old-timer. In the early 1980's, I was on the team (six people, all with developer background) to write a bisynchronous communications package (HASP station emulator). We had a standing offer--anybody who could find a bug would get a free dinner at any restaurant. We only had to pay off once.
Nobody seems to care about doing this anymore, or maybe they never did in the first place, and we were all just naive.
- The Beagle tool is the work of my (recently graduated) M.Math student, Qiang Tu, done under my supervision. We reused the PBS landscape viewer as part of it (which is the work of my colleague, Prof Ric Holt; hence the confusion). More info about Beagle can be found in this paper.
- The study of Linux's growth and evolution is best documented in this paper, not the much shorter paper given in the salon.com article and elsewhere in these postings.
- The systems my group has looked at in some detail include the Linux kernel, GCC, and VIM. fetchmail was a quick one and the results were not terribly interesting.
- I am not an open source evangelist; I am a researcher
:-) I did not say that "large system development is best handled in an open-source manner". I said something more along the lines that open source development seems to work very well for certain classes of systems, especially large, service-based, infrastructure-ish systems like operating systems and compilers. I might have said that I thought open source development might be the best way to engineer these kinds of systems. That would be a personal opinion, tho.
Michael Godfrey PhD, Assistant ProfessorNortel Networks Jr Chair, Telecommunications Software Engineering
Univ of Waterloo, Dept of Computer Science
email: migod@uwaterloo.ca
URL: http://www.uwaterloo.ca/~migod
Congratulations (belatedly) on your team's accomplishment. HASP was no small product. I once (early 1970s) had a copy of a foot-thick HASP technical document.
See this link for the generally accepted historical view.
Where I work, it has been a commonly held belief that all software evolves until such time as it can send and receive email. If it doesn't do this, it isn't complete. :)
Jason PollockThis is my favourite quote - it *needs* to be said more than it should. I figure if I beat enough people with it and eventually someone will say it to me (when I turn into a manager perhaps).
"There's a lot of piss poor code out there because there are a lot of piss poor programmers out there -- people who should not be in this industry, people who took a couple of classes in VB and think that qualifies them for the title of "Programmer." "
Isn't there. I was hiring for a C++/Win32 coder a couple of months ago and I has some *real* bad candidates. IN the end my opening question for C++ was "Is your entire C++ experience solely with MFC". If the answer is yes then interview ends...
javac, the standard JDK compiler, is actually a Java program. Also, kopi.
Here is Mittermeyer's Law-
.01 <script kiddie> to 1 <average competent programmer> to 10 <certified productive genius>),
F x (H-Q)
___________________ = man-hours
(P1+P2+P3, etc.)/nP
where F is number of Features,
H is Hobbling (range from 1 to 100),
Q is Quality Assurance (range from 0 to 100),
P1 is skill level of programmer on particular problem (skill level being from a range of
and nP is number of Programmers.
On (H-Q), use even if a negative number (excessive quality assurance can be as hobbling as normal bureaucracy), and treat 0 as 1.
This formula takes into account the complexity of the work the package is performing, the skill of the programmer(s) involved, the fog of bad user/organizational ineptitude, and the amount of QA work being done.
Software should not be described in terms of lines of code or database sizes, but rather by 'feature'. A feature could be thought of most easily as the list of salable points Microsoft and others use to sell their products. A feature can also be an internal function not recognizable by the user, but which aids development, error tracking, security or maintenance.
To use this formula in the context of the article, simply add a feature change to the feature number. Static applications will be a slight drain, but constantly changing applications will require man-hours, genius programmers and/or intensive QA work and a minimum of hobbling to reduce man-hours. Even with a fantasy level of positive factors, the complexity level will require features to be taken out, or a process to be started from scratch.
This formula also covers why Open Source has worked so well. OS brings together programmers to work on tasks they are well suited for on modules rather then one monster product, thus avoiding insane complexity levels. The peer review aspect quickly weeds out inept behavior, thus keeping the programmer skill level relatively high, whereas a corporation may not be able to recognize incompetence quickly.
Obviously plugging numbers into this formula is a highly subjective business, but then again so is Lehman's approach. Try this on for size- I would be very interested in hearing about real-world results.
________________________________________ History Must Not Fall Into The Wrong Hands ___________________________________
nothing evolutes.
evolution is a man made concept/thought.
the only thing that exists is the development of the human mind.
nothing evolutes into nothing.
Who was it who said, "Those who are ignorant of history are doomed to repeat it."
(* Just try to tell your boss that you can't add a feature a particular customer wants because you don't want to contribute to the rate of entropy of the system... *)
That is one of the biggest problems: there is almost no factoring in of "complexity costs" in the decision process. Although it might take the same amount of time to add feature A and feature B, one may significantly complicate the system in the long wrong while the other may not.
As long as the complexity costs are never factored into decisions, it *will* grow messy over time.
Messy software is a management issue more than a technical one. If you (as an organization) spend complexity like a crack addict, you get what is coming to ya.
Table-ized A.I.
I don't know, to listen to some religious nuts you'd think porn was the greatest threat there is, worse than being stretched on the rack or fed to the lions.
Open source software OTOH is built by widely separated people with narrow bandwidth links between each other and only a shared vision of the Right Thing to guide them. The result, as predicted by Conway's law, tends to be highly modular architectures focussed around a few core protocols or APIs that capture the vision.
Modular systems are inherently more flexible and reusable than monolithic systems because they exhibit low coupling between the modules. In contrast the monolithic software is more likely to have high coupling between modules, even though they are supposedly independent.
(There is also a related concept of "cohesion", which is the extent to which the features of each module hang together as conceptual wholes. I suspect that OSS will show higher cohesion than closed source software)
It would be interesting to get some statistics to test this theory. Does anyone know of any good software for measuring coupling in C code? I'd like to run some commercial and OSS software through it and see what it says.
Paul.
You are lost in a twisty maze of little standards, all different.
For several years, this subject has been accepted as part of Computer Science.
However, this field is not to be taken too seriously for its purpose is to overburden the programmer rather than making software better. We must understand that while "Programming Languages", "Compilers", "Verification", "Functional Programming", etc. *is* Computer Science, "Software Engineering" hardly is a science of any sort.
"Software Engineering" is in my understanding an icon of bigotry and inability to accomplish a good design, contrary to its popular advertisement as a means to achieve engineering goals. "Apply our methodology, and you can get 1000 monkeys to write your next OS!!"
If you have good scientists and programmers who know how to design software and their toolbox, you will be set. Don't waste your time with this bulls***.
--exa--
I am trying to coordinate a small team generating version 3.6 of their product. (I came in as part of 3.5). I shift all requests for new features to my 4.0 list, bump 4.0 stuff to 4.1, ask the developers for their best deadline estimates on their 3.6 components, and add my SWAG to shift their own estimates further out. Then you want to blame me when the deadlines are all bullshit. You are right, I must be stupid for believing them.
Given:
X = amount of time 1 programmer of X skill to complete the project.
Y = the required time for the project to be finished, provided that Y does not go below the minimal possible time for the project.
P = the number of programmers needed on the team to start the project, then:
P = (X/Y)^2
2. As the number of transitors per area quadruples, which is roughly equal to the speed of the entire computer system, which happens about every 3 to 4 years, the amount of processing power that a currently maintained software package will be required will at least double.
You're new hers, aren't you?
... often exposed to some of the most atrocious blasphemy possible
...
No
Jesus fucking Christ, that's awful!
I have had my e-mail address given to a pornography address list
Sorry!
Now, about that "truth" you can tell about minorities and homosexuals. Tell me, please.