Slashdot Mirror
We Really Don't Know Jack About Maintenance
davecb writes "The ACM has been kind enough to print Paul Stachour's and my 'jack' article about Software Maintenance. Paul first pointed out back in 1984 that we and our managers were being foolish — when we were still running Unix V7 — and if anything it's been getting worse. Turns out maintenance has been a 'solved problem in computer science' since at least then, and we're just beginning to rediscover it."
And that means the validation people run through their test book and create maybe 1000 or 2000 bug reports. These bug reports bounce around through the development teams acquiring cruft along the way. When the bug eventually stops bouncing somebody might have a go at fixing it. So they change something and if they can't see the bug anymore they go cvs commit right then and there. At the same time the other 1999 bugs are bouncing around, looking for a home.
At the end of the (as we call it) bug fixing process the original software doesn't exist in its original form. It appears to pass most of the test sbut you would not be advised to lean against it or anything because something else might break.
http://michaelsmith.id.au
Doesn't modular programming solve this problem? If you design the core software correctly from the get-go to be truly modular, then any changes in the future are a matter of swapping in or out desired modules, and snapping on new ones?
Maybe I'm too naive, but is there REALLY that much spaghetti code these days?
"Software maintenance" has absolutely nothing to do with computer science. I wish people would stop calling business programming computer science. Computer science work gets done at universities and research institutions, not at Initech.
Taking code and cutting its size by half, fixing up all the screwed-up inconsistent formatting, while adding functionality and reducing bug counts, is a pleasure.
It's all in the mindset. It's only boring if you limit yourself to the boring parts.
No, I'm serious. It's the latest Trend (tm).
Cut the IT production support budget, cut the IT staff, move the functions that used to be IT Prod support to business and let their budget handle it.
If you can't see the humour in seeing a business person make direct live updates to a database table in Production (IT doesn't have Production access.. but Business does .. [go figure]) then you probably can't see that SEV1 sailing in from over the horizon to make your day just that little bit more special.
As for business editing files directly in Production because the cost of having IT do it (process - backup file, edit file, copy file to prod with due authorisation, verify the change) can just be avoided.
After all, we don't really need to pay for Production Support and System Maintenance and Documentation. The system works without these things, doesn't it?
What could possibly go wrong?
You have a sick, twisted mind. Please subscribe me to your newsletter.
But I really love maintenance. Sure, the job comes with tremendous amounts of stress but it really entertaining. In CCNA, there's nothing better than subnetting a large network :)
"The difference between genius and stupidity is that genius has it's limits" - Albert Einstein
So they change something and if they can't see the bug anymore they go cvs commit right then and there.
Yes. Bug trackers should have statuses like "Developer in denial" for situations like that. (Mozilla's bug tracker has a "WORKSFORME" status which is used far too much.)
"Software maintenance" has absolutely nothing to do with computer science.
Actually, it does.
I have a real Computer Science degree, so I know what computer science is about. And while a lot of corporate programming is more drudgery and form assembly than anything, there are a lot of applications of computer science in the real world - from scalability issues in large systems, to proper use of encryption.
Furthermore the supposedly boring area of "Software maintenance" has a ton of research potential focused around the optimal path to producing correct code. Do code review help? How to team dynamics factor in to code quality? Does Test First really improve code? What even defines code quality? There are programs within companies that experiment with different methods to improve code output, and those experiments are even more valid than ones performed at research institutions since they work on a real-world code base solving a real problem using real people. In fact I would go so far as to say and research being conducted outside of a company on aspect of code quality improvement is pretty much worthless, which is why it's important for researchers to partner with real companies for some studies.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
A lot of admins are pretty wary of throwing the latest and greatest on their boxes for the simple reason that it may, or shall I say, will break things. Its no fun to throw a service pack on to find it has nuked your installation, or upgrading your mail server to find out that your configuration isn't global to the new version. Now that sort of thing happens. I'm just saying there are all kinds of little issues (or huge ones) that can arrive. At least with older software you already know the faults and are prepared to work around them. Security has become the number 1 reason that software is now a continuous version stream with patches rolling out every 6 minutes. The UNIX-like OSes of the world have always handled this concept more gracefully, but when you look at windows that wants to reboot 4 times a week for updates it seems much more like an afterthought. Even linux can handle most updates without a reboot or hiccup. You don't even have to know that they occur of you like. If they can figure out how to swap the kernel live, we might start to see some really insane up times. I don't understand his argument. Most server type operating systems have few issues with updating themselves on the fly. There are a lot of insecure, unpatched boxes out there, but that is more bad administration than anything. Is there something that I am missing here or is he talking about ease of migration with data and new versions of software and keeping types universal so that they will interface with different versions? THAT is a problem. Like, for instance, I've been having some fun compiling old versions of stuff on Ubuntu and trying to figure out what library versions it was compiled against. The fat binary blob is not such a bad idea when you have terrabytes of space... :)
zosxavius photography
FreeBSD 1 and FreeBSD 2 had slightly different semantics for some system calls, but FreeBSD 2 changed the system call numbers, so it was possible to modify the FreeBSD 1.1.5.1 kernel to run the binary Netscape for FreeBSD 2.x by implementing the new API for one call in the old kernel. Alas, I can't find the patch now, which is embarrassing because I was the one hosting it... about 15 years ago.
Taking code and cutting its size by half, fixing up all the screwed-up inconsistent formatting, while adding functionality and reducing bug counts, is a pleasure.
Yes it is.
But that is not maintenance, as practiced by any rational company. That is development or (more specifically) optimization.
Maintenance is about solving a problem code is having, with the absolute smallest number of changes possible. Even new features can fall under this heading when software is in true "maintenance mode" in order to avoid a lot of excess testing.
I actually don't mind it, as it is a different sort of challenge to take a code base you know little about and introduce a working change with as little code as possible. But it's not as glamorous as fresh, raw coding.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
That is terrible.
On the other hand, thanks! I'm going to suggest this for our system.. perhaps condense it to 'DiD" and wait to see how long it takes for them to notice..
You have a sick, twisted mind. Please subscribe me to your newsletter.
Could a six sigma program help here? A systematic and structured approach to problem solving is needed. Whenever someone fixes a bug that creates a new bug, then it's a waste of everyone time and effort.
When you start claiming that the ability to add 1 to "1" and get 2 instead of a compile-time or even runtime type error makes software more maintainable (or is even useful in any way), I stop listening to your advice.
Your ideas are intriguing to me and I wish to subscribe to your newsletter.
Given a working upgrade, rolling it out while keeping things running is 'just' a matter of careful preparation and attention to detail. Being able to understand the existing system so you can get to a working upgrade is often the hard part, and while modular programming might be part of the solution, doing it right is not so easy, judging by the frequency with which it is done poorly (the phrase 'modular programming' itself belongs with 'buy low, sell high' in terms of the completeness with which it specifies a solution to a problem.)
Decades ago, companies which developed technology were...technology companies. With real engineers, and highly technically skilled management. Today, companies with business-oriented management and zero technology background own and develop systems. They often do it poorly, with insufficiently empowered engineering teams, and insufficiently skilled engineers.
So today we've got a lot of Java and .Net shops filled with junior-level programmers and no disciplined, experienced systems engineers. Is it a surprise that when MS brought programming to the masses that the masses failed to learn engineering?
Doesn't modular programming solve this problem?
That is one of those answers that sound great in theory but in practice suffer from all the same problems from which all the other answers suffer.
Too much upfront cost is a contract-killer, so compromises in designing for scalability must be made.
Furthermore, today you have ideas about how the software will be used, and tomorrow those ideas will be changed, and you will discover that the dividing lines you have set up between your modules are not optimally efficient. This will result in simple-seeming enhancements being prohibitively expensive....unless you start crossing your module boundaries directly. The pressure to do this is too intense to resist, and you wind up violating the modularity of your design in the name of getting it out the door in time.
Sooner or later your system will get complaints that it is too slow, and you will need to open further holes through your module boundaries in order to optimize performance within acceptable delivery deadlines. You will warn that these are quick fixes which must be refactored properly after their release, but the opportunity to do this never comes.
As such changes form layers around your ever-growing onion, the core modules will be come too precious to change. Every little tweak you make to a core module will conflict with assumptions made by other modules, and will cause surprising bugs that are hard to detect in QA testing (since test case count grows exponentially with each new feature). This will result in more demand to fix problems without adjusting the core modules, or by making minimal adjustments to them, which will wind up forcing you to further compromise the modularity of your design.
As the complexity of your system increases, the cost of each new feature also increases, which will displease management and prompt them to say "we used to be able to do this sort of thing in a few hours...now it takes weeks!" They will continue to make unreasonable demands of their senior staff until they push said staff right out the door, leaving the ongoing maintenance of this (now very un-)modular system to the junior level programmers who never shared in the original vision of the system, and hence have no sense for where the module boundaries should be, and just fold to managerial pressure to hack in changes as quickly as possible.
Eventually the slow turnaround times and general bugginess of the system will drive clients to start looking for alternatives. Some brand-new ones will be available (some of which written by the disgruntled senior staff, in fact). Thus begins the end of the product, and of the company if this was their flagship.
Yes I wholeheartedly agree. We had a new employee who did just that -- refactored a pile of mission-critical code, added unit tests, cut unused code, added data constraints, and made it more maintainable. At the end of the day that person was deemed to be unproductive and fired, and all the code thrown to the wind. This is what business types think of refactoring.
and the sentiment is no less true.
My observation is that we shaven monkeys that make up the human race are fundamentally incapable of maintenance in any sense. Rather than maintain something in as-new functional condition (maintenance) so it does not fail, we choose to either fix it (fixenance) or replace it (buyenance) when it does.
A factory that makes plastic widgets is just as likely to make these same mistakes in relation to their machinery.
Just my $0.02
err!
jak.
In the early 70's I had the pleasure of working with both Prof. Tony Brooker and I.R, MacCallum, both of whom had made major contributions to the Atlas timesharing system, Brooker had designed and MacCallum implemented most of the Compiler-Compiler, a meta system that included a parser generator and an MDL with interpreter to walk the parse tree and generate code. I bring these guys up to make a simple point, both were skilled technologists in their own areas, but as many University academics, they were narrow; an could not understand the need for maintenance, incomprehensible looks, the software will ROT ... etc etc.
They just could NOT get their minds around changing external circumstances, working in tiny groups, version control, V2... was easy as was getting their all 3 systems updated. Pre/Post/Co - requisites, SCM, package management, bug-tracking were all to come as the world started to really use this stuff and they also had no communication disconnect, because there were less than 1000 and they all knew each other,
Yes its a HUGE problem, but unfortunately does need real understanding!
Hey, look, no offense, but this kind of just seems a bit like self promotion to me?
Communications of The ACM publishes lots of awesome articles quite often, and those of us interested often subscribe. I guess I just don't see how the thoughts of two software professionals on software maintenance really needs to be a front page slashdot article.
*watches as he is modded troll*
I thought WORKSFORME is Mozillaese for DeveloperInDenial?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
In this age of 20yo CEOs and single-quarter companies it's hardly suprising that most software is no better than a rigged demo.
Just make it shiny enough for someone to buy the company and then let their support staff of MS trained monkeys deal with it
Then we have the "artists" (in both the software and hardware field) who have survived for twenty years without "all that sh1t". Course, like the CEO, they've gone on to their next challenge long before the chickens come home to roost. And it's not their fault everyone else is incompetent, is it?
I continue to be amazed, on a weekly basis, by the complete lack of experience shown by the actions and products of very large companies.
Oh, I reject the claim that
The author has obviously never maintained hardware: it has bugs, patches, upgrades just like any other part of your system.
-- Butlerian Jihad NOW!
This guy seems a little *too* optimistic about his solution. It's a great solution, and if you want to see a good example of its use, check out the K42 operating system. It solves a bunch of the performance problems a naive implementation as the article describes would have.
However, it's not a free lunch. Runtime marshalling and switching between interfaces? Does the author have any idea how hard that is? One tiny mistake and your entire database is full of junk data because you forgot that bit-X means Y, or your regex P had a typo. if you're going to adopt this "Continuous Change" model, you need to do an awful lot of validation testing on all the possible inputs from all the different API versions, and make absolutely sure you get the internal state you intended. Another way to say this, a hidden feature of the discrete approach is that you know the state of your system at all points in time. You're not running a franken-process.
BTW, I'm not trying to dump on the idea, I reckon it's cool, just the article doesn't really represent it accurately.
Comment removed based on user account deletion
Long, wordy, buzz-word heavy article with a little bit of interesting content buried deep inside. I wish I hadn't bothered to read it.
In case you haven't, but are thinking you might: you can run machines that are never down, even when software is being updated, if you use a few tricks. I knew most of the one's they mentioned already, and use them on my company website, which is far from downtime-proof, but has a 3-year uptime so far: call my software maintenence status "fairly sturdy".
If you're interested in upgrading to software maintenance status "bulletproof", then read something about fault-tolerant computing in Erlang. You'll learn more that way.
He is talking about the need to version protocols to simplify future change, you're talking about code refactoring which usually don't change the interface..
I've needed worked on a project where we used 'versionned protocols' though :-(
And that means the validation people run through their test book and create maybe 1000 or 2000 bug reports. These bug reports bounce around through the development teams acquiring cruft along the way. When the bug eventually stops bouncing somebody might have a go at fixing it. So they change something and if they can't see the bug anymore they go cvs commit right then and there. At the same time the other 1999 bugs are bouncing around, looking for a home.
At the end of the (as we call it) bug fixing process the original software doesn't exist in its original form. It appears to pass most of the test sbut you would not be advised to lean against it or anything because something else might break.
software maintenance =/= bug fixing.
Like many others, most of my work is maintaining legacy code (and we're talking everything from SUN's old Forté UDS to Visual Basic 6 and everything in between).
I wanted to do a Master's degree in CS, and during my interviews, they asked what would I be interested in researching... I told them I found software maintenance to be a line of study that hasn't been properly studied, and they dismissed it (mostly for it not being "sexy" enough). So it's not exactly a "hot" field of study...
There are three kinds of lies: lies, damned lies, and statistics.
Just wanted to say that the article was probably one of the best I had read in a while. I am a former member of the ACM (and may renew because of articles such as this).
Great reading!
Suppose you were an idiot and suppose you were a member of Congress
This reminds me of the work I did in Macro 11 back in the day. I used to spend a lot of time documenting each block of code, to the point that my partners on the program thought I was nuts. However, when we had to go back and maintain the code later, my blocks were quick and easy to decipher, and make any fixes.
ah, the good old days of PDP Assembly language....
Suppose you were an idiot and suppose you were a member of Congress
Was it just me who thought the article was awful? What it described was nothing more than versioning for applications. I fail to see how what he described as good practice was different from his openoffice wanting to be updated. Was it the need to restart the program afterwards? All examples he provided seem to need to restart at least one program. In the case of the addition of Canada, for instance, the whole server had to be updated and, I assume, it needed to be restarted. Then he said there was two version of the software, one in canada and one in the US that used the same server. If the clients talk to the server using any reasonable protocol the server would be able to handle both versions just as easy. It could be the case that the only difference is that one client is able to send one sort of message that the other isn't. I can hardly think of a software that has the maintaince problems he exposes. When he goes on to talk about the glibc with an old version of linux kernel. That library exists exactly to export an interface that any program -- old or new -- can use. It does that by communicating with the kernel directly, and I think it's pretty reasonable for one or two functions to stop working on old versions of ther kernel.
It should be referred to as "software extension and fixes". Or something else, but not "maintenance".
Many managers have been trained to defer maintenance in production environments as a way to cut costs. Which may be true if the equipment costs are low enough. If the labor + parts + other materials are all more expensive than replacement costs and downtime for broken equipment is cheaper than replacement costs, so what?
This creates a dangerous attitude. Too often downtime costs are not computed. It is also dangerouse when this attitude is transferred to software, which is *not* an industrial process. Many managers do not seem to understand you can not manage software development like an industrial process.
putting the 'B' in LGBTQ+
It's why leadership and direction are such important qualities in management, and why companies try to hire and keep the best.
If only that were so...
It's been my experience that most medium-sized and larger companies actively take steps to hire mediocrity and lose their best people, whether or not they realise it.
Such companies can typically be identified by a combination of three give-away signs: they are too large for everyone to know everyone, which creates room for a layer of middle-managers who are neither hands-on team leaders nor directors, and there exist dedicated people/departments for things like HR and IT.
As soon as you've got that kind of culture, you're likely to be hiring cookie cutter candidates. Then you have to introduce business processes that cater to the least common denominator, which by design stifle individuality and innovation. Unfortunately, those are often exactly the reasons that the best people are the best people: they bring unique technical strengths, creative ideas inspired by their personal past experiences, or insightful points of view. Having made individuality thoroughly subservient to the machine, you have discarded the extra value good people could have contributed, so those people feel a lack of motivation; you compensate them on a level similar to the cookie cutter guys, so they feel a lack of recognition; and since they are exactly the people who can most easily find new work, they are the first out of the door, leaving your overall workforce even lower on the scale. And thus the downward spiral continues.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
way to tell the entire internet that you, Brian Gordon, don't know jack about computers.
And that is the problem. Everyone thinks the glamour is the new code... which they invariably *screw* up and expect someone else to come in and fix those "details". Developers/Architects suffer from ADD.
The real problem here is the use of an atrociously bad metaphor. The term "maintenance" refers to keeping something in its original state, so that it continues to perform its original function. Software never needs this, because software doesn't wear out or degrade. Except for the occasional dropped bit, which usually totally destroys the software so you have to restore it from backup, software stays the same forever, without change. I have programs that I wrote 25 years ago that still do their original task perfectly, despite constant use.
The problems is that we're using "maintenance" to mean making changes to the software, to change its behavior so that it can do new things. Before software, such changes were never called "maintenance". They was called things like "revisions" or "alterations" or "redesigns", something with a "re-" prefix to remind you that you're changing things.
To use the usual transport metaphor, it's like you have a very functional road that has long been maintained by fixing the usual cracks, potholes, and other degradation that happens to all roads due to traffic and weather. Then management comes along, and decides that the road needs to be usable by trains, and this is a simple enough enhancement that the Maintenance Department can do it in a day or so. The maintenance folks have no experience with railway building, but they dutifully rush out, dig up the road, and install tracks. They don't have the training or time, so they haven't properly rebuilt the roadbed to support the much heavier traffic, and nobody had told them about the need for a much larger turn radius that trains need than cars do, so trains can hardly use the result. It takes several weeks, so management criticises them for not adhering to the scheduled two-day delivery time, as well as for building a low-quality railway that constantly needs more "maintenance" work.
Then, some time later, management decides that the road should also be "slightly augmented" to handle boat traffic. Never mind that it goes over a hill; that can be handled by building a system of locks. So the "maintenance" crew sets out to do the job, again without any training in canal building, using tools designed for maintaining auto roadways. And again, they're criticised for getting behind schedule and doing a poor job. And the water supply for the lock system is a constant resource hog.
Of course, with roads, railroads and canals, people can see the physical results. Even the dumbest manager can see that they're not at all similar (and why you try to avoid building canals over hills). A problem with software is that it's just bits hidden inside the computer's memory, so it all looks the same from the outside. This makes it hard to understand the difference between small bug fixes and total redesign and rewrite. It's all just "maintenance", right? How hard can it be? And why are the software maintenance people always so slow and behind schedule? They can't be very good at their job, right?
I suppose there's no chance that we can ban the use of the word "maintenance" in software. That would be the best approach. But it would require admitting that we'd made a major blunder in our terminology. So we're probably stuck with doing "maintenance" to make major revisions and retargeting of software.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
Traditional (or "everyone's first project"). This one is easy: don't even think about the possibility of maintenance. Hard-code constants, avoid subroutines, use all global variables, use short and non-meaningful variable names. In other words, make it difficult to change any one thing without changing everything.
Sorry, they already lost me. They start out with the assumption that "traditionally" people write bad code. Sorry, but if you're writing code like this you need to go back to school and learn not to. Then come try to get a programming job again.
To make an opening statement of "your code sucks" turns me right off.
- Jasen.
Perfective maintenance makes the application easier to maintain in the future.
The original poster had this to say:
Taking code and cutting its size by half, fixing up all the screwed-up inconsistent formatting, while adding functionality and reducing bug counts, is a pleasure.
Reducing bug counts is software maintenance as we know it (it's why you are there), not perfective.
Cutting code size, I could possibly see as perfective although that should be done with a lot of testing around it. Changing formatting, is actually rather bad in a maintenance role as it increases the amount of code to review for no good reason (I personally find such changes neutral in regards to the ability for others to understand code).
Adding functionality though? No way is that "making an application easier to maintain"
I don't really see his comments as relating to Perfective kinds of maintenance. I was mostly reacting with horror to the "add new features" aspect which is development.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
s/Maintenance/Development/
Oh sure, there are a handful who know something, but to a first approximation, the above is correct.
The Classic Computer Science college courses in the 1980's and on back to the 1950's taught the following things:
Software Psychology
Software Maintenance
Analysis and Design
Logical Methods
Quality Control
Writing Secure Code
Writing Optimized Code
Best Coding Practices
Debugging
Scaling
Migrating Legacy Software
Managing Garbage Collection
Managing Memory Usage
And many more.
The Modern Computer Science from the 1990's on up to now, has skipped many steps above and just goes on to writing the software code and not worrying about anything else. Which is a lot like teaching Natural Science classes without even bothering to follow the Scientific Method. The Above are the Computer Scientific Method of Computer Science, but somehow got over looked and thrown out of the modern Computer Science classes.
For example many Computer Science students couldn't handle garbage collection and memory management so they had invented Java and C# and other programming languages to do it for them. It is because of lack of optmized code and memory management that we get huge programs that need a 3Ghz or faster processor and at least 3Gigs of RAM to run, when really a 200Mhz processor and 512M of RAM would be enough to do the same thing with garbage collection and memory management.
I am not the old guy saying "Get off my lawn!" I am the exceptional old school comp sci worker saying we need to "think up nifty new ways on how to combat the problems that we currently have in the computer industry" by improving upon our computer science skills and adding in the above to them to solve those problems. I am not even close to retirement age, but I know a broken system when I see one. The current comp sci system is broken, and colleges are not teaching the right stuff we need to "think up nifty new ways on how to combat the problems that we currently have in the computer industry" and very few if any are teaching that.
We need to bring these old school skills to the 21st century and modernize them.
If the current computer scientists refuse to write books on the subject, and shun people like me, then I guess people like me are going to have to school the young and old with books on those subjects and make millions selling them to the clueless and the firms that have no idea how to fix their own problems. I'm Orion Blastar, and I am coming to save your arse with a different and better way of developing code and an improved computer science, not seen since the days of Ben Snyderman.
Make way for Fringe Computer Science and Modern 21st Century Computer Science.
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
Yes. Bug trackers should have statuses like "Developer in denial" for situations like that. (Mozilla's bug tracker has a "WORKSFORME" status which is used far too much.)
I'm sorry, that's code for 'Gimme a test case worth a damn!'
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
This sounds like a familiar story. You've got the right attitude now.
Over-the-top Response Guy! Giving "Over-the-Top Responses" since 1970.
Either it is because you are a junior, or you make consumer-level software.
Once you get to be mid level you will start getting requests like this from management sometimes:
"Hey we are on site at potential customer X that is evaluating us, and feature Y (some critical product defining feature recently released) is not working in their environment because they have Z (a corner case never accounted for). This is a (insert V hundred thousand or larger dollar value) deal and we need this fix in before end of day (which is in 3 hours) - can you look into it?"
Stuff like this happens at any private software company who is in the business of selling software to other companies. Of course if you don't make the deadline it is not like anything bad happens to you (hopefully), but MAKING the deadlines and winning the deal sure makes you look good when the next annual review comes along.
"Utilize the power of version numbering your structures".
It's a cool suggestion, but I wonder if it was worth reading the whole article... perhaps it pays of in the future...
A useful article, but does not address the hard part of software maintenance. It is about how to design software so that it is easy to distribute and install updates, not about how to track down the bugs and determine what those updates are.
This is nothing new. I have been in the IT industry for about 40 years. I cannot tell you the number of times I have run into this in various installations. I can also tell you the horror stories that would leave you wondering what were they thinking.
For about 20-30 years IBM was pretty much in the same mode until user groups united and screamed at IBM to come up with a solution. Mind you it was not overnite that IBM (and their customers) came up with a pretty darn good solution. *IF* you followed the rules you could pretty much bring a new system in and reapply all the local modifications within (most of the time) with either a small amount of work or a reasonable amount of work to get the local modifications onto the new system in usually less than a weeks effort by one person. There are some exceptions I know but if you kept a orderly system and followed the rules you were pretty much a minor clerical work on local modifications to install a new release of the OS or a major OS upgrade. BUT everyone had to follow the rules (yes even IBM) the rules are quite easy and straightforward. No need to list them here as only a person with IBM OS background would understand them, but they are reasonably simple. Depending on how you chose the system type to be installed it was for the most part straight forward. It also took thumping of management heads to get into the mode of either complete replacement of the the OS or upgrading the OS with another release. Also, the fact that (currently) if I recall correctly IBM drops support every two years on their flag ship OS so you are on a treadmill (so to speak) of keeping everything reasonably current.