Keeping the Lights On

An anonymous reader wrote to mention an IBM article examining the role that older workers, experienced with legacy systems, should play in system maintenance. From the article: "Many enterprises still execute critical business operations ... via older software systems that run on large, mainframe computers rather than individual PCs. To meet changing business needs, these companies continually update, extend, and integrate their systems. Paradoxically, many of these companies also have policies that threaten the single greatest source of knowledge about their older systems: their most senior personnel. Although the aging workforce represents a vast pool of talent and experience, these businesses neither actively recruit senior workers nor provide incentives to retain those on staff.1 Instead, they mistakenly assume that they can hire younger, lower-paid people to perform the same tasks."

Comments

Anyone can do this job

[totallygeek]

I feel the effects of this all the time, and I am not old yet. I have been asked for years, "What if you get hit by a Mack truck?" Now, I would say that in the last five years, things with Linux have standardized to the point where my Linux stuff could be outsourced. But, how do you replace intimate knowledge of network layout, homegrown code, machine function, and how to get around policies to get things done?

Re:Anyone can do this job

[BiggerIsBetter]

In an ideal world, much of that would be handled by appropriate documentation. In the real world, it gets harder...

We like to call ourselves professionsals, but compare our jobs to that of, say, a mechanical engineer. An engineer who jumps on the lathe and starts welding without designing and documenting what he's doing is little more than a skilled craftsman IMO - same for most of us IT guys. There's little to evaluate and test against when something goes wrong later, and the next guy to face your work has to reverse-engineer it before actually doing his job.

The "intimate knowledge" should be written as explicitly a possible, and common workarounds can be put in a cookbook format. Some things - like the political stuff - probably has to be passed word-of-mouth though. We all know about the ongoing costs of IT, and how big an issue maintanence is, so even 1 afternoon a week to write it up is worthwhile in the long term. The problem is pretty common in IT, but IMO it's not good enough (I'm not perfect either). It's a cultural issue, I think... too much hacking and "getting it done" and not enough planning and documentation.

Re:Anyone can do this job

[bladesjester]

"We like to call ourselves professionsals, but compare our jobs to that of, say, a mechanical engineer. An engineer who jumps on the lathe and starts welding without designing and documenting what he's doing is little more than a skilled craftsman IMO - same for most of us IT guys."
snip snip
"even 1 afternoon a week to write it up is worthwhile in the long term"

That's a nice thought, but you'll find that the technical staff often isn't to blame. When you're understaffed and the world is constantly falling around your head because the beancounters see IT as a drain instead of an asset, you often don't *have* an afternoon to sit and write documentation. The time ends up being spent just trying to keep the place running and/or meeting the deadlines set by the pointy hairs.

Re:Anyone can do this job

[Kadin2048]

I agree with much of what you said, but in terms of a resource, an actual person is many times better than even the best, most well-written documentation. Sometimes, if the system is complex enough, I could see the retaining of an additional person being justified, even if their only function was to act as a 'living encyclopedia' and help other (more junior) employees troubleshoot the system.

I can't count how many times I've had a problem with a well documented system, and even after wasting hours of my time (and in many cases, hours of other people's time who just have access to slightly different documentation) only to finally find someone who has intimate knowledge of the system in question, and get an answer in five minutes. There are limits to how good documentation can be, even when it's searchable, indexed, and cross referenced.

No amount of documentation in even the best information storage and retrieval system can compare with the power of a person who actually understands a system intimately, and then applies that understanding to other people's problems as needed.

What usually happens...

[Dekortage]

The article states, "For a truly objective assessment, it is usually best to engage an external consultant who is not involved with system maintenance. However, senior organization members are an invaluable resource for these consultants." No, what usually happens (in my experience, 20+ years IT) is that the seniors get fired, then have to be hired back as consultants at 3x their former pay.

Ask a relevant question

[Anonymous Coward]

Who would you rather operate on you: A young surgeon, or an older surgeon with years of experience? (I know, programming/administration != surgery, but I think most people will understand the point.)

Re:Ask a relevant question

[Now.Imperfect]

Also the young doctor(IT guy) is fresh out of training and knows the latest techniques and is probably more willing to drive forward and encourage growth. Where as the older doctor is losing his steady hand(or his memory for the IT guy) and has been using the same stuff and probably has nearly no drive to keep the technology somewhat current. I see it in my dad, who I'd say is one helluva a sys admin with his grasp of security/networking in general, but its obvious that his age does effect his way of thinking... and its not always "wiser" imho.

IBM is trying the save a piece of his bizness

[timeToy]

This article comes from IBM, this is no surprise, given that they must have the most interest in maintaining the "mature" workforce in the enterprise
Example of discussion:
Manager: We need to increase the throughput of our Mainframe system
Old engineer: Let's contact IBM, our mainframe hardware manufacturer and add a couple of processing units to the system
Young engineer: Nah, just let migrate to Linux, I can get you the same service, same performance, for a fraction of the price if we get a cluster of cheap Opterons, plus this will scale easily in the future and we will be vendor independent.
(Obviously the young engineer didn't had to deal with migration issues in the past, and obviously the manager is going to be sold on the bottom line)

Re:IBM is trying the save a piece of his bizness

[deanj]

Ha! Don't assume anything is obvious.

You assume that the "old" engineer has no idea about Linux. You also assume that the young engineer has the first clue about what the company really needs.

Both very very bad assumptions.

Experience is the key here. Sure, the younger guy might have a good idea, but spitting buzzwords at the boss without the first clue about what it's ultimately going to mean to the company will do only two things: 1) Peg you as a loudmouth know-it-all or 2) Get you put in charge of making that migration.... Neither of those are going end up pretty.

When YOU get to be an older engineer, sit back and listen to all the younger guys that assume YOU don't have a clue about what's going on. You're not going to like it one bit.

Re:IBM is trying the save a piece of his bizness

[QuietLagoon]

Unisys knows their mainframe business is dying, and is pushing Dell servers with Windows now. They've been trying to migrate some of these mainframe systems to server groups (not exactly clusters, each box has a role) with horrible results.

The root problem in your sentences I've quoted is not the migration from the mainframe, it's the word "Windows". To expect Windows to have anywhere near the reliability and performance of an IBM mainframe is, at best, humorous.

As you go on to say, IBM mainframes are highly optimized computing systems, systems that excel at moving data from the disk to memory to processing as quickly as possible. Windows boxen don't even come close in this regard. And then there is the reliability of Windows vs. IBM mainframe OS's.

For the past 20 years I have been hearing how Windows is going to kill the mainframes, yet IBM's mainframes still seem to be doing rather well. I hear they even run Linux nowadays. Far out (to use the terminology that seems appropriate).

Re:Training

[Pichu0102]

It's not just training, it's also experience. Those with experience are more than likely to be able to do things slightly better than those that have just finished training, since they already know almost exactly how to do things.

I got out of mainframes 15 years ago

[Anonymous Coward]

They were dinosaurs then. What younger workers would deliberately learn a system that was already obsolete when they could learn leet new skills instead. This is really an issue of who should bear the burden of maintaining a skills base, the companies or the workers. The companies will naturally try to pass the cost of doing business on to anybody else they can if they can get away with it. Now that there's a shortage of mainframers because they laid off everyone they could to save on pension costs, well I say that's poetic justice.

Same old problem, different questions....

[zappepcs]

Disaster recovery, and maintaining operations in the face of reduced knowledge base and personnel are the two sides of the same coin. The military regularly does this (no comment on efficiency here) but business as a whole do not do this. /.-ers think in terms of IT, but there are other issues too. Think about customer service departments, billing departments, all facets of a business. Disaster recovery is not a simple or trivial issue.

Data back-ups and documentation are not sufficient. To truly be prepared, a company has to have an agreement with temporary worker agencies to replace certain people, and practice to make sure that the documentation is enough....

In the case of New Orleans, they not only need people, companies there need their buildings and hardware replaced. Other, less demanding situations are losing people because of personal responsibilities to family in the aftermath of the storms. Those people have to be temporarily replaced in some cases.

A truly thorough disaster recovery plan is both large, complex, and on some levels, very scary. It has to cover situations where the entire IT department is in the same bar when a bomb goes off. Who does what then? Do they tell the IT staff not to socialize together?

When the only legal person in your SMB is now missing, who steps in to sign that paperwork?

There are tons of things to think of. The simple things stick out, but true disaster preparedness is a horrific thing to accomplish, and it costs big $$$$$$$$$$$

Google for information, it is scary....

Two cents used...

Re:Training

[Fitzghon]

I think you misunderstood the article (did you read it?)
Companies that hire young tech people need to train them to use the mainframe apps. The article points out that this is a losing situation, because the company went from having an experienced and trained worker to an inexperience worker that they have to train.
Consider it this way: a company's tech guy gets hit by a truck, they have two candidates to hire from:
Candidate A: Has no knowledge of the platforms being used in the company, has no experience with similar systems.
Candidate B: Is familiar with the platforms and has extensive experience with similar systems.
Who will the company hire? Candidate B of course!
The problem is that companies are now firing Candidate B because he is old and hiring Candidate A because he is young. Is it a worthwhile decision?

Fitzghon

It's not just IT

[Anonymous Coward]

The comming retirement of the baby boomers will cause a loss of institutional memory in many companies. THe next 5 years are the start of the boomer retirement. What is going to suprise many is the smallness of generations X and Y. There just aren't enough people to fill the retirees slots.

Pretty good advertising by IBM...

[NeoBeans]

Disclaimer: I work for a company that competes with IBM. But then again, don't we all?

IBM encourages older workers to stick around and keep mainframe systems running. Of course they do. The maintenance contracts that IBM is paid on for these mainframes (and the IBM Global Services guys who sit on-site to babysit these applications) are priceless. I was part of a team of consultants that was involved in moving a major mainframe-only app to Unix/J2EE and IBM did everything possible to forestall and prevent it. When we were done, we were saving our customer almost $500k a month on the costs associated with maintaining that (admittedly simple) legacy application.

I know this is blasphemy on Slashdot, but when companies like IBM get in bed with open-source and with technologies we (okay, that I) favor (in my case Java & J2EE), you have to remember they are *not* a product company in these spaces -- they are a consulting company. Sure, they sell their hardware by pitching it's flexibility (a good thing), but they slash prices in order to place their consultants in your organization to "help out".

This is not to say they are evil or bad. But only that all of this is wonderfully self-serving and really doesn't pass for news...

3X their former pay

[dpilot]

Those that I've spoken with who have gone into consultancy say that once ALL of the benefits are gone, you need 2X your former pay just to break even. That's vacation, health care, dental, sick days, etc. Not to mention that consultants need to provide some sort of office space, communications, etc. Those costs were formerly paid by the employer. That's why the burden rate for you usually ends up looking like somewhere in the realm of 2X your salary.

So the 3X price might seem high, and it is, a bit. But to the company, it's only 50% higher than having the employee would have been. To the consultant, it's 2X the "discretionary" (after fixed overhead) pay rate. From the company's point of view, they only need the consultant for a fixed task or set of tasks. From the consultant's point of view, he needs some padding to weather the lean times.

So it's not as extravagant as it looks, for either party. By the same token, I don't really think it's a win-win situation, either. It just is.

(I used to know JCL, and be pretty decent at it. I'll bet I could relearn fairly readily. Actually, JCL was kind of neat, because everything stayed put until you did the SUBMIT. Then it was too late, and you'd better have done it right.)

Re:duh

[tomhudson]

Old people = dumb p<b>oe</b>ple

Well, we sure won't have to worry about your code crashing ...

... because it will never compile with your typo rate ...

... and nobody's going to hire a kid whose english makes perl look good.

Agreed...

[Jeian]

I'm 19, so predictably I had little mainframe knowledge in the course of my (mostly self-taught) computer education. I too thought that mainframes were a thing of the past.

Then I got a job at my current employer, where mainframes process all our data, and got to talk to some of the datacenter operators who actually work with all the different platforms we use. What did I find? Mainframes, while not the most user-friendly beasts on the planet, are still indispensible when you have hundreds of employees and hundreds more clients needing numbers crunched without bringing your system to its knees.

Look at the computers that keep the world running - the ones that process your bank statement, the ones that process your credit card statement, and so on. I think you'll find that mainframes are the backbone of the processing infrastructures of the organizations that do this.

Personally, I think it's the "de-geekization" of computing that's to blame. Fewer people are being trained on the complex workhorses behind the scenes, and instead are being trained on the EZ-2-USE REVOLUTIONARY TECHNOLOGY OF THE WEEK/MONTH/YEAR!

Re:Rubbish

[Spy der Mann]

In all my experience, older employees become mentors to the new recruits.

Speak for yourself. Here in Mexico people over 40 aren't hired *AT ALL*. No matter how much they might know. And there's no way to sue the companies because there's no way to PROVE they didn't hire you for your age.

But everybody knows.

Re:Training

[Anonymous Coward]

This is a patently naive perspective. Not that there aren't older workers with minimal competence who have managed to hold on one way or another to jobs over the years, but in general an experienced professional is going to have a lot more work-related wisdom than a new hire. That's why they've been kept with the company over the years. IBM generally only hires entry-level people, not experienced professionals, and isn't proficient at assessing whether the new hires are competent or not. There is a widespread misconception there that anyone can be trained to do a technical job. If you had any experience whatsoever with the general workforce, you'd know only a fraction of the people applying and/or hired for technical positions are actually extremely competent.

The bigger truth, particularly at IBM, is that an older worker (read that closer to retirement) is a much bigger liability because of the cost of retirement to IBM.

It's also the case that most IBM managers have short term, personal goals at stake, rather than the welfare of the company. And their widespread inability to accurately assess the productivity of their employess makes it impossible to justify someone that has a 2x salary versus a junior programmer. It's like their corporate-wide Cost Cutting programs back in the 80's & 90's that did away with lesser-paid secretaries, support personnel and office supplies at a tremendous savings to the company. That meant their programmers and managers (only the executives were left with secretarial support) spent their time searching for non-existent supplies, or running to Office Depot, and managing more logistical tasks. They had no way to measure this impact to the much higher-paid employees, and so considered this major impact to productivity a cost savings.

Why don't you read The Mythical Man-Month (which ironically of course, was written by an IBM-er)?

Re:A misunderstanding

[ScrewMaster]

True ... but a good engineering manager who is capable of leading his team and establishing a work environment where such SOTA solutions are commonnplace is just as rare, and just as hard to replace. Good engineering and good management are not by definition adversarial, in fact they are complementary. And the bigger your development team, the more essential it is to have good management.

It's not enough to just put a bunch of talented engineers in a room and expect results. No matter how smart they may be, engineers are people too, with their own distinct personalities, strengths and weaknesses. They can benefit from good leadership just as much as a platoon of soldiers. In fact, several times I've been surprised to find a fellow engineer whom I considered to be second-rate turn out remarkable work when given proper leadership and encouragement. Conversely, a bad manager can turn a roomful of Wozniaks and Hertzfelds into so many paperpushers. Speaking as an engineer who has been in both situations, I'm not inclined to dismiss management as irrelevant: but I will agree that the bulk of it is mediocre at best, and that pretty much guarantees mediocre engineering.

When you get older....

[deanj]

I'm surprised at the number of young engineers who think that the "old guys" don't know anything about the "latest" tech.

A while back I heard an intern going on and on about how the young engineers (and he considered himself one, even though he hadn't graduated yet) were the best ones to come up with the new ideas for everything. The "old guys" just didn't have what it takes.

What a fool.

This isn't a "young" or "old" thing. This is a "good idea" thing. That comes from being a good engineer, not being young or old.

Not every young guy pays attention the latest technology, just like the old guys don't just stick their head in the sand when it comes to the new stuff. As a matter of fact, the older guys were the ones that were dinking around with all that new computer tech back in the day. Most of them did a lot more than the "fresh outs" do today.

If you're one of those guys that believe that the young guys are the stars when it comes to engineering, and the old guys should just step out of the way..... well, you're going to get old too. When that happens, I seriously doubt you'll feel the same way that you do now.

Re:Training

[lordperditor]

What a crock.

You can't seriously think sending some youngster on a training course is going to replace the 20-30 years of experience and knowledge of an older senior staff member.

Yes they may manage to get by, if it all goes well, but when it goes pear shaped you are gonna wish you had that more experienced staff member around.

I once heard a good definition of an expert:
An expert is someone who has made all the mistakes there are to be made in a particular field of expertise.

Sending a youngster on a training course is no replacement for years of experience.

Maintenance Strategies

[anorlunda]

There's more to it than senior versus junior IT workers. The following clips from the article help illustrate.

"Precisely when the organization is trying to gain a return on investment, software operating costs may start to climb. ... At this point, support costs can start to consume a larger and larger part of the IT budget, severely limiting new investments."

The company often feels that software maintainers are extorting money from them. That's especially true when the application is not an external package continuously upgraded with new features. Managers expect that a paid-up static application should cost zero to maintain. This was made very plain when Y2K remediation work was complete and the Y2K workers, young and old, were booted out the door with parting greetings that sounded like "good riddance."

As a senior (now retired) software type I wrestled with the software maintenance dilemma for decades. I saw that old code was designed for the CPU and memory limitations of its day. As time marched on Moore's Law rendered old code useless faster than poor documentation or obscure programming languages.

At one point I resolved to put an upper limit of 10 years on the life of any code. After that it would have to be discarded and replaced. Then I realized that if everyone followed that policy future generations would be doomed to reinventing the wheels (i.e. the logic) invented in earlier versions. Actual progress would approach zero asymptotically. Consider for example code to control a nuclear plant. The plant has a 45 year lifetime, and the laws of physics and principles of control don't change in that time. If we had to reinvent all the control software four or five times in the life of the plant, it would be a terrible waste. The most modern implementation might be more efficient and superior in quality, but there is no assurance that it does a better or as good a job at controlling the plant as the first version.

Both extremes are wrong. Maintaining old static applications indefinitely is wrong. Periodic discard and replacement is wrong. My final conclusion was that old applications need to be rewritten and re-implemented and expanded and modernized gradually. If we re-write or re-implement 10% of the code every year, then none of the parts get to be more than 10 years old. We also deliberately blur the boundaries between old and new applications and the boundaries between developers and maintainers.

In my experience developers resist this notion more than management. Developers love reinventing wheels. I bet every open source developer worth his salt would love nothing more than his/her own chance to invent Unix from scratch, and along the way every application and algorithm that went with it. In any case, they really hate the idea of re-implementing some predecessor's cleverness embodied as code. They would much rather create their own fresh version confident that they can be cleverer than anyone else. It goes with the territory when we seek creative people to program. They like to create -- duh.

One other thing, when our gradual rewrites of old code reach the point where everything is fully expressed as objects, then the burden of rewrites and maintenance should be drastically reduced forever after. Isn't that the promise of objects? Expandability? Adaptability? Any large application well founded on objects should be able to morph itself into any future application one little bit at a time.

Software Development Is not Like Engineering

[iangoldby]

It's probably worth mentioning a 'seminal' essay written by Jack W. Reeves that suggests that software development is fundamentally different to other engineering disciplines, because the construction phase costs essentially nothing. (He considers coding to be part of the design.) Well worth a read (although it's no excuse for the complete absense of documentation).

Regarding the above analogy, the engineer jumping on the lathe is analagous to the software developer hitting the compile button. And so the analogy breaks down.

Code as Design: Three Essays by Jack W. Reeves.

Lathe? Welding?

[sczimme]

An engineer who jumps on the lathe and starts welding

Welding... on a lathe? Such an engineer is either very, very talented or someone to avoid at all costs - quite possibly both.

:-)