Slashdot Mirror
Institutional Memory and Reverse Smuggling
An anonymous reader writes "Anyone who's worked in a large engineering firm is familiar with institutional amnesia. Things get built, and then forgotten. Documentation is supposed to help, but rots, is lost, or uses obsolete methods and notation nobody understands anymore. I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works. I even have some personal copies of documents they seem to have lost, which I have to awkwardly smuggle back in. I don't find these kinds of experience written about often, but I'm convinced they're more common than you'd think."
A very relevant area where this problem can be readily seen is computer data formats.
But the NDAs keep me and everyone else involved from talking about it. :P
I have personally witnessed this happen with my former employer. Higher ups seemed to think that there was some benefit to the company "memory" or "team experience" if we almost brought a product to market. People would argue that at least we now had the experience to do this again in the future if we had to. Totally wrong. Teams were broken up and key players re-assigned. Some people left the company or moved to divisions physically remote from their former bosses. Any reports written could not be used to replicate anything useful and, to be honest, who would write a report explaining the history of an unsuccessful product? It never happens. Real knowledge is lost in a barrage of wasteful spending.
Sorry, but gray text on gray background is making my eyes bleed.
Somebody writes the code, doesn't bother to comment it at all and then you come in years after the fact. You look at the code and wonder, "why did he do it that way instead of this way?" Then the big gotcha, you think I could ask him but he left the company 5 years ago(At this point slap your forehead and hope you don't break anything working on the code.)
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
I design automated testing systems. I find myself in the position of re-solving problems that were solved as much as 10 years ago in prior automated testing systems. There's an odd tendency of people to simply passively accept whatever the manufacturer of the new system gives us as somehow "better" or more advanced that what we had, even when the new solution is an obvious fail, or (ahem) undeveloped. So I fix things to make up for the shortcomings of the new software, more or less the same way I fixed them 7 to 10 years ago. Things start working again. Everybody is happy.
I seem to have discovered job security by re-inventing the same thing every 7 years or so in a slightly different form using a different programming language. Of course, that sort of describes the entire computer industry, more or less.
Please do not read this sig. Thank you.
Why bother trying to smuggle them back in. Tell them you'll work on figuring out the bits you can't remember and will write a new manual. You get to charge them for time you don't need to spend and avoid getting caught.
More like this, please, and less about the Apple/Andoid/MS/Samsung/**AA suit-of-the-minute. I know, I know, flamewars == pageviews == Step 3, but you've occasionally gotta throw something out there for us old-timers.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
I currently work for a company that has instituted an incredibly restrictive development methodology they bought from a big consulting firm. It requires multiple forms be filled out for every program written, every requirement for every program written, every request for every change to every program or application written. All of these reviewed by coworkers who are not about to alienate team members, and reviewed by lower management who want to look good to upper management by having everything go smoothly.
These documents are then stored on the LAN. To never, or rarely ever, be read again.
The one thing it does not enforce or require, is meaningful documentation in the code itself.
It doubles, or more, the time it takes to do everything. But it does nothing to stop the mistakes any better than the procedures that went before. If anything, it finds less errors, because we were not given more time to do this double or more amount of work. So time is so compressed, we do not have time to do anything other than get it working and get it in.
One thing it does do very well, is prevent problems from getting fixed. The only people that will start a change effort are those that notice a problem and are affected by it enough to have it cause them problems. Otherwise no one wants to go through the bureaucracy to kick off any sort of change effort, which leaves a lot of ticking time-bombs in the infrastructure configurations, application designs and application code.
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Documentation located anywhere else will be lost, or obsolete many more times than not, before you ever really need it.
If anything, documentation in code should be reviewed by people with absolutely no connection to the application it is for. If it is good enough for them to figure out an understand what is being done, and more importantly why it is being done, only then is it worth anything more than the bytes is written with in storage.
I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works.
I certainly hope that you negotiated a much higher salary than before.
I have known people who were fired in a "chain saw" maneuver who proved to have core corporate competencies in their heads. They negotiated for a much higher rate the second time around...
I'd be very careful in this situation. Even though you are supposedly "doing the company a favor" by smuggling the documents back in, were you even legally allowed to take the documents in the first place? Most employers have contracts or rules that state you can't remove documents from the office or that when you leave the company you must return all property to the company. Furthermore, it is most likely that this documentation was property of the company all along and in that case did you break any laws by removing it from the company and keeping it? IANAL but this is a very tricky situation. You may be doing good for the company now but you are profiting off of documents that you do not own and may not even have a legal right to possess.
Personally, I'd stick with just what's in my head and not keep any documents, files, etc. from a previous employer. It's just asking for trouble.
If the history channel read this story, they would undoubtably conclude that the plant was built with the help of aliens.
My situation is different, yet similar. I just have so much responsibility for so many things at work, I simply can't remember how everything works. At times it is frightening, especially since year after year there are more things happening and I get older, so it is already inherently more difficult.
I also find myself "protecting" my sanity by almost intentionally forgetting how some things operate, especially those things I don't like.
I think as a manager, one of the most important things to do is learn to effectively document everything. I always knew documentation was important, to the point that I have a log of everything I do at work, every day, that spans 23 years! But those are just "reminders", not proper documentation. It helps to jog memory or memorialize certain facts for reference. Nothing replaces *proper* documentation, though- the kind that is well thought-out, has great detail, and is organized properly. Further, it will have insights into defining the problem and the solution and WHY you selected certain paths and not others.
Proper documentation takes considerable time, and many people think it is a waste of time, especially when one is drowning in projects and deadlines. When you reach the point that you can't handle the load without continuing to document properly, you know you are in trouble. In the short-term, you can "win" by getting more things done. But in the LONG TERM, you are basically screwed, because you will spend a tremendous amount of wasted time later trying to relearn what you did and why. Or worse, trying to defend your decisions and can't because you can't find the information, it doesn't make sense, or it just doesn't exist.
i did so slavishly at first but then my boss pointed out that comments are rarely updated with the actual code. a contrived example:
int foo() // always returns 4
{
return 5;
}
now I have a coding style that the simplest fool can understand. that's quite a different thing than the arrogant coders who feel that their colleagues are incompetent if they cannot follow uncommented code. if someone cannot follow my code then I feel I have done something wrong, not them.
Request your free CD of my piano music.
Primo Levi, "Chrome" in The Periodical Table describes a similar problem with a paint formula at his factory in the 1950s. Evidently, during the war, they had a QC problem, included some chemical to correct for that, and the formula became a cargo cult, long after anyone knew what it was for. Someone changes some other factor, and their batch of paint gets the consistency of liver. So our hero had to reverse-engineer the trade secrets of a previous generation, back when he was working in the lab at Auschwitz.
Until you write it in cobol.
Deleted
if your own employees do the work your company retains the knowledge of how to get it done.
if you outsource you never learn how to do your own work, instead you finance your outsourcing firms efforts to learn how to operate your business. not only do you not gain that institutional memory but that outsourcing firm can now perform that same work for your competitors.
Request your free CD of my piano music.
now one does need to document the expected inputs and results of subroutines. better than commenting is to use assertions as well as unit tests. so for that contrived example, what I would actually do:
int foo()
{
int result = 5;
assert( 4==result);
return result;
}
if you don't maintain the assertions and unit tests with the payload code you'll find out about it right away.
Request your free CD of my piano music.
Maybe if the company had given itself a name it would have been easier for it to keep track of information.
.sig withheld by request
Maybe some companies need a Historian or two.
;).
An important task for Historians is to regularly remind the bosses what Historians are for so they don't get sacked
NASA has lost ALL of it's Saturn V knowledge.
Uh, no it hasn't. It may not be readily accessible, but there are (literally) tons of Saturn V information still available.
The scientist have died, the documents have rotten and been lost.
That'll be a surprise to the people who've put many gigabytes of Apollo-era documents on ntrs.nasa.gov.
What they have lost is much of the software; the Apollo Guidance Computer software only exists because some of the programmers had old listings that were OCR-ed, and the Saturn guidance computer software seems to be long gone.. there are a couple of computers left that may still have the code in their core memory, but so far no-one has been brave enough to try reading it out since that's a destructive operation and if you screw up you won't get a second chance.
Fortunately if someone was to try to rebuild a Saturn V they'd use completely new electronics and software anyway. Heck, it would probably have to run Windows...
At my institution we routinely buy back equipment from salvage yards it was sold to. But this actually makes sense. We clear out lots of old stuff rather than pack ratting it. It's a small price to pay for the convenience of getting stuff we need back.
Some drink at the fountain of knowledge. Others just gargle.
Management don't care, why should you? It really can't be all that much of a problem.
The manager who didn't document anything finished ahead of schedule and below budget. He got a big bonus and moved on to another job at a different company based on his reputation for delivering early and cheap.
The manager who comes in to organise an upgrade to that software ends up taking much longer than expected and going well over budget because nothing was documented in the initial spaghetti code. He gets fired for being incompetent.
Years ago people solved this problem but they didn't document their solution well, so here we are again.
One of the uncomfortable truths (uncomfortable for MBA cost minimizers) is that know-how is between the ears. It is not in the manuals or specifications, which merely prove that their writers had the know-how. Even more important is the know-why, which is part of the institutional memory which also resides between the ears.
I have witnessed some "technology transfers" where a set of working products was transferred with documentation, design data, and much background information to a group in another region (mostly EU/US transfers). The first group is then wound down or redeployed. The new group does fine at first, until a new version of a product is needed. Then they always make mistakes the original designers would consider childish or stupid. They don't know the why behind design decisions, and don't know what to avoid. In my experience, it takes 5-10 years to build a decent product design group from scratch. On the other hand, an existing design group will sustain itself by mentoring and guidance of new inductees by experienced members.
The only successful technology transfer which I participated in was one where a few senior designers were transferred with the set of products they were responsible for. It was far from cheap in up-front costs, but it avoided the crises and financial disasters which afflicted the other transfers after a year or two.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
I worked 30 years in astronaut training facility (full-fidelity simulators), and wrote many many documents on software that I wrote. I always kept my own digital copies, of course. Over the years, the contracts changed hands many times, and different document systems were implemented, and "all" documents were "always" converted from old to new. I was never able to later re-locate *any* document I had submitted to *any* of the document systems. So my copies of my documents were the only ones that actually existed that I knew of. This included meeting minutes, peer review notes, design and 'as-delivered' documents. So I think institutional amnesia is more the norm, and actual memory beyond 3-5 years is rare.
Pavlov wouldn't be so famous if he'd used a can opener instead of a bell.
This is incorrect. When I was working for Boeing, who was contracting to NASA, I saw the Saturn V drawings with my own eyes, in the data repository building at the Marshall Space Flight Center in Huntsville, AL. They are punch card aperture cards, which is IBM style punch cards with a piece of microfilm stuck into it. If you need a paper copy, they print from the microfilm. There are about 3 million of those cards, filling stacks of drawers, with all the drawings and many of the documents in them.
On the Space Station project, which I worked on for many years, we had a data vault at Boeing, which was literally a vault in the basement, where all the project data got copies stored, and we of course had to supply copies to NASA, who then put it in their repository. If there is one thing government is good at, is storing documents.
I saw a company lose probably 60% of their IT knowledge base. They announced outsourcing six months in advance, and directed current employees to document their jobs so that offshore workers could take over. Employees instead used the time to find other jobs, as one would expect. Transition was a disaster, and a year later they're still discovering things that the company no longer knows how to do.
One of the mistakes made was to assume that everything the old team did could be described as a procedure that a junior offshore employee could follow. This is a fallacy at a basic level, and shows a basic misunderstanding of IT. Knowledge is more than just memorization of individual procedures, it's information about topology, architecture, the flow of information, how the business works, and where the knowledge points are in the organization. (Who knows what.) Degradation of that knowledge base began on the day outsourcing was announced, and by the time transition arrived, there wasn't much left.
The basic misunderstanding, I believe is thinking that IT is generic procedure-based stuff that anyone could do. There are cases where this is true, but if you are in the business of providing a web based business, your IT *is* your value add, and your success is probably based on in what way your team does things that *aren't* generic.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
Management don't care, why should you? It really can't be all that much of a problem. More a perception than a problem. By all means go ahead and create/use a documentation system for your group but clearly there just isn't a requirement for anything more comprehensive.
All too true. The sooner you learn not to care, the sooner you will stop feeling the pain. Surrender to mediocrity; obey all rules and your managers without question. Above all, do not let your work interfere with your life. You will be a happier person, and probably get promoted. Listen to me, oh young ones...heed the words of the Ancient One: Despair!
Once upon a time, I worked in a relatively small organization within a Very Large Company. The VLC was renowned for its inventiveness, until a new manager came who felt that inventing things was much too expensive; it would be more profitable to just to put the word "Invent" on the corporate logo, and then re-sell stuff from countries who make things cheaply. Because our corporate logo was on these re-branded products, we must obviously have invented them. Thus, marketing ceased to be a partner of R&D, and instead replaced it. Clever, eh?
One of my contributions to the VLC was a simple document management system (DMS). My boss mentioned that all the engineers in our lab kept complaining that "they can't find anything"—specifically, nobody could find old documentation for code that was written by vanished civilizations of programmers that had previously labored at the lab. In fact, it turned out that this was a general problem among the labs of the VLC, and that they all wished mightily for a solution. Apart from really ancient projects (from, say 10 years ago), it was pretty difficult to find documentation for even recent projects, as each lab had a different way of organizing its documentation, different document formats, and no channels for distributing the documentation to other groups who might have a legitimate interest. So I brought in an easy-to-use and easily maintained DMS, pitched it to the engineers at our lab, and got them to use it. My boss then made me contact a bunch of other labs, and offer them the use of the same DMS. A lot of their engineers liked it, and there was much happiness. I got lots of pats on the back for what was really a pretty simple idea, and my boss loved me for it. A year later, I was laid off.
As I was the only one maintaining this system, and no one ever asked me to train a replacement, I doubt that this DMS—and its contents—survived, save perhaps as digital ghosts on some unlabeled backup media in a faraway storage cave. In a sense, I'm one of the perpetrators of information decay; I caused all the knowledge deposited in "my" DMS to get black-holed. Come to think of it, this is probably not a problem; I'm pretty sure they've laid off all the engineers by now.
I understood very well that simply setting up a DMS was not the cure for the I-can't-find-anything problem. It was a beginning, at best. What was needed was a comittment by the organization as a whole to make sure the information continued to be available for as long as the company exists. Because engineering practices change, so do documentation practices; documentation technologies themselves change—not to mention physical storage media and data encoding. My DMS probably should have been replaced with something better by now (if it had continued in use), and that means someone would have had to plan the transition, and move the old docs to the new system. Maybe the best thing to do is print everything as hard copy from time to time, and hire a librarian to keep track of this backup.
Considering how much has been said and written (and blathered) about "knowledge management", it's amazing how little attention is ever given to the temporal aspect of institutional knowledge. Every engineering company should have an Office of Knowledge that has as part of its responsibility not
Great men are almost always bad men--Lord Acton's Corollary
"But of course, this would imply a corporation with the ability to see beyond the tip of its own nose, which is rare."
The problem is not only corporate culture but societal.
Say Big Corp A has in fact the ability to see beyond the tip of its own nose and then spares efforts to prepare for the next decade which, of course, impacts its bottom line for this quarter. At the same time, Big Corp B doesn't see beyond the tip of its own nose and this quarter's results look better than that from Big Corp A. Then, all of a sudden, everybody, *you included*, sell their stocks from Big Corp A to buy stocks from Big Corp B which makes Big Corp B even bigger while Big Corp A is not a big corp any more.
Which, ironically, makes Big Corp B to be the one with the ability to see beyond the tip of its own nose because since Big Corp A is a big corp no more, its investment for the future is now moot and shouldn't have been done to start with.
That's exactly why you see from time to time companies exploding basically out of nothing (say Google or Facebook) despite of the fact that other Big Corps near the business niche have much more than enough muscle to crunch them -their stockholders wouldn't allow for the investment. And that's exactly way you see those Big Corps investing much more in rising the bars for new entries with lawyers, IP, patents and lobying legislators than in real R&D.
I have seen a lot of examples where the problem was not lack of documentation but rather that the available documentation documented the obvious things (this is a piece of paper) instead of this is the formula for turning garbage to gold. It is very important to document what the code accomplishes instead of documenting each line of code.
Good
document once a purchase order has been approved, prevent changes to the line items
Bad
instead of if the PO status = 'APPR' and the item number or price is changed, raise an exception.