Slashdot Mirror
Institutional Memory and Reverse Smuggling
An anonymous reader writes "Anyone who's worked in a large engineering firm is familiar with institutional amnesia. Things get built, and then forgotten. Documentation is supposed to help, but rots, is lost, or uses obsolete methods and notation nobody understands anymore. I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works. I even have some personal copies of documents they seem to have lost, which I have to awkwardly smuggle back in. I don't find these kinds of experience written about often, but I'm convinced they're more common than you'd think."
A very relevant area where this problem can be readily seen is computer data formats.
But the NDAs keep me and everyone else involved from talking about it. :P
A unnamed corp where I use to work would hire retired employees for contract work at high per hour wages for months due to issues like this.
I have personally witnessed this happen with my former employer. Higher ups seemed to think that there was some benefit to the company "memory" or "team experience" if we almost brought a product to market. People would argue that at least we now had the experience to do this again in the future if we had to. Totally wrong. Teams were broken up and key players re-assigned. Some people left the company or moved to divisions physically remote from their former bosses. Any reports written could not be used to replicate anything useful and, to be honest, who would write a report explaining the history of an unsuccessful product? It never happens. Real knowledge is lost in a barrage of wasteful spending.
Sorry, but gray text on gray background is making my eyes bleed.
Somebody writes the code, doesn't bother to comment it at all and then you come in years after the fact. You look at the code and wonder, "why did he do it that way instead of this way?" Then the big gotcha, you think I could ask him but he left the company 5 years ago(At this point slap your forehead and hope you don't break anything working on the code.)
Did you know 80 to 90% of the moderators on slashdot wouldn't recognize a troll even if one dragged them under a bridge.
I design automated testing systems. I find myself in the position of re-solving problems that were solved as much as 10 years ago in prior automated testing systems. There's an odd tendency of people to simply passively accept whatever the manufacturer of the new system gives us as somehow "better" or more advanced that what we had, even when the new solution is an obvious fail, or (ahem) undeveloped. So I fix things to make up for the shortcomings of the new software, more or less the same way I fixed them 7 to 10 years ago. Things start working again. Everybody is happy.
I seem to have discovered job security by re-inventing the same thing every 7 years or so in a slightly different form using a different programming language. Of course, that sort of describes the entire computer industry, more or less.
Please do not read this sig. Thank you.
Why bother trying to smuggle them back in. Tell them you'll work on figuring out the bits you can't remember and will write a new manual. You get to charge them for time you don't need to spend and avoid getting caught.
One of the courses of my university study was called 'Software Evolution' in which tutors showcased all kinds of problems with legacy software. Software using languages or platforms no one knows about anymore or lost source code of mission critical systems. Especially older banks and insurance companies suffer from this problem, often choosing the 'wrap around the bit that we know that works' instead of a complete overhaul of a system no one really knows what it does but does it really well. I fear everything we build today will cause the same problems, but worse. Perhaps the old idea that readable code should help people understand its function saves us from this scenario...?
More like this, please, and less about the Apple/Andoid/MS/Samsung/**AA suit-of-the-minute. I know, I know, flamewars == pageviews == Step 3, but you've occasionally gotta throw something out there for us old-timers.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
I currently work for a company that has instituted an incredibly restrictive development methodology they bought from a big consulting firm. It requires multiple forms be filled out for every program written, every requirement for every program written, every request for every change to every program or application written. All of these reviewed by coworkers who are not about to alienate team members, and reviewed by lower management who want to look good to upper management by having everything go smoothly.
These documents are then stored on the LAN. To never, or rarely ever, be read again.
The one thing it does not enforce or require, is meaningful documentation in the code itself.
It doubles, or more, the time it takes to do everything. But it does nothing to stop the mistakes any better than the procedures that went before. If anything, it finds less errors, because we were not given more time to do this double or more amount of work. So time is so compressed, we do not have time to do anything other than get it working and get it in.
One thing it does do very well, is prevent problems from getting fixed. The only people that will start a change effort are those that notice a problem and are affected by it enough to have it cause them problems. Otherwise no one wants to go through the bureaucracy to kick off any sort of change effort, which leaves a lot of ticking time-bombs in the infrastructure configurations, application designs and application code.
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Documentation located anywhere else will be lost, or obsolete many more times than not, before you ever really need it.
If anything, documentation in code should be reviewed by people with absolutely no connection to the application it is for. If it is good enough for them to figure out an understand what is being done, and more importantly why it is being done, only then is it worth anything more than the bytes is written with in storage.
I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works.
I certainly hope that you negotiated a much higher salary than before.
I have known people who were fired in a "chain saw" maneuver who proved to have core corporate competencies in their heads. They negotiated for a much higher rate the second time around...
I'd be very careful in this situation. Even though you are supposedly "doing the company a favor" by smuggling the documents back in, were you even legally allowed to take the documents in the first place? Most employers have contracts or rules that state you can't remove documents from the office or that when you leave the company you must return all property to the company. Furthermore, it is most likely that this documentation was property of the company all along and in that case did you break any laws by removing it from the company and keeping it? IANAL but this is a very tricky situation. You may be doing good for the company now but you are profiting off of documents that you do not own and may not even have a legal right to possess.
Personally, I'd stick with just what's in my head and not keep any documents, files, etc. from a previous employer. It's just asking for trouble.
Hi !
I used to make construction blueprints, a lot of them were for NY sites(company was very small - 20 people).
Once, we were forced to pull old drawings, back from 70's(done by the company I worked for). It took us a while to figure out wtf is going on
Losing information over the years is common but rehiring for consultation is fairly rare in comparison. The easier method rather then reverse smuggle the data in, since you have worked on the system before, simply sign a NDA with the company that all knowledge remains there and kept secret. Since you are not a true employee, companies require at least a contract of that type in order to feel safer for "outsiders" to work on sensitive data. Barring that solution, just work with the internal team. Rather then trying to get the old data official recognized, quietly give them your data so that they may create "new" documentation based off on the old to be stored (guaranteed to be alot easier when working with management).
If the history channel read this story, they would undoubtably conclude that the plant was built with the help of aliens.
Where I work, us, the IT teams that program and support the applications, often have to tell the employees how the business works. So many people retired in the 2000-2010 that a huge chunk of institutional memory walked out the door. The new folks doing the jobs often don't know how to handle the Once Every Five year situation, etc. I know the company keeps buying these Knowledge Sharing portals, but it seems more like oral history is the primary knowledge transfer.
The bitter lessons of a veteran coder: http://bitterprogrammer.blogspot.com
My situation is different, yet similar. I just have so much responsibility for so many things at work, I simply can't remember how everything works. At times it is frightening, especially since year after year there are more things happening and I get older, so it is already inherently more difficult.
I also find myself "protecting" my sanity by almost intentionally forgetting how some things operate, especially those things I don't like.
I think as a manager, one of the most important things to do is learn to effectively document everything. I always knew documentation was important, to the point that I have a log of everything I do at work, every day, that spans 23 years! But those are just "reminders", not proper documentation. It helps to jog memory or memorialize certain facts for reference. Nothing replaces *proper* documentation, though- the kind that is well thought-out, has great detail, and is organized properly. Further, it will have insights into defining the problem and the solution and WHY you selected certain paths and not others.
Proper documentation takes considerable time, and many people think it is a waste of time, especially when one is drowning in projects and deadlines. When you reach the point that you can't handle the load without continuing to document properly, you know you are in trouble. In the short-term, you can "win" by getting more things done. But in the LONG TERM, you are basically screwed, because you will spend a tremendous amount of wasted time later trying to relearn what you did and why. Or worse, trying to defend your decisions and can't because you can't find the information, it doesn't make sense, or it just doesn't exist.
The bean-counters and mindless penny pinchers who run most corporations like to keep everything secret and proprietary without fully appreciating the costs associated with maintaining that level of secrecy. The solution is simple, only keep something secret if you've first: taken the effort to understand what it will cost to do so and second: taken the effort to make sure the secret won't actually be forgotten. In short, keep fewer secrets, and keep them a lot better.
i did so slavishly at first but then my boss pointed out that comments are rarely updated with the actual code. a contrived example:
int foo() // always returns 4
{
return 5;
}
now I have a coding style that the simplest fool can understand. that's quite a different thing than the arrogant coders who feel that their colleagues are incompetent if they cannot follow uncommented code. if someone cannot follow my code then I feel I have done something wrong, not them.
Request your free CD of my piano music.
Primo Levi, "Chrome" in The Periodical Table describes a similar problem with a paint formula at his factory in the 1950s. Evidently, during the war, they had a QC problem, included some chemical to correct for that, and the formula became a cargo cult, long after anyone knew what it was for. Someone changes some other factor, and their batch of paint gets the consistency of liver. So our hero had to reverse-engineer the trade secrets of a previous generation, back when he was working in the lab at Auschwitz.
Until you write it in cobol.
Deleted
if your own employees do the work your company retains the knowledge of how to get it done.
if you outsource you never learn how to do your own work, instead you finance your outsourcing firms efforts to learn how to operate your business. not only do you not gain that institutional memory but that outsourcing firm can now perform that same work for your competitors.
Request your free CD of my piano music.
Now let's try something a little closer to Slashdot...
Remember that circuit/macro/chip you worked on long, long ago, back when dimensions were reasonably measured in fractional microns? What schematic editor did you use? There is some chance that it was Cadence Composer, "the standard of the industry." What are your chances of reading that schematic today? You may have saved the data, but did you save an installation of the tool? Will it talk to the new license server? From what I've seen, any new version will "read and convert" the previous version. But how many people have time to go back and convert all of your previous designs to the latest'n'greatest design database? More likely you kept an installation of the old tool around, "for legacy purposes." When was the last time you tried to actually run that old tool?
Bringing it back to Slashdot again, I'll have to say that we don't often have to go back to old designs for any sort of serviceability purposes, generally one or two generations back is sufficient for that. But LEGAL purposes... You know, the Troll that is suing for something you know you did 15-20 years ago, if you can just find a date-stamped schematic....
Once upon a time, I gave an internal technical talk on schematic entry. I used as my example the ultimate schematic entry tool. It made semi-permanent archives, maintained a constant, readable database, etc, etc, etc. About the only problem was that it didn't directly interface with any of the downstream tools, like simulation or extraction. I actually used this tool when I first got into the design business. It was, of course, pencil and paper.
Helloooooooo, RUP
And a new dark ages are born.
Oh enough hyperbole. Wouldn't worry about it, the companies in question will collapse and be replaced by someone else if the demand is high enough. Or not if it isn't.
Deleted
I work in a library at a large organization and I see this every single day. The short answer to the problem is that it needs to be someone's responsibility to ensure access to KNOWLEDGE (NOT data, documents, etc, but knowledge) in the long term. I get requests from our employees every single day asking for information on project x that was shut down in the 90's. It is my job to track down any information that I can. Unfortunately, I have neither the directive or funding to start gathering and organizing the knowledge in such as way that it is useful a few decades from now. So instead I tell our employees "sorry, you will have to spend a few hundred thousand to recreate the information."
There are lots of people out there that can fix this type of problem, but they are never given the tools to do so.
now one does need to document the expected inputs and results of subroutines. better than commenting is to use assertions as well as unit tests. so for that contrived example, what I would actually do:
int foo()
{
int result = 5;
assert( 4==result);
return result;
}
if you don't maintain the assertions and unit tests with the payload code you'll find out about it right away.
Request your free CD of my piano music.
This guy worked at a company for 30 years. He retired and his company called him back in when an important machine started to break down. He placed an X where they had to fix the machine and then left. He sent the company a bill for $10,000. The company asked him to detail the bill. He sent: Placing a mark where to fix the machine $1. Knowing where to place the mark, $9,999.00. Old story and these days I make a few bucks because people improperly documented their work OR had no clue what they were doing. For example using a 1/4" industrial air disconnect to feed 2 air cylinders which hold a maximum of 10CF each at 90psi and then trying to fill them in about 10 seconds with 60psi of air from a 90psi source. Ain't guys with Phd's really smart :-) If it weren't for those really smart and educated people I might never have made any money.
NASA has lost ALL of it's Saturn V knowledge. The scientist have died, the documents have rotten and been lost. The parts were SOLD AS SCRAP. NASA is currently trying to buy up all the lost parts from scrap dealers. All this old knowledge is for the 'new' space program. Yes, we are indeed regressing.
Simply convert to COBOL. It's "Self Documenting" !!
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
If the organization is that bad that it loses its version control (or doesn't have one), you might consider not having the documentation as comments in the code, but rather as strings (that are not optimized away by the compiler). That way some poor bugger going through the executable years later can read them despite the complete lack of source code.
I'm kidding of course. ;)
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Deleting documentation is far more important than writing it. There is nothing worse than documentation that is clear, concise, convincing and that assumes something that is ever-so-subtly wrong due to the code base having been changed since the documentation was written. It's better to update the wrong documentation, but deleting it is far better than leaving it in in most cases. So no, people deleting comments are not necessarily fools, especially not if working to a deadline.
It costs money to maintain secrets. If there is no reason to keep it secret anymore it is cheaper not to guard it.
Request your free CD of my piano music.
Management don't care, why should you? It really can't be all that much of a problem. More a perception than a problem.
By all means go ahead and create/use a documentation system for your group but clearly there just isn't a requirement for anything more comprehensive.
Deleted
But have the firms outsourcing their document "management" ever given a thought to the firms handling the documents being copied into digital format?
What a wonderful source of industrial espionage especially when the documents cross nation-state borders for low cost labor in other Nations.
Something like this would never happen in private enterprise, where the invisible hand of the free market weeds out such inefficiencies.
Oh, wait ...
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
I have a situation a liitle more strange. A customer of a former employer (now twice aquired) has asked me to act as a customer advocate on a system I wrote as they have "limited" confidence in my former employer being able to maintain the system and the NDA/Non-Competes/IP Ownership prevent me working on the system for the customer.
We solved this problem year ago, trouble is we didn't document it properly, so we are coming back to it again.
I had a similar experience, from the other side of the spectrum. I was the summer intern doing a project that required digging through the paper archives to find some hardware data that was to be entered into a computer simulation.
It was nowhere to be found. What was documented for this perticular plant was little more than the type of equipment used. Not it's technical ratings or anything. Just what was there. And good luck finding any of that documentation only.
I also found full documentation for a plant that'd been shut down decades ago. Then demolished. And was now a blank wasteland. "In the hope they would be useful"
So there I was, scribbling down some notes off the PC screen by hand, when I reached for the keyboard and Ctrl-S'd.
Speaking from experience, corporations are unable to give themselves just one name. They have to change their names regularly (because it obviously makes things better, like go faster stripes) and more, they have to change the names of departments even more regularly (again because it improves everything). The result is that any documentation system which is created rapidly becomes fragmented, out of date and lost as the paths to the documents are changed to match the names.
The conclusion is that using names only makes things worse.
Deleted
I once worked for a digital aerial photography company. It had been acquired by another firm. The new owners desperately hoped that I'd kept a personal copy of all the source I'd written. "of course not" I replied.
While I was very good at backing up my code some cluebot had taped over it all with aerial photos.
Request your free CD of my piano music.
Institutional memory is the only asset that matters in the long run. It is the only asset that is not factored in on day to day efforts. The disconnect is both institutional and personal.
If one were to fix one thing in the USA it would be institutional memory. Globalization might be the second and Keynsianism the third.
JJ
That paper work can get lost and or end up in some 3rd party's hands.
or that the documenting part get's passed around and it's not done or is done poorly.
They could do one thing, just one thing, but do it well...
Now... Where have I heard that philosophy before?
Deleted
How about a built in documentation system within the program as both a user available help, and developer documentation?
At my institution we routinely buy back equipment from salvage yards it was sold to. But this actually makes sense. We clear out lots of old stuff rather than pack ratting it. It's a small price to pay for the convenience of getting stuff we need back.
Some drink at the fountain of knowledge. Others just gargle.
Years ago people solved this problem but they didn't document their solution well, so here we are again.
next up is the bobs.
Sure, end user documentation is one thing, but another reason your hire a technical writer is so they document how systems and processes work in easy-to-understand terms, so that when someone leaves the company, the loss of institutional memory is isn't so great.
Lawrence Person (lawrencepersonh@gmailh.com (remove all "h"s to mail)
http://www.lawrenceperson.com/
One of the uncomfortable truths (uncomfortable for MBA cost minimizers) is that know-how is between the ears. It is not in the manuals or specifications, which merely prove that their writers had the know-how. Even more important is the know-why, which is part of the institutional memory which also resides between the ears.
I have witnessed some "technology transfers" where a set of working products was transferred with documentation, design data, and much background information to a group in another region (mostly EU/US transfers). The first group is then wound down or redeployed. The new group does fine at first, until a new version of a product is needed. Then they always make mistakes the original designers would consider childish or stupid. They don't know the why behind design decisions, and don't know what to avoid. In my experience, it takes 5-10 years to build a decent product design group from scratch. On the other hand, an existing design group will sustain itself by mentoring and guidance of new inductees by experienced members.
The only successful technology transfer which I participated in was one where a few senior designers were transferred with the set of products they were responsible for. It was far from cheap in up-front costs, but it avoided the crises and financial disasters which afflicted the other transfers after a year or two.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
to do the wiring part so that the tech people have time to do the tech work.
As a contractor I was shown where the docs were on the LAN.
A frame relay link went down on the other side of the world causing trucks to begin backing up as bills of lading were being done by hand.
I was able to look at docs turn up a modem on one of the router interfaces to a backup link and call the carrier give the circuit ID and get the link back to full speed.
I an not a CCNA or any of that crap. Made me hero for a day.
Man I miss good documentation only found it that one time.
and then some messes up and then can have to get them back to fix it / rework it in to the new system.
mod up!
There's profit in confusion
If you can't be part of the solution there's money to be made in prolonging the problem
If the client doesn't grimace when you state your fees your were too low
Seriously, get them in writing to ask you to go through any material you may have that is relevant to the problem at hand and ask you to bring it in without any concern over previous agreements. I'd get a lawyer to draft up the email and agreement. Tell them, off the record, you may have old notes that could help but want to avoid any problems.
I'm a consultant - I convert gibberish into cash-flow.
That same company sent me to the University library to learn all I could about Gravity Meters, so we could bid on a government contract to find a bunch of buried waste tanks from a World War I Nitroglycerine factory.
The land where that factory once sat was scheduled to be transformed into a public park when some old codger died. Among his personal effects were the plans for the plant showing the waste tanks, but no one could figure out where on that parcel the plant had actually been located. The chances were pretty good that one of those waste tanks was going to detonate someday.
It happened that there was a great deal of published research on gravity meters. They are just very sensitive Mass On A Spring scales. This was in 1988; at the time they could detect a deviation of one part in ten to the minus fifth from the Earth's average gravity.
Most of the work had been done by some US Army officer who looked into using them to detect tunnels used to smuggle spies from North to South Korea. The effectiveness of the meters depended directly on the solid angle occupied by the void in the ground you were trying to detect. One could detect small holes near the surface, while larger holes could still be detected even when deeper.
While I concluded that we could easily find all the waste tanks, my company didn't win the contract. I don't know whether they were ever found.
This problem is so common in the defense/aerospace industry that there is an institutionalized response to it.
I could tell lots of stories of engineers I've meet that were in their late 40's early 50's that were being paid to sit around and play poker at work because the contracts required that knowledge of the system be maintained,
And back in 'ye olden days" at the start of my career, when a old engineer that was responsible for some important project would retire, he would "bequeath" his stash of personal documentation to the person that he knew was next in line. That used to be the one and only attempt at institutional memory. But this was back when there was actual paper documentation. I won't be able to this when I retire because their isn't any paper documentation. Hell, I can't even do my day-to-day work if the network goes down and I can't get access to the info I need.
I worked for an organization from day 1 and was involved in every project we ever worked on. After several years there, that organization wound up merging with another organization and I decided it was time for me to move on.
It's been almost a year since I left, and every day I get 2-3 emails asking me for information, and as a way for them to not feel embarrassed for asking, they actually created a 1/10th time paid position for me and the job title is, literally, "Institutional Memory."
I go in for a couple of hours every few weeks and do a data dump with one team or another, and they're slowly starting to amass documentation as to what was done and why it was done that way.
At my new job the first thing I did when I came onboard was to set us up with an internal wiki for every single project we have and everyone puts notes on the projects, why they're doing things the way they are, etc. on that. It's a good way to have as much of a paper trail as possible.
Since I can't tell them apart, I treat all ACs as the same person.
One of the things they do very well is... no, not designing chips, they suck at that, but duplicating fab plants. Down to getting everyone that's gonna work in there up to speed before the plant is finished.
The key is of course not to just build a plant. It's to build at least two plants, at least one in reality and one in the documentation. Fred Brooks (yes, exactly in his book _the mythical man-month_) explains how this works for software and computer hardware. It won't be different for processing plants, except of course that your production numbers are lower. But even with just one physical plant, you need full documentation on how to replicate it, with every task and every job described. And then, of course, you update the documentation along with any updates to the plant, bring it back up to the level where you can xerox the paper into reality again.
The test is exactly that: Can we give this to another engineer and have him duplicate the thing from that? Yes, most software documentation fails that test and is utterly useless as api docs or user manual to boot-- as if those code monkeys don't even read what they wrote, the illiterate bums. And yes, you probably should check every five or ten years if it's still understandable. If that's expensive, well, you get to pay more thirty years down the line. Or you can start from scratch, if you prefer.
Notwithstanding that this documentation archaeology an interesting excercise, but as TFA illustrates, it makes for rather schizophrenic situations, is way out of the usual corporate flow of how things normally go, cost a bundle, takes a lot of time, and is fraught with uncertainty. Care and feeding of the corporate library is future-proofing the IP assets, far more so than filling a war-chest with patents. It's about as sexy as caring for the backup tapes, but you have those too for a very good reason.
Accounting, for instance, has been done the same way for almost 500 years, yet there are thousands of accounting packages available, written in dozens of different languages, even to run on the same hardware. Some programs (Quickbooks, for instance) have been "improved" beyond comprehension while still preserving their deficiencies and shortcomings.
Once a problem is solved in programming, it should never have to be solved again. Most of the underlying code in computer systems is based on programs written back in the early days of digital computers. By adding the basic "components" together, more complex systems problems can be solved at a higher scale of interaction and complexity. So, scaling can be a problem. And, how the heck are we supposed to know where to find the original solution to the problem? (Note: There have been attempts to "hash" the common solutions and create unique, searchable identifiers, but I don't know what ever happened to those experiments. The ACM maintains and makes available a "library" of fundamental algorithims, but it is getting unmanageable.).
Now the person coming in to review or improve legacy code has a "complexity" obstacle that prevents him from easily determining the purpose and scope of the code.
In the digital world, programs work on data. Every digital program relies on sequence, alternation, and repetition. Many researchers (Djikstra, Jacopini, Martin and more) have shown that these are directly comparable to concepts in mathematical logic (sequential proof, decision rule, and quantification) and have demonstrated ways to "prove" programs correct. In today's world, we have the power to actually analyze programs, derive the structures, and prove the validity of the results at a meta-level (giving a nod to Goedel on the way). A meta system of this kind should discover what a program or system does, whether it does it correctly, and whether it does ONLY what it is supposed to. Now the problem becomes one of making the results understandable. The various packages that analyze source code and produce UML documentation are a very good start in that direction. I am concerned, because, (as recent demonstrations of Stuxnet and Duqu show) the interaction of flawed systems can have accumulating ill effects on those of us who have to depend more and more on computer systems to control our cars, our environment and the tools we use in our lives.
Not every industrial process is as easy to analyze as software. And the complexity of modern industrial processes makes it even harder to understand once the process has been analyzed. Much of industrial behavior is analog; much of it is probabilistic; and much of it depends on uncontrollable inputs such as human behavior, and human mis-interpretation of the situation. If I was part of the investigating team trying to preserve lost company know-how, I'd probably go back to anything resembling requirements analysis, and work forward from there.
"The mind works quicker than you think!"
The morons responsible for the business push from the very beginning are the ones responsible for not giving a rat's ass afterward as long as the widget works. If it works, they continue to perpetuate this very issue with the next widget, otherwise, you wind up with more job-cuts, terminations, etc.
...is a knowledge manager.
Sadly, they do not exist in most organizations and when they do they report to the wrong people (it should be something like CIO->KM->IT director). The job of the knowledge manager is to ensure that the necessary knowledge is kept around as long as needed.
Why is this needed? Let me give an example: In the late 90's most of our regional offices were closed. Staff were given one week to pack all the company records up, organize the contents of the servers (and burn to CD), and leave. Everything was then piled together in a warehouse for the next 15 years. During that time all the stuff from the different offices jumbled together, discs were destroyed by the climate, and the environment severely degraded much of the paper. Even now we are just starting to go back through all of that information. The cost to the organization to recreate that information has already reached the millions and that does not include the lost business opportunities.
People ask us "why did IT/legal/someone take care of this at the time" and the answer is that it is NOT their jobs. Their jobs are much more narrow in scope, and frankly, would they even know in time to do anything about it?
I would like to think that one day large companies will realize this and start implementing knowledge management practices, but considering the millions I have seen the company I work at waste I try not to get my hopes up.
Everyone knows that documentation is important, just like everyone knows that regular backups, writing highly readable code, checking tools into version control, and many other things are important. It's not exactly a revelation when these are proselytized in fashionable "best practices" books like "Code Complete" and "The Pragmatic Programmer".
The problem is that that's not how people get rewarded in an organization, particularly larger organizations with many levels (let's say six or more) between the CEO and individual contributor. In fact, there is a disincentive to do many of these things because they would make it easier for senior management to replace the workers who did them.
Many companies have a tiny number of people who manage things, and then a sea of consultants who do things. Often the management of the consultant groups are consultants themselves.And even the managers of the managers are often consultants. The result? Corporate Memento syndrome. No long term memories get formed because the people who do the remembering are gone. Stuff is rar'd or stuf'd or tar balled into a series of docs that sit on a server than is run by an bunch of consultants in IT who have no idea what is on the drive.
Then the drive either fails or is dumped into a bigger storage drive and labeled in a folder "management data 2009" or whatever, and there it rots because no one has any idea what's inside, and the people with access are consultants who don't/can't know.
and why? Because it's cheaper to hire a river of consultants for 6 months than it is to hire people and keep them around for years. The result? Permanent amnesia. All of it will be lost, and no one will care. the essence of 21st century culture - digital business - will be erased and forgotten. Forever. Some would say "Good riddance", but I'm enough of an armchair anthropologist to feel the loss.
Shoes for Industry. Shoes for the Dead.
An architecture firm I contract for was trying to submit a handful of their projects for energy efficiency tax credits. This involved assembling a set of drawings and CAD files for a outside consultant to create energy models of the buildings in question. Architects are required to retain these drawings for about ten years for liability reasons, so this should be easy right...?
Haha, wrong! Sure, they retained the drawings... digitally. But nobody bothered to check that all of the cross-referenced CAD files got saved, or that the links inside of them them weren't broken when they got moved to near-line storage, or that consultant drawings were saved, or that Jimmy The Intern hadn't linked in a dozen files from his desktop for the sake of expediency and never got around to moving them to central storage before he went back to college, and then IS wiped his account. What should have been a quick dive into the archives turned into a three-month-long wild goose chase. Let's just say that I got really familiar with the large-format scanner and the AutoCAD Reference Manager tool.
My company doesn't understand the meaning of "end of life" and will for the appropriate service contract pretty much support our customers for ever....
15 year old ? No problem. Still running version 1.0 firmware - when you bring it in for service, and we have to completely rebuild with modern flash etc., we will rebuild version 1.0 of the firmware to work with whatever hardware.
No big deal right? Well, we already have maxed out our engineering resources (scheduled for new projects) that these request are like throwing a monkey wrench into the gears. This is all justified because some sales guy convinced management that the service contract is worth it. Keeps are parts people busy as well - trying to find suitable replacements for 10 year old hardware. 10 years old. That would be Pentium III/933 era. And that is for a mass produced consumer computer. Now, try finding flash or ram from that era.
I worked 30 years in astronaut training facility (full-fidelity simulators), and wrote many many documents on software that I wrote. I always kept my own digital copies, of course. Over the years, the contracts changed hands many times, and different document systems were implemented, and "all" documents were "always" converted from old to new. I was never able to later re-locate *any* document I had submitted to *any* of the document systems. So my copies of my documents were the only ones that actually existed that I knew of. This included meeting minutes, peer review notes, design and 'as-delivered' documents. So I think institutional amnesia is more the norm, and actual memory beyond 3-5 years is rare.
Pavlov wouldn't be so famous if he'd used a can opener instead of a bell.
Outside of the code, all documentation is worthless
Don't look at man pages much, I'll wager.
And then they wonder why people think that consultants are basically thieves with a title.
What makes you think consultants get a title?
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
I saw a company lose probably 60% of their IT knowledge base. They announced outsourcing six months in advance, and directed current employees to document their jobs so that offshore workers could take over. Employees instead used the time to find other jobs, as one would expect. Transition was a disaster, and a year later they're still discovering things that the company no longer knows how to do.
One of the mistakes made was to assume that everything the old team did could be described as a procedure that a junior offshore employee could follow. This is a fallacy at a basic level, and shows a basic misunderstanding of IT. Knowledge is more than just memorization of individual procedures, it's information about topology, architecture, the flow of information, how the business works, and where the knowledge points are in the organization. (Who knows what.) Degradation of that knowledge base began on the day outsourcing was announced, and by the time transition arrived, there wasn't much left.
The basic misunderstanding, I believe is thinking that IT is generic procedure-based stuff that anyone could do. There are cases where this is true, but if you are in the business of providing a web based business, your IT *is* your value add, and your success is probably based on in what way your team does things that *aren't* generic.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
"If you have a procedure with ten parameters, you probably missed some."
- Alan Perlis
Management don't care, why should you? It really can't be all that much of a problem. More a perception than a problem. By all means go ahead and create/use a documentation system for your group but clearly there just isn't a requirement for anything more comprehensive.
All too true. The sooner you learn not to care, the sooner you will stop feeling the pain. Surrender to mediocrity; obey all rules and your managers without question. Above all, do not let your work interfere with your life. You will be a happier person, and probably get promoted. Listen to me, oh young ones...heed the words of the Ancient One: Despair!
Once upon a time, I worked in a relatively small organization within a Very Large Company. The VLC was renowned for its inventiveness, until a new manager came who felt that inventing things was much too expensive; it would be more profitable to just to put the word "Invent" on the corporate logo, and then re-sell stuff from countries who make things cheaply. Because our corporate logo was on these re-branded products, we must obviously have invented them. Thus, marketing ceased to be a partner of R&D, and instead replaced it. Clever, eh?
One of my contributions to the VLC was a simple document management system (DMS). My boss mentioned that all the engineers in our lab kept complaining that "they can't find anything"—specifically, nobody could find old documentation for code that was written by vanished civilizations of programmers that had previously labored at the lab. In fact, it turned out that this was a general problem among the labs of the VLC, and that they all wished mightily for a solution. Apart from really ancient projects (from, say 10 years ago), it was pretty difficult to find documentation for even recent projects, as each lab had a different way of organizing its documentation, different document formats, and no channels for distributing the documentation to other groups who might have a legitimate interest. So I brought in an easy-to-use and easily maintained DMS, pitched it to the engineers at our lab, and got them to use it. My boss then made me contact a bunch of other labs, and offer them the use of the same DMS. A lot of their engineers liked it, and there was much happiness. I got lots of pats on the back for what was really a pretty simple idea, and my boss loved me for it. A year later, I was laid off.
As I was the only one maintaining this system, and no one ever asked me to train a replacement, I doubt that this DMS—and its contents—survived, save perhaps as digital ghosts on some unlabeled backup media in a faraway storage cave. In a sense, I'm one of the perpetrators of information decay; I caused all the knowledge deposited in "my" DMS to get black-holed. Come to think of it, this is probably not a problem; I'm pretty sure they've laid off all the engineers by now.
I understood very well that simply setting up a DMS was not the cure for the I-can't-find-anything problem. It was a beginning, at best. What was needed was a comittment by the organization as a whole to make sure the information continued to be available for as long as the company exists. Because engineering practices change, so do documentation practices; documentation technologies themselves change—not to mention physical storage media and data encoding. My DMS probably should have been replaced with something better by now (if it had continued in use), and that means someone would have had to plan the transition, and move the old docs to the new system. Maybe the best thing to do is print everything as hard copy from time to time, and hire a librarian to keep track of this backup.
Considering how much has been said and written (and blathered) about "knowledge management", it's amazing how little attention is ever given to the temporal aspect of institutional knowledge. Every engineering company should have an Office of Knowledge that has as part of its responsibility not
Great men are almost always bad men--Lord Acton's Corollary
Gosh, what a good idea...
I've worked for 6 companies over the last 30 years, and it's happened at every one. Just like layoffs. Companies spend millions developing processes or technology and even if it works well they eventually shitcan it and the employees for something new. I suspect a lot of it has to do with new management who want to make their mark before they move up the food chain.
I left a company 6 months ago and they are now rebuilding the system I build because the new guys can't figure out how it works. When I started there 2/12 years ago I rebuilt the current system because I couldn't figure out how it worked...
I write all my programs as a LISP interpreter and a shitload of string constants containing sexps to evaluate -- any fresh-from-college CS kid can implement a LISP interpreter (or reuse one he wrote in school and got a B-), and the text is self-explanatory -- plus it's great for portability, as only the small interpreter (and string syntax, which is automatable) is language-dependent.
Most companies and managers do not appreciate experience. Just left a company after 9 years. They lost ALOT of knowledge, for a few dollars of a pay raise I did not get and working conditions. Watched another company I was at that lost the documentation, trench for a while to find where the break in the conduit was, when it was just a turn at a junction box. In discussions with my current company about installer that left the company, and did an inadequate job, I want to fix, customer is already mad, I have the experience, but spent Friday doing nothing and probably Monday talking about it while the no actual work is getting done.
So YES it is RAMPANT!
Sure, it's great to help out the company and as engineers we like to just get things done. The problem is that the legal department doesn't give a fuck and would sooner sue you than allow anything to get done on time. Burning your personal copies of the documents might not help the project get done, but it will save your ass should anyone with malicious intent start asking why you have documents you're not allowed to have. You're a consultant so it's not like a project failure is going to cost you your job. I don't know how risky this is because I don't know the details of the situation, but there's no way I'd risk my neck, and career, over a company I left many years ago that wasn't competent enough to keep its own damn files.
-1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
I'm in the military and everyone who's ever been in the military knows they love to change everything. A new general or colonel will get a new job and has to make himself look good by improving everything, changing names of everything, restructuring, making lots of tours, etc. My job is highly technical and requires coordination with many different agencies and contractors. They don't worry about making the job easier or better, they only worry about aesthetics. They put holds on personnel so people are stuck in one place too long, then when the hold is lifted a sudden export of airmen occurs, along with all of those years of experience, to be replaced by new airmen that have no experience. The new airmen try to figure it out and fail, then ask the ones who have left and the names and processes don't line up anymore. Old ideas that were developed over years are gone and replaced with new ideas that are just as green as the people trying to develop them. People throw things away. People lose things. People get lazy and cut corners, because no one knows enough to realize why those corners were there. Over time things slowly get better and people gain experience, just as new supervision takes over and it starts all over again. I love being in the military, but I think I will love leaving it just as much.
There have been a lot of nice and informative comments to this point about the various facets and symptoms of the problem, but here is the problem itself: Corporate Attitudes.
1: Companies don't give you time to write peer-reviewed documentation because that doesn't contribute to Time To Market and overall productivity.
2: Companies seldom ask you to train your replacement in the I.T. world because they don't want you around once you realize that you're about to be laid off/fired. Who in I.T. hasn't had the experience yet of being marched off the property under the watchful eye of somebody (I've had it done where they guy actually carried a gun) within an hour of being told that you don't work here any longer. You find that you passwords were disabled during your 15 minute (at most) exit interview and given a box to clean out your work-area under guard.
3: Most importantly, the corporate belief is that nobody is indispensable, and they're willing to prove it with you no matter how much you know that no one else does because the more senior that you were, the less anyone else was watching over what you did in the first place.
In short, you're gone and they simply don't appreciate or value what they lost with you. And find this far more prevalent in I.T. than in most other areas.
The flip-side is that you've probably also been hired at least once to pick up and complete an undocumented project of someone else that they previously let go. Isn't that fun?
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Sounds like you work for a defense company. We have maybe a dozen people writing code yet over a hundred in the SW group.
I've seen it happen at a software company. They have a Knowledge Base that the technicians recorded every solution, the causes, and any fixes that we could find. There was a bit of duplication of entries due to some issues being described or even experienced improperly, not to mention the occasional insufficient or improper use of keywords and descriptions.
Some higher up got the bright idea of merging the duplicate documents to save storage space. That was fine. Then they decided to expand it to eliminate entries that hadn't been accessed in 6 months or longer. That was moronic. Sure enough, it wasn't long before it turned around and bit us in the backside. But management wouldn't budge. They refused to restore the old documents, even when it was on something we now needed, nor would they stop deleting stuff that hadn't been accessed for a while.
To add insult to injury, they would then use the count of how many documents you had in the KB against you if it wasn't high enough. Well, Soandso, you only have 120 documents, why should I listen to you instead of Otherguy, he's got 14657 documents and is in charge of the KB cleanup.
You see, the guys doing the KB cleanup and deduplication were going through all the documents and replacing the owner/creator names with their own so they could bloat their own credit, and management (at least the part I had to deal with) was too stupid to understand this. I have no idea how they rationalized that somebody who'd only worked for the company for less than 2 years could have written thousands of documents that were over 4 years old.
Yes, in a somewhat different form, this probably affects all companies in one way or another.
Maybe that B.A. in History isn't such a bad degree to have after all. Now all you need to do is convince corporations they need to have a history department to manage all of their documentation in such a way that it is discoverable and accessible.
One of my favorite lines is, "Your data is secure. We just can't access it." In other words, if you are going to use a particular medium as an archive, you need to ensure that the data can be read from that medium, in perpetuity. If the technology is in danger of becoming obsolete, the archived data must be moved forward to new media.
Of course, that isn't enough. The data must be indexed in such a way that the information you need can be found. Finding a needle in a haystack is easy with modern technology; finding a hand-drawn flow chart explaining how a particular "black box" in your manufacturing plant converts X to Y (and why it's painted blue) among hundreds of thousands of pages of poorly indexed documentation is exceedingly difficult.
Give me my freedom, and I'll take care of my own security, thank you.
Lost or misplaced institutional knowledge can be leverage for those who know where it was misfiled. Its what keeps some otherwise unemployable people in their jobs.
Have gnu, will travel.
I was hired to implement improvements and document them. I started writing about them, but my boss insisted that every procedure changed every time one particular application was run. He was an asshole, desperate to keep his job. I got no help from anyone further up. I put up with narcissism, micro-management, and idiocy for three years. Then, I helped clean up the office for about two months (cleaned up a lot of old stuff), and then left. They even ignored my very good advice about putting the system live to the internet. Less than two months after I left, they got hacked (and it wasn't even me). A few years later they were looking for someone to look after their system. Apparently they couldn't get anyone. No surprises why.
I recall interviewing long ago for a job maintaining a program with about 25,000 lines of uncommented code.
Fortunately, I turned out not to be the best candidate for the job.
Usually there's nothing but the highest level of design collateral left after the nerd doddles off to greener pastures after 6 months.
The problem is with the nerds, not management. It's an ongoing fallacy that management is all to blame, it's fun to sneer at the managers. But the real problem is the incompetent, arrogant nerd who can't document anything, or even explain what he's spending all those hours doing, who knows he's moving on at the end of the project anyway.
You have to whip, threaten, kick them to get anything down on paper. And when you ask them a simple question about how their shit works, they look at you like you're stupid and give you a dumb vague hand-waving answer. Assholes are completely up themselves and think the rest of the world is stooopid.
This reminds me of Pump 6, by Paolo Bacigalupi: "...it made me nervous thinking about all those maintenance warnings glowing down there in the dark: Mercury Extender Seal, Part #5970-34, Damaged, replace. Whatever the hell that meant."
...from memory. Just make sure to create new files and copy and past the information into it. Create the documents in new formats (i.e. Word 2010, PDF, etc.) Otherwise you could get screwed by the creation and modified dates or old formats.
I used to work for Cedars-Sinai hospital in Los Angeles in their IT department. They brought in a Document Management System, put people in charge who knew (and still know) little to nothing about document management to force everyone through the "ITG" process. ITG stands for nothing in particular - we've asked - and a department that started with 1 FTE and 1 contractor has grown to almost a dozen people.
Nothing anyone ever puts into this system is ever really pulled and used - the programmers just go back to the code. In order to "follow the process" of peer review, testing, etc., you'd never hit a due date. So, right after all the code goes live, the developers and analysts play "ITG Ping-Pong" to push all the right buttons and in all the right order to catch up the tickets. It's all bullshit, but it's the CTO's sacred cow. Talk bad about it and you get put on his Cone of Silence shit-list, never to be considered for any promotion or perk. Ever.
I remember putting all sorts of Easter Egg fillers - like quotes from Shakespeare, blog posts about my day, etc., in to the documentation for the 2 years I used it and never once was any of it found. Pretty sad, actually.
They tried to use their DMS as a catch-all of all the knowledge the developers had, and nobody has yet to use it in any meaningful form or fashion. I am pretty sure the data being shoveled into that train wreck is just the same duplicated BS processes again and again and again...
The Leaky Establishment, where the large engineering firm is the Atomic Weapons Establishment and the item to be smuggled back in is a physics package.
Worked for one company as the Engineering Manager and couldn't figure out why the General Manager knew so much about the old projects - he kept coming up with long lost engineering documents. When we moved to a new building, we algamated 5 offices and I came across an old filing cabinet tucked away in a store room. It ended up to be the long lost engineering documents that the General Manager was hiding from us. I moved it to the engineering dept and the General Manager never said "boo" about it.Go figure that one out!
In another company, I was the Sales Manager and got laid off. The president never even asked what was on the go and status of any projects. Then the president phones me up about 3 weeks after and asks for all my personal notes on the current sales and projects on the go. I told him, that I threw out all all of my personal records when I as let go because I don't work there any more. Was he pissed!
For my entire adult life I worked in the medical diagnostic device industry and somewhere in the late late 80's and electronic documentation & email really started to take over. Then following a series of lawsuits the corporate SOP began to change. We went from loose organization in directories to using versioning tools for documents. And we went from what was essentially unlimited email storage to smaller and smaller... eventually ending up in 2005 with mandated culling policies. (mostly as a proactive defensive legal strategy).
By my nature, I am digital packrat. I still have all the email I have ever received or sent, in curated archives. I still have all the documents I have created. I still have all the code I have ever written. I still have all the design docs I have ever created. And I still have the knowledge management system I created to curate all of that data.
So, my nature and corporate policy really began to conflict more and more strongly. For about 12 years I used my own hardware for backups with my management looking the other way. Eventually I was told the backup strategy had to go and to take all my stuff home. That was replaced by corporate supplied laptop which I routinely took home to backup.
I took early retirement in 2009 and in late 2010 was asked back to resolve a thorny problem with some of the in-house equipment I had a hand designing. The current site manager, who I have a lot of disagreements with but is a nice guy, assessed the parts of my personal archive that I brought in with me as "The largest and most frightening example of industrial espionage he had ever seen"... and wanted to buy it from me so he could destroy it.
Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.
comments that were formatted with aster ices in a certain specific way.
a colleague discovered that the cause of a certain memory leak was an allocation with the c++ new operator without a corresponding delete. after receiving her proud email announcing her discovery, I filed a bug against her specific fix, then blasted thebentire company with a short angry written lecture about the critical importance of using smart pointers for c++ memory management.
the company president ordered me to stop filing such bug reports. I resigned in protest not long after.
Trihedral Engineering makes Human Machine Interface Software Control And Data Acquisition products.
HMI SCADA software is some of the most human life critical software there is. our product was used to automate a pickup truck assembly plant in Kentucky. The Stuxnet worm that attacked the Iranian uranium centrifuges attacked the HMI SCADA product produced by our competitor Siemens.
If you don't use smart pointers in c++ your code won't be exception-safe. if you don't use smart pointers in the c++ HMI SCADA code used to drive a giant automotive assembly plant, you will drop a pickup truck on one of your plant's workers, making a widow of his wife instantly.
Request your free CD of my piano music.
I seem to remember Primo Levi wrote on this subject - not sure which of his books i've read, but something about a paint factory process springs to mind. Companies do loose information - it's entropy at work. It costs to keep information intact, and unfortunately there is much information and little indication of what may be important in the future.
to do the wiring part so that the tech people have time to do the tech work.
Technical writers to do the wiring. Aha! That explains the electrical problems I've been having.
Watch this Heartland Institute video
http://www.xkcd.com/984/
One other useful source of documentation is version control software. If used correctly that is(small commits that do one thing with a proper commit message). I recently joined a team working on a project that has been underway for quite a while and whenever I find myself wondering what a certain piece of code does or why it was written the answer is usually easy to obtain by going "svn blame [codefile]" then "svn log -r[version of last change]" as well as "svn diff -c [version of last change]" if I want to know what that commit did as a whole.
I used to work as a consultant for a very large, global computer manufacturer/consultancy as Solutions Architect. More than once I had to quickly find an available system to implement a customer's outsourcing solution as the normal lead time for a new system was 1 month+! (And before you start yelling "Virtual Servers", these customers required independent physical servers.)
Sometimes I had to go through a list of servers on a specific subnet, match that up with a list of servers from the routine network scans to see if all were accounted for, if not, I would try to find the server in the server room, get its ID from the chassis, go back to the inventory system, try to find the server there and hopefully an owner too.
Occasionally I could either not find the system in the inventory system or no owner was assigned. I would then have a private chat with the hardware people responsible for the area the servers were placed and suddenly the relevant servers would have an "unscheduled network outage", i.e. the operator would unplug the system and we would wait for somebody to start screaming. If that did not happen within a day or so, we would dump the system's data to the backup system and reassign the server. This time properly documented.
These systems almost always exist as a due diligence cover my ass exercise for upper management. Mistakes are inevitable as you very astutely point out, however you didn't document that it might be a possibility. Management is covered because they signed off on only what you documented. The mistake is yours and yours alone, you get marked down on your year end review, management is unscathed and everything continues as normal.
If people are going to think badly of you, you might as well be doing it while sitting on what used to their money.
Nobody likes a poor thief.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
1.) Additional project costs.
2.) Liability -- Proof of a faulty idea or process.
What company wants that?
At the company I work for there are two types of teams. The most common type is the lip-service team where a group of people are forced together and grudgingly share as little information as possible. Each person in this team has his or her area of expertise and think they will keep their jobs longer if they are the only ones to know what they know.
The other type of team, the 'real' team, is one where people actually share their information with co-workers. In this team everyone eventually knows (or has access to) everyone's information. This type of team ensures the information is never lost. As far as people pushing each other out of positions, truth be told, most people just don't want to do each other's work. It's a win-win for the company.
BTW, I have worked in a government position where the environment was so toxic to sharing information that anyone who trained anyone else on how to do their job would be 'layed off' within a month. Yeah, don't share the knowledge in that case.
My girlfriend was moved from another group into Cambridge Systems when UCCEL bought them and SKK to keep track of engineering documentation, publications and whatnot through the transition. Six months later CA bought UCCEL and laid off everyone over their target salary range, which included her and lotsa senior engineers. Two weeks later she was back as a contractor at +100% pay bump because they couldn't find their pooper with their fingers. Took about 6 months to transition it to Long Island. But since CA still sells and supports ACF2 they must have retained enough to make a go of it.
When I left GD many years ago, there was no formal document archiving process. I conscientiously (hadn’t yet lost my soul back then) organized documentation for all my projects, including custom analytic code, telling my boss and everyone who might do something with it where to find it. A few years later, they called me and expected me to have copies of those things they had lost, many despite having security classifications. (Some of what they asked me for was actually Top Secret!) They were astounded when I told them I hadn’t taken anything with me.
When I left LM a few years ago I was told to get with a clueless suit and go through my files giving him copies of what he wanted. What he wanted proved to be totally pointless dross. All else was to be wiped per corporate policies. I did as requested, not daring to interfere with whatever-the-Gods-that-be's clear plan to punish evildoers by granting their wishes.
One might note that many corporations have black-hole document retention policies for some documents. LM had a rigid (and enforced) corporate email destruction policy launched with great fanfare just after the Darleen Druyan scandal at Boeing erupted from retained emails. My guess is that tens of thousands of man-hours were lost because engineers had been using Outlook as a filing system.
in the title. That bothers me.