How Do You Document Technical Procedures?

ChadDa3mon writes "I work for a large MSSP type operation and we deal with a plethora of vendors, versions, and .... skill sets. We're facing a critical problem as we grow when trying to deal with these varying degrees of technical competency. The end result is we're getting to the point where we have to document every procedure and process, no matter how mundane or 'common sense' it may seem." How, ChadDa3mon wants to know, can complex skills be documented to account for various users? Read on for more details of what he's seeking. "I've got a picture of how I'd like this to work in my head, but I can't find any software out there that seems to go along with it. I'm a big fan of keeping things simple, so I'd like to start with high level overviews. Each step in the process would be a general statement like 'Look for valid traffic on the monitoring interface'. For those who already know what 'valid traffic' means, it's easy to follow. However, if there was someone who was unsure what it meant, there would be a link they could click on that would pop open a new window (or something similar) explaining in detail what we're looking for and how to find it. It's my hope that using this process, people aren't just blindly running commands, but gaining an understanding into what that command is, and why we use it, what to be aware of, etc.

This seems like a job for a flow chart, but I don't like the setup of any of the ones I've used, such as Visio. It could also maybe fulfilled by a wiki, but there's so many out there I don't know where to start. I have to assume I'm not the only person who's facing a problem like this, so I'm hoping someone else out there can make some recommendations."

Comments

bad

[tritonman]

Documenting technical procedures is bad for job security.

Re:bad

[D Ninja]

No. No no no no. Wrong. [RED FLASHING LIGHTS]

If documenting your technical procedures puts your job at risk, then you aren't the right things, to be valuable enough to your company in the first place. I would even venture that you don't know what you're doing that you can't sustain your career on your skills alone and have to use methods like this to maintain your employment.

This idea of "write crappy code" "no commenting" "don't document" and other ideas is what causes half of the issues that I deal with on a day-to-day basis. I would keep around someone who does the top three far before I would keep around someone who tries to maintain their job by making them the only one who can perform their job.

(Alright...rant off. I get really upset by this type of thinking.)

Re:bad

[jetsci]

Its also bad when 4 years into your position you forget what service X does and you can't get it up and running after a major failure because you failed to document it.

Re:bad

[ChadDa3mon]

I think you misunderstood what I'm saying. I'm not documenting this for 'my' job, nor am I worried about my job security. I am trying to write documentation for 150 people around the world to follow so that we're all doing things the same way.

Re:bad

[JaredOfEuropa]

If documenting your technical procedures puts your job at risk, then you aren't the right things, to be valuable enough to your company in the first place.

Exactly. Also, do not forget that the best way to ensure you will never be promoted, is to make yourself indispensable at your current position. As all career counsellors will say: on any job, you should be training your replacement from day one.

Re:bad

[Samschnooks]

I would even venture that you don't know what you're doing that you can't sustain your career on your skills alone and have to use methods like this to maintain your employment.

Wait till:

1. Your company decides to send the work you're doing overseas.
2. You get over 40.
3. Your company decides that it is switching technology and is unwilling to train you.
4. Your company decides that it needs new college graduates because they are the ones that are up to date with current technology.
5. You burn out after having to work 60 hours a week for several years straight. Not necessarily because you have to, but because the bosses equate time in the office with amount of work and the fact that they always give unreasonable deadlines.

You will change your tune.

Other than that, I agree with you. Competing in the Global economy, especially in IT, is extremely difficult and competitive. I don't care what you do or how good you are, one day, you will lose out to cheaper folks overseas - exception: defence work. It is inevitable.

Re:bad

[Clover_Kicker]

True, but the easiest way to get a raise/promotion is to change companies.

Re:bad

[dintech]

Absolutely. Sometimes you can even come back to your old company a year later to the salary and position you should have had anyway.

Re:bad

[axafg00b]

True story - I start my new job as Network Team Lead with one cabling tech and one temporary consultant hired to fill in after the rest of the network team resigned. My documentation? Six large moving boxes full of c**p from the previous two leads and their managers. No online docu, no drawings, and a whole lot of head-scratching. This lead to us scrapping a major disaster recovery test because it was based on a network that had been dismantled two years prior, and no one had bothered to maintain the documentation. Yes, the company was in bad shape, but it was improving. By the time I left four years later (after a merger with a larger firm who did not require my services) we knew down to the specific patch cable where things were, and what processes we needed for hardware/software configuration.

Write it down!

Re:bad

[nine-times]

Also, do not forget that the best way to ensure you will never be promoted, is to make yourself indispensable at your current position.

This is an excellent point. I've found the best way to get promoted, in fact, can be to eliminate the need for your current position entirely. It's counter-intuitive, but it can work out

Imagine you're paying someone a full-time salary, and they take the initiative to figure out how organize everything such that, instead of working 40 hours a week, he's getting the same amount done only working 2 hours a week. And then he comes to you, tells you this.

Now, are you really going to say, "Your position isn't necessary anymore, so you're fired."? Or might you possibly think that he's a helpful guy and decide to put all his new free time to good use?

Re:bad

[DrLang21]

I used to work at a company where this was a standard career advancement strategy.

Re:bad

[DrLang21]

This is why you should always have your eye on a higher position and be formulating a plan to get there. This partially resolves several problems.

1. Your company is less likely to send managerial positions over seas.
2. At age 40, you should have aquired experience that makes your time more valuable than gold.
3. If you're in a managerial position, being fully trained on new technology is not necessary. You only need to know enough to properly manage your associates.
4. New college graduates do not have the experience needed to effectively and efficiently manage.
5. Ok, so you can't do too much about absurd deadline demands. This is not unique to IT by any means. But with higher levels of experience comes more bargaining power.

Re:bad

[berend botje]

Even better, after you're done optimizing your own job, start working with your boss on making him obsolete. He gets promoted, you get his old job. Which isn't all that much work anymore.

He then leverages you into a cushy advisor role, and you advice to make him a board member.

He makes you advisor to the board. You advice to make him VP.

Get the picture? Been there, done exactly that. We make a mean, lean, promoting machine! :-)

Re:bad

[smooth wombat]

When I need mod points, they are no where to be found. Your comments are exactly what I am trying to do where I work (government agency which makes it even more difficult).

Everything that needs to be documented is written down and saved in a specific location (Procedures) so anyone can go there and see the exact step-by-step procedures which need to be done to install and configure software, who to contact for X problem which is handled outside the agency, screenshots, and anything else that may come up. Especially those once-a-year happenings which no one can remember how to resolve from the previous time.

We still have a mainframe which, supposedly, is to be used to update patch panels, wiring schemes, etc, but which isn't remotely up-to-date, so instead, I've taken the initiative to create a spreadsheet with a tab for each switch documenting cable numbers and what is on them. Needless to say, this involves cleaning up the mess the guy who is as organized as a rabid wombat has created because he's too lazy to do things the right way and then swears off when you tell him he needs to update the file (Naw, I don't use the spreadsheet. [You don't use the mainframe file either moron]).

Your final words are the exact phrase I use when I tell people who have solved a problem. Write it down! Don't let us guess at the answer.

Yeah, I know, preaching to the choir. It's not as if employers want someone who can get the job done, correctly, on time and fully documentated so they know what's what.

Re:bad

[Samschnooks]

2. At age 40, you should have aquired experience that makes your time more valuable than gold.

Hahahaha. No. Most folks stay in tech and do nothing else. In the meantime, their management experience is nothing.

"Should". Who says? You? So what you're saying is, staying in tech is worthless? Experience in the tech world is meaningless. I have plenty of experience with distributed applications design, but now, who needs it when you can get an SOA app that does exactly what I used to develop - from scratch.

No sir. Experience only gets you so far in this industry.

You are replaceable - no exceptions.

Re:bad

[Cro Magnon]

Also, if nobody else can do your job, it can be a PITA scheduling vacations.

Re:bad

[racermd]

I've had a bit of experience with this, so I'll chime in...

The first thing to do is determine the precise audience for the documentation. Is it your peers for use in operations? Is it for managers or other process owners to keep track of processes and procedures? Once you have that figured out, you can determine the minimum level of knowledge/experience/etc that the audience is supposed to have (assuming the reader is qualified and competent in the position they hold). The purpose to this is to set boundaries on the level of detail you need to provide. Assuming no boundaries at all is a mistake and your documentation project will fail.

The next step is to outline the items that need covering. I stress the term outline. That outline should be written by those with as much experience/knowledge/etc as needed to give a high-level overview of the process. One outline should be made per process and should also be maintained in some sort of version control system.

THEN you begin writing the documentation. I typically write up step-by-step guides for people with a little bit of knowledge and absolutely no experience, so you might want to try starting with something similar and go from there. Remember to include as much or as little detail as required by the intended audience. Again, these documents should be written by those with enough knowledge and experience to perform the functions being documented and should also be maintained in a version control system. Additionally, these documents should be reviewed regularly for accuracy and relevancy, so some sort of time-out mechanism would be good, too. Lastly, someone needs to approve final drafts before they're added to the repository (which is a whole separate process that should be documented, giving rise to a chicken-and-egg problem).

As a final note, it's much more difficult to start writing something from scratch than it is to modify an existing document. Buckle down and start somewhere. First drafts are almost always going to suck (ask any professional or amateur author). Accept those facts and you'll feel much better.

Text document

[tsa]

I just use a text document with point-for-point descritions on how to follow the procedure. It's practical because you can print it and take it to the workfloor and cross the points you've finished. When you find out something new you can easily write it on the paper and add it in the computer file later on. Just make sure there is only one person maintaining the file, to avoid chaos and misunderstandings.

Re:Text document

[indytx]

I just use a text document with point-for-point descritions on how to follow the procedure. It's practical because you can print it and take it to the workfloor and cross the points you've finished. When you find out something new you can easily write it on the paper and add it in the computer file later on. Just make sure there is only one person maintaining the file, to avoid chaos and misunderstandings.

This actually works quite well in a number of settings. Checklists work. Years ago, when I was working through college at a warehouse, I had a job of some responsibility that usually involved me working late nights. Eventually, my boss quit and I was left in charge. To take a night off, and make sure the wheels didn't fall off, I created a text file with a checklist of the nightly procedures. Not only did it force me to think about everything I did, it also helped everyone know how much longer we would have to work each night before we could call it a night. You would be surprised at how much morale can improve if everyone has an idea of what's going on. Managers making decisions, seemingly arbitrarily, don't instill much confidence.

My advice would be to document your procedures, what you actually think needs to be done, and then take some time to distill them into a list. Then, following the list, does the procedure accomplish the task? If yes then move on to the next task.

Folding + Wiki might get you closer

[Red Storm]

I was reading through some of the Trac hacks for the wiki component and they had a folding plugin. If you create a table of steps you could then create a "fold" with greater detail should someone want to open it up and see it. The nice thing is you are not taken to a new page and you can continue to work and read the page you are already on. You can also imbed folds which can also allow a user to drill down to as much or as little detail as is needed or available.

All that is left now is to write enough information for the lowest common denominator.

Make it simple, or you won't do it...

[eth1]

Whatever you do, make sure the process for creating and updating the documents is simple, or it'll quickly become out of date and useless.

I would recommend NOT using any special software like a wiki for your primary documentation. It should be simple and printable, and not need a special server set up to access/use your docs in case of a disaster recovery situation.

Create a template document. If you use a word processor, set up standard formatting styles for sections, TOC, etc, so the docs are easy to create and navigate.

Put enough detail in the document so that you can hand it to a marketing droid and have them successfully complete the procedure. You'll be glad you did when you're stressed and under pressure from the Big Boss in an emergency (or if the whole IT department keels over from a tainted batch of Mt. Dew, and the only people left are completely unfamiliar with your environment).

Re:Make it simple, or you won't do it...

[merreborn]

I would recommend NOT using any special software like a wiki for your primary documentation. It should be simple and printable, and not need a special server set up to access/use your docs in case of a disaster recovery situation.

Getting your average wiki up and running from scratch should be pretty simple:

1. You'll need something like a LAMP stack. You've probably got at least one LAMP box left, even if your whole datacenter burnt to the ground. If push comes to shove, you can set up a package like MAMP or Apache2Triad on a desktop with zero-effort -- we're talking a half dozen mouse clicks through an installer, and you're done.
2. Many wikis (e.g., PMWiki) don't even require a database -- they use a file-based datastore. Unpack your gzip'd and tar'd backup of your wiki directory, and you're done
3. Okay, so maybe you chose a heavier-duty wiki, like MediaWiki. Ungzip your database backup, and dump it into mysql. Now you're really done

Version control, edit history, and ease of connecting documents make wikis vastly superior to a directory full of Microsoft Word documents.

Re:the easier to edit the better

[JCSoRocks]

A wiki is perfect for this. You can write simple step-by-step instructions for the more experienced techs. Within those you can easily integrate links explaining processes and terminology that aren't understood by the new guys. You also get the free benefit of searchable change tracking. You're not going to get that from a text file or a visio document.

Not only do I know what you need...

[gr3y]

but I can be bought.

You need a competent process engineer, and good ones are expensive. Bad ones are a dime a dozen, and will drive your costs up by insisting, for example, that every procedure be a documented procedure, that every process needs to be flowcharted, that there's a distinction between a process and the document describing the process, or that the fiction that is RACI is not reducible to a single directive: "accountable".

This is, as they say, job security for the process engineers because they're constantly chasing a moving target. Also, the instant employees realize or suspect that they're being made interchangeable by participating in any process engineering effort, they'll learn to conceal key details which will make all work to date useless.

Organizations that don't realize that have chosen the way of pain. Yours is probably one of them.

There are degrees for this

[ktappe]

Speaking as someone who has a degree in technical writing, your question is symptomatic of the entire industry's attitude toward (read: lack of respect for) technical writing. You hire people with degrees to run your networks and program your computers and execute your business processes, so why are you not hiring someone with a degree to do your technical writing? It's a specific skill that most people do not have and those who do it have had to study and learn. Realize that and you'll be better off. [/rant]

hire a Technical Writer

[spyrochaete]

I can't stress this enough. There are professionals who have post-graduate education and vast experience documenting and communicating technical procedures. Hire one of these people.

If you don't hire a technical writer you will face all kinds of problems. You'll have technical people with poor English skills writing incomplete directions because they make assumptions about what the reader knows. You'll have 50 manuals with 50 different writing styles. You'll have 10 instructions in one sentence with no commas. You'll have unfortunate typos and grammatical errors which change the meaning of the sentence.

Technical writers are both technical and writers. Hire one (and not the cheapest one you can find) to do the job right and chock the expense up to preventive maintenance. The alternatives are putting faith in poor documentation and, in the best case, spending needless cycles working out the kinks, or, in the worst case, spending needless money on a settlement to your injured customers because your staff were improperly trained.

Re:hire a Technical Writer

[Todd Knarr]

You'll never succeed at this. The Army has a test for officers in one of their advanced programs. The officer is given a mission and has to write out a set of orders for that mission. Those orders are then given to a unit to carry out, with one addition: the soldiers are to do their best to fail to carry out the assigned mission without disobeying those orders (technicalities are perfectly fine as long as no order is actually disobeyed). If the soldiers manage to fail to carry out the mission, the officer fails the test.

I believe this test has a completely unblemished record: no officer has ever managed to pass it.

Sometimes the fault does lie with the person carrying out the instructions. Every set of instructions assumes some minimum level of skill and/or intelligence from the person carrying them out. If you don't think so, remember that even a character-by-character walk-through of logging on assumes that the person knows to turn the computer on first. The instructions on how to turn it on assume that the computer's plugged in or the person knows where the cord and plug are and how to plug them in, and that either the instructions are written for one particular revision of a particular model of computer or the person knows how to find the power switch on an unfamiliar computer. I happen to have personal experience with just how bad it can get. Helpdesk call from one of our midwest truck stops, reporting that their pumps weren't working. It finally got escalated to me as the developer who wrote the pump-control software, since nobody on helpdesk could figure out why the pumps weren't responding. I went through a few basic checks, then told the manager to go out to the pump island, gave him a walk-through of the internal self-test of the pump, and asked him to run the self-test and tell me what codes it reported. His response: "I can't go out there. A tornado came through and tore the islands up, and the main electrical line's laying on the ground out there sparking.". Turns out that, besides the pumps being physically damaged, the power was completely down and the computers were running on UPS batteries. One would think that any reasonable person would've connected a tornado tearing the pumps half off the islands with the pumps suddenly failing to work correctly, but apparently not.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Asked to document how I Program

[Direwolf20]

I started doing a little software development for my company (when I was hired to do help desk), and I was once asked to write a procedure for that. How do you write code. I replied -- Step 1) Go to college for 4 years and get a computer science degree.

Postits

[12357bd]

Postits Everywhere.

Re:the easier to edit the better

[skeptical_monster]

We are using mediawiki for this; it seems to work great, eventually someone catches stuff that has gotten outdated, and it is pretty easy to edit and track changes. The real problem is that search in mediawiki is just dreadful; the index they use is weird, word highlighting is weird, and search is very important for this kind of application of a wiki. So we have some pressure to move to something else. Personally, I'm looking for a wiki with decent search. Wikis are the perfect medium for internal documentation, you need everyone involved to braindump all the time, and it needs to be editable by others. In other words, it needs to be admin-free and grow organically.

Re:And When Technical Procedures==Scientific Metho

[billcopc]

I worked very briefly at Dell, doing corporate tech support (hardware), and well not to brag but I was still #1 in the stats ranking a month after I quit until my averages rolled off... anyway I spent much of my off-call time trying to figure out a way to condense my smarts and experience into easy-to-follow instructions for my neighbours, who were often struggling with what I considered very basic problems.

To put things into perspective, my average call time was 5 minutes. That includes the occasional hour-long clusterfuck, which means the great majority of my calls were under 5 minutes. The top 10 techs had averages in the 15-20 minute range, and most everyone else was 45 minutes and up. Well now what was I doing differently, besides being the guy with the most hands-on PC experience ? I was being lazy, that's what! And I was committed to sharing my lazy ways with the rest of the crew.

During the exercise. I learned a few things:

1. It is damned hard to put into words things that are trivial (to you)

2. Logic, much like common sense, is a rarity these days

3. Most people fail to factor in the true cost of support time

So for #1, I had a non-guru friend take instruction, helping me dumb things down and bridge the gaps until we both agreed he had mastered the situation. He would express his frustration at my poor instructions, and ask a zillion questions until I had a grasp of my own internal thought process, of things that I did automagically.

For #2, I ran through a large number of scenarios and wrote down my own inner thoughts at each decision point. At the end, I trimmed these down to a rather large, multiple-entry flowchart. The neat thing is it covered both specific problems like "my hard drive is dead" and fuzzy symptoms like "my screen is blank". You would start at the edge, identifying the main symptoms and then you followed the paths until they crossed. At the very middle was the final solution "Replace entire system", the catch-all in case no other endpoint was satisfactory.

For #3, I dug up details on the approximate cost of support time. This included salary, utilities, and amortized equipment cost. Then I made a list of common support resolutions and their total cost (parts, labour, shipping). This allowed me to show how a long call can actually be more costly than replacing the computer in its entirety, especially if the extensive troubleshooting still leads to component replacement.

So in the end, we had verbose instructions, a troubleshooting cheat-sheet, and a cost guide. It may have enabled incompetent techs to perform tolerably, but more importantly it gave everyone the tools to learn very quickly. After a week or two with these troubleshooting aids, many people had the common issues memorized and had developed strategies of their own.

That's what I consider a successful transfer of knowledge. Knowledge is much more than just static information, it's also the process to create more information.

Re:What is this document word you speak of?

[jbengt]

Being a mechanical engineer in the construction industry (what's left of it, anyway) I would be willing to bet that the way builders build buildings is not much better than the way programmers write programs.

Code plan review and building inspections largely consist of filling out forms, having an architect/engineer/contractor's signature in the appropriate places, adding statements that some particular part of the code is being followed, along with a generous helping of checks to see that more or less arbitrary rules are enforced, and the occasional check to see that the numbers add up. Of course, structural designs for high rises may get a more useful review compared to architectural review of a one-story strip mall using typical designs.

While building codes are no subsitute for good design, they're (bad analogy coming) like the documentation for a programming language - if you don't follow them, you won't be able to build your project.

One final point, before I get too cynical, poor documentation is not just found in code, one of the most common weaknesses of blueprints is poor documentation.

Can't be done

[KGBear]

Not the way you're thinking. Where I work we use a wiki (mediawiki) and it works as well as any system I've seen, probably better than most. Now for the "can't be done" part:

Tasks like building a server, installing some software, connecting cables, etc., can and should be documented. Troubleshooting however cannot. Troubleshooting takes profound understanding of all the parts surrounding an issue and how they inter-operate; the ability to think long, hard and meaningfully; the ability to devise experiments, execute them, collect and analyze data and arrive at meaningful conclusions based on all that -- the scientific method, in other words. This is the kind of job that humans do better than machines and it is proverbially difficult to even understand, let alone document -- witness the "expert systems" of 20 years ago.

The big problem is that all non-technically-inclined people, including most people in management positions, think what we do is something mechanical akin to what plumbers do -- no offense to plumbers, who do things I can't do. From a user's point of view, "I can't print my e-mail message" is not much different from "I can't wash my dishes". Locating a clogged pipe and clearing it does require some technical skills and can certainly be documented. The e-mail printing problem however is several orders of magnitude more complex than that and, although it's probably possible to write a flowchart to document any possible thing that could be wrong in this scenario, it would be ludicrously huge, silly really. And that's talking only about day-to-day. If I'm really good at what I do I will design and implement systems so that errors of that sort are minimized. In other words, we're being asked to do the job of civil or hydraulic engineers on a plumber's salary.

There, of course, is the crux of the matter. Microsoft in particular, but the whole industry actually, are guilty of selling to managers the idea that this job can be done by drones, at all levels. Management is happy because drones don't cost much. Then, of course, drones are all they can get for what they offer.

What you want is knowledge, knowledge that can be provided by some, but by no means all, not even most, of the people who work for you. And you're tired of seeing those few good ones go away, leaving mostly drones behind. The only advice I can give you is: do document what can be documented. Make it a priority and make everybody responsible and accountable for documentation. But do acknowledge the really knowledgeable people in your team, treat them and pay them just as you would any other kind of expert. I know you're probably not in a position of paying your tech guru at orthodontist levels, but try to treat her at least as well as you do accountants.

Hey, nice reality check.

[Anonymous Coward]

You saved me some typing!

The older they were, the better they were built.

This is because of the building code.

Today, builders build to code, and the code specifies bare minimum safe method. As new techniques are developed to cut costs, the code is modified to allow cheaper, less redundant methods.

Before the code, builders built in competition with each other. They tried to build better houses, so they would have better reputation, so they could charge more.

But now, they just build to code. Full stop.

There's a place near me in PA that has no building code. All the buildings there are fundamentally stronger and better, no lie.

I try to apply this to programming. If a programmer doesn't write code good enough to stand without external documentation, I find another programmer.

Re:Hey, nice reality check.

[quanticle]

Yes, private homes may have been better built before building codes, because the occupants had a direct interest in making the home safe. However, commercial buildings (like the Triangle Shirtwaist Factory) are not subject to the same restrictions. There are numerous examples (U.S. in the 1800s, third world countries today) where the lack of adequate building codes allows builders to get away with murder (quite literally, since their buildings often kill people when they collapse).

Yes, I agree that the phenomenon of building "to code and not a whit more" isn't ideal. However, I'd argue that its better than the preceding practice of building as cheaply as possible, safety of the occupants be damned.