Slashdot Mirror
Documenting a Network?
Philip writes "Three years ago I was appointed as a network manager to a barely functioning MS-based network. Since then I've managed to get it up and running — even thriving — but have been guilty of being too busy with the doing of it to document the changes and systems that were put in place. Now as I look back, I'm worried that I am the only one who will ever know how this network works. If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run. Ultimately, this won't be a good reference for me if they are trying to work out technical details for years to come. It looks like I'm going to have to document the network with all sorts of details that outside consultants could understand too (no, I don't want to be the outside consultant), especially since it's likely that my replacement will have less technical expertise (read 'cheaper'). Are there any good templates out there for documenting networks? Is anyone who has done it before willing to share some experiences? What did you wish your predecessor had written down about a network that you inherited?"
The Passwords.
Use Post-its.
I Need someone to rebuild a Digitech Digital Delay pedal for me....for me...for me...for me.
That's completely redundant. If it's not the shitty network stack back in the day, it's the malware-ridden security nightmare of the present. Either way, there's no need to waste so many words.
Your successor will never find any documentation that you leave behind (or if you show it to them they won't bother with reading it) and by the time they notice it they'll have already screwed things up to the point where the documentation will be obsolete. This means you can save yourself the trouble of doing the documentation unless that documentation is going to make you more effective while you're there.
Remember RFC 873!
Sounds like a very easy way to over work and over stress your self, get some help one way or another. Summer is coming and I'm sure there are plenty of Comp Sci/Network Engineer/IT students that could of help. It may not be a bad idea if you make a plan of some kind before you go head in.
It's really not as complex of a network as you may think. Go ahead and quit - the sun will still rise and the email will still be delivered....
1. Simply begin documenting, its a work in progress..never finished. 2. Select a worthy staff member from your org, with the brain cells (and desire)to start learning networking, and begin to train him/her on what you are documenting. 2.a refrain from selecting the network thinks-he-knows-it-all type, instead pick anyone else with the skills listed in 2.
Professionalism be damned, we're in the middle of a recession and you're wanting to make yourself dispensable? Go ahead by all means, but pass your boss my number :)
A good tool like dia which can allow you to create a network diagram. When it comes to documenting a network, a picture can be worth a thousand words. Or you could also use MS Visio as it is, perish the thought, a good tool. A good, detailed diagram can come in very handy as a reference tool for your own use in case of a failure.
Basic network documentation:
I've found that starting out with the very basic physical layout and working your way up in complexity is greatly beneficial.
i.e. start out documenting network cable runs including cable type. follow it by switch layout. follow that by routers and vlan setups. follow that by the servers that provide basic network functionality(e.g. DHCP, etc...). If this is a windows network, that would likely mean detailing the domain controller setups. From their systematically document the systems in order of importance to the business, etc...
Also, visual diagrams are extremely helpful.
Short answer: don't worry about it too much. Put together enough that it looks like you've done something then go have a beer.
You could have the most amazing docs the world has ever known - with passwords and clear instructions - ad the odds are about 20% that the next guy will even read them.
The next guy will figure that he/she knows much more than you as evidenced by the fact that they are there and you are not. And, the cheaper they are (read: inexperienced) the more likely this is to be the case. When things go wrong, they will blame you anyway.
So document away, but for YOUR sake so that if/when you are called in after the new guy horkens everything, you can have an easy time putting it all back together. But don't wait for the call... people will put up with almost anything when pride is on the line.
And go have a beer.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
For me, visio's are great and everything, passwords too, but really the most valuable thing you can do is document single points of failure, outdated software/hardware, etc., license keys/expiration dates, cert expiration dates, personal support contacts you have and all vendor relationship details as well are essential. Do you use change control? If you do, go back and comment your changes, if not, do the best you can at explaining why things are the way they are. Get some open source software that is good at indexing data and create a searchable knowledge base from the information above. Don't concentrate on docs that can be found on the web at first because any admin worth their salt will know where to look for how to's, etc. Focus on the why's, the where's and disaster recovery.
My two cents...
1. Viseo overview of the network drawing with complex areas drawn out specific detailed viseo's (even a scanned sketch or paint drawing is better than nothing)
2. A spreadsheet with circuit ID's mapped to router and interfaces.
3. Document the trunk interfaces as well as the LAG's (Link Agrregation Groups, port channels, whatever you want to call it)
4. TACACS passwords / domain logins in a secure location (or radius or diameter or whatever you use)
5. Data center capacity as a function of 1. Rack Space, Cooling Capacity, Electrical Load.
6. Write brief knowledge articles describing any problem areas and explaining a history of anything you think would be hard to figure out easily. No need to go hog wild, just re-brand the RCA documentation you have. You do have Root Cause Analysis right?
7. Network protocol hierarchy map. Where are your major redistribution points, what is your routing strategy etc.
8. If you have a voice network document all your DID's, PRI trunks, Gatekeepers, Dial Plan, and any translations you use on h.323 gateways or how MGCP or SIP is configured. If you have a complex call center you should probably pay to have it professionally documented in down to the minute detail.
9. SSID's and BSSID's for any wireless you may have as well as passwords, 802.1x authentication methods, along with linking documentation.
10. Make the documentation part of your CMR process (Change Management Review) and incorporate it into the time allotted for a change.
I know these are just rough ideas and you should get many more ideas from all the smarter people on here than myself, but whatever advice you get I would say you would need to have the documentation update able via subversion, or some document control system and have some kind of review process for it, even if it means getting together over pizza with some of the other groups and asking them about their environment and getting pointers and possibly help on documenting it all. Documentation is a full time effort and IMHO there is no such thing as too much documentation. You would be surprised how good documentation can aid you in problem resolution down the road or aid vendor support in helping you resolve a major outage. The three basic principles of network care are document document document. :)
Cheers,
Anonymous Coward.
An overwiew map. Helps me figure out the first steps. Firewalls and routing hints are good to place on the map. Any other equipment is good too.
An IP plan detailing networks, their intended use, DHCP scoops, ant things with static IP on those networks.
Passwords to the equipment. And how to access the equipment if it is something unusual.
Two or three small lines detailing anything special with anything.
On the side, I manage a small network, and I've also wondered the same sort of thing: if someone else needed to find their way around, where would they start.
A Wiki makes for a really nice way to document things, not least because you can include all sorts of cross references. For example, a list of servers, with links to the services they provide - and a list of services, with links to the servers. But Wiki's normally run on servers, which leaves your successor with a chicken-and-egg problem.
A bit of random surfing turned up TiddlyWiki, which is a Wiki in a single HTML file. A really elegant bit of engineering, and very handy for self-contained documentation. Since the entire Wiki is just a single file, it's easy to protect. I wound up with two: one with "public" information describing the general architecture and one with private information (including passwords). The private one you can put on a USB-stick in a safe, hand to your boss, or whatever seems appropriate...
Enjoy life! This is not a dress rehearsal.
At my last few companies and my current one that I work out, one of the first things I do is setup an internal only Wiki server.
Not only does this let me document everything I can about the network but I also try an train my co-workers in using it to document information they feel is important for their job too.
The effectiveness though seems to be related to the level of computer literacy of the user, i.e. my last company the software developers went crazy with it and documented everything they could think of. But other than them or us in the I.T. dept, no one use would hardly touch it at all.
Either way it's still a simple method for your to store notes, diagrams or any information about your network in an easy to find single location that you feel would be important to the company should you leave for any reason.
The fact is, you're doing this because "Ultimately, this won't be a good reference for me..."
As you leave, say "The network started bad, and will always be bad. I will sue you if you give a bad reference."
Problem solved.
This means you have some pretty good job security, wouldn't it?
Why not use an automated too?
www.open-audit.org
Unlike software documentation, in a network there is more than one approach, pretty much a function of the amount of people that have to fiddle with the network at any given time. Usually there is physical laying of cables (where are they), box location and naming (labeling and Visio-sheets), peripherals (printers, faxes, phone system), box setup (OS and processes), server process configuration specifics and client requirements. And then there's that (what I think) very important document that describes what you wanted and why you implemented it in this way. It's a lot of writing, that's for sure, but like some other poster said: take a brainy, young intern and create this stuff on the fly.
Religion is what happens when nature strikes and groupthink goes wrong.
Passwords!
Vendor support numbers
Circuit ID's
Customer / Client numbers that support asks for
Links to KB articles that have resolved problems in the past.
When where and what the backup system runs. (You do have a backup right?)
Am I the only one who first thought, "If you have done the network correctly, it should explain itself"? Overly-complex networks take overly-complex documentation and overly-complex people to run and maintain. Mind you, there's a difference between correctly-complex and incorrectly-complex, but at the same time, every level of difficulty you go upwards in the configuration scale you exponentially increase the need for carefully calculated and layed-out systems. Ok, ok...now that I'm thrust back into reality...and given that you aren't likely to rebuild your network (although you might consider some re-work to help the self-explanatory aspect)... High level details, locations of closets, routers, switches, major cable runs, passwords, IPs, big details. All the small things the other person will have to figure out anyway and catalogue in their own mind in their own way, so doing it for them won't help a ton. Give them the tools to do their job properly and you'll be sitting fine.
Blog,Twitter
The phone numbers/emails for points of contact in other departments/companies.
You likely don't run *everything* and the new person needs to know who to contact when the interaction between inside and outside fails.
20 characters max for the password? How will I use my favorite poems as passwords?
The consultancy company I work for uses Novell Teaming and a follows a structure described on the main page as:
"This documentation is based on the Network DNA framework, originally developed by Don Krause. Although this is not longer developed by Don, it provides a good starting foundation for documenting a computer network."
This template has been perfect. If you can find the same template and implement it in [insert your favourite Wiki software here(MoinMoin)] then this should be just fine and cost $0.
...why make yourself obsolete? Irreplaceable people can't be promoted, so that may be a motivation to make yourself replaceable, but otherwise nothing good can come of this.
I maintain a largish network with vmware-dominated x86 Linux and Windows platforms.
I started with a live document containing a network diagram and a list of servers - meaning anyone can add to it. I use word - and have it in "outline" mode for easy collapsing..
No Passwords are kept in it (passwords are referred to a documented, password protected area within the IT group folder; a sync'd copy in the DR site and a copy with the DR documentation (just in case)).
The document is replicated to DR for survival when you have to rebuild.
My documentation stated with a network diagram now includes a rack layout and wiring diagram; and have now added details such as DHCP configs (although they change... thus the live document - so don't print it or its out of date).
I've even added underground cable runs (found when they pop the lid of the pit - with photo goodness) where fibre run to pits and the electrical company details to how the generator kicks in - who to contact i the event of a transformer blow etc... just in case I forget in 10 years time!
All IT team and top-level managers know "its in the network documentation" if they need to refer and I'm not there. They add their relevant parts - its a live document afterall.
Of course, you MUST make stipulations that this document should NEVER be released outside of the company - otherwise it will be handed over for tender responses; sales/marketing reports; sales people interested in the network; due dillegence reports; accounting audits and toilet-reading-for-those-who-have-nothing-better-to-do... for some reason managers love to use your hardwork to reduce them having to do hardwork - no matter how detrimental it is to the business!
Anyway - keep it a work in progress... and don't put anything in there that could compromise the company if fallen in the wrong hands (see above)..
Good luck... this job is never finished... but will feel good when you have it up-to-date.
My job requires me to do exactly what you're looking to do but for multiple companies/networks. Then, as soon as I'm done, I usually pack up and go or get hired in and fix the network.
Since I'm writing the Network Overview for managers AND potential future network managers I tend to write mine in the following format:
1) Synopsis of what the network does for the company, what general technologies they use (Windows AD vs *nix OD, thin clients vs Windows boxes, Cisco vs Brand X), and what the LOB software is.
2) Points of contact for the ISP and other providers (anti-spam, anti-virus, hardware, etc). Passwords for various accounts and services.
3) Logical network overview map (visio), containing firewalls, routers, switches, other devices, open/forwarded ports, IPs, what the servers do, what vlans are in place, Quick explainations for why (such as why vlan vs a seperate subnet).
4) Physical map of devices if the complexity of the network calls for it.
5) Software notes, what apps are critical for the business and which systems they rely on.
Then, for my specific job I have to do the following:
6) Licensing issues.
7) Network weaknesses/points of failure.
9) Other rec's.
Draw a horrible diagram in Visio (or similar) of what's connected where without any indication how it actually works!
If the network itself (switches and routers) is built on Cisco, there are commercial (SolarWinds) and Freeware (NeDi) tools to document the interconnections, VLANs, and configs. Assuming you've left CDP enabled and standardized your SNMP community strings, NeDi does a fine job of documenting a network with minimal effort. I've found switches the network team didn't even realize existed just from using the NeDi discovery mode and adding 'public' to the list of SNMP strings to try.
I do not deploy Linux. Ever.
nmap -sS -O 10.0.0.0/8 > handover.txt
Present client I am at I inherited a network of about 15,000 clients that was previously managed my a very incompetent IT department. Started by looking at the existing flowcharts and discovered that almost everything that was documented was wrong... Long story short I have been spending a fair bit of time reverse engineering their production environment so that we could accurately document it. Unfortunately we had come to the conclusion that we can use /nothing/ that the previous administrators left behind for documentation. You don't want someone like me coming in and looking at your documentation and declaring you incompetent, it can cost you your job.
You haven't detailed the size of your organization to know if you will need sign off from other departments or not. If possible try to get sign off so that they have a reference and you can create a standard that can be used to fix things and to ensure your designs don't get trampled by a new admin in another department. You really need to provide more detail on your environment for people to answer you.
I do most work in Visio, starting at 50,000 feet and working my way down. At this level I need to document network topology, server distribution and database server distribution. I work my way down from there using a zoom in style that has served me well for 30 some clients. Depending on the size, complexity and your area of responsibility you may need to flowchart anywhere from a 2-3 levels to potentially dozens of disparate processes. You haven't mentioned much about process development, I assume you want people to know how to do at least critical portions. Never write a process without flowcharting it, this will save you grief by getting people to focus on the process instead of a step by step set of directions. It takes someone fairly good to document the complex and make it look simple, that is your job at this point in time.
The bottom line is that your documentation should show dataflow for each critical system. As long as you can do this someone else can step in and work with what you have, even if they may not understand a given piece. One of the big advantages of flowcharting everything (especially processes!) is that this will readily show you weakness and holes that may have been previously overlooked. When flowcharting complex processes don't be afraid to have a single point represent an entire additional complex process that can be distictly referenced of it's own accord (as an car repair manual of mine once described the process to replace a crankshaft "Step 1. Remove Engine".) If you try to put to much detail in a given process you lose your audience and the value of the documentation.
Bottom line when I am done with a design document it covers server, network, database and client topology in varying levels of detail with dataflow. A typical design document I would turn over would be 150 pages with most of that broken down into different sections describing what was done, why it was done, the best practices followed for build, and best practices for lifecycle. The document typically does not get read by any one person, instead it would be a reference for a number of different departments that will each reference it according to their own needs.
after reading most of the 35 'replys' received in the 1/2 hour or so since the article was posted, it seems that most reply-ers may not be familiar with the requirements of 'electronic discovery'. If an entity becomes a participant in a federal law suit (even a non-party participant), and, increasingly in the state courts, a logical inventory of your network is the minimum that must be ready to be shared with the other parties in the suit (basically). [i am not an attorney, nor a spokesperson] I say minimum because the rest of the requirements are a whole lot more detailed; inventories of all network devices and data storage devices, including how each may be used, when it was last reformatted, last replaced, what happened to that replaced storage device, whether or not there is any policy for that, how it may be enforced, etc., all data on any part of your network, how\why it is accumulated, what types of document formats, policies & procedures for the same and including data destruction, all users and their permissions to that data, back-ups, web sites, blogs relative to the entity or where entity-specific information may be located, emails, etc, etc... . . oh, and then there is also the data authentication procedures if you hope to use anything electronically stored in court. And by the way, data in RAM may also be an issue, sometimes!!?? - just some incentive for all of us to work this out.
cjacobs001
This is why you document while you do the work.
Most equipments systems and application have fancy features that allow to do elaborate things efficiently with less resources. This is an enjoyable part of our work, unfortunately it should be banned.
Restrain Yourself from the temptation to use those features.
Implement everything with the most basic and standard approach.
This may be frustrating, you may feel that you are wasting cash and time and sacrificing performance, but actually you'll get a more reliable and flexible system. And and outsider will be able to understand it more quickly.
Most systems allow to insert comments in the configuration. Use that extensively. The comments are the most immediate documentation and usually the most up to date.
One last hint: once your system is running and you have removed anything fancy from it leaving only the necessary complexity, take 15 minutes to describe the profile of the person that is eligible to manage it. Include books with the general knowledge that this person will need. Handle the description to your management.
This approach has following advantages:
- screening out totally unfit candidates
- helping the successor filling gaps in his knowledge
- avoiding to describe in your documentation common knowledge (in my experience this is 30-70% of the document and could be replaced with references to appropriate books)
- (free bonus) giving the management a better understanding of your own value
There are drawbacks as well:
- Going through books would take more to get a grasp than if you explain everything inline.
You can palliate by giving references to specific chapters. And stress on the fact that no one should be allowed to touch the systems *before* having the knowledge in the book. It's like driving the car: you should learn *before*, not *while* going to the highway.
I work as an architect rather than at the coalface of keeping a network of the kind described going, but one policy I would suggest is that if there are particular aspects of your network you think will be hard to document, it would be profitable to change the truth about those aspects to something you will find easier to describe.
Ideally you should do this by changing the implementation without noticeably affecting the services provided to end-users. Software development people may recognize this as relating to the concept of refactoring.
I only have one tip: label cables clearly at both end.
If you are in the server room, and you have:
A: a spreadsheet that your predecessor made.
B: a post-it note on the switch saying it what it does.
Which one do you trust?
For the physical/low-level network the documentation should be in the network. Just like source code should contain comments about this particular piece of code, a similar approach works reasonably for the physical network. I see no point in a having an outdated spreadsheet. It is more useful that the cables and ports are labelled and numbered, that there is a post-it note on a switch say where the links go, etc.
The grand overview should be in electronic form, though. A scanned hand drawing is fine. A photo of a whiteboard drawing is fine too.
For the logical network put comments whereever possible. On settings, VLAN configurations, server connections, account setups, ...
Again, the grand overview should be in electronic form.
I have found it useful that the information is timestamped, so you know when it was last valid.
http://www.spiceworks.com/
Good free tool.
Please, tell me your real name, I don't want to have you anywhere close to my systems as soon as I don't get a pension cheque.
IANAL but write like a drunk one.
Why not hire a technical writer to work with you to develop meaningful documentation. That's what we do. With your help, we do it better/faster/cheaper than you ever could by yourself. And your successor -- if he/she reads it -- might even admit, "Oh!! this makes sense!"
No wonder our field and many of our professions have such a bad reputation.
I have read only a few posts and two (moded up 5) say pretty much to ignore the issue.
In several networks I have worked with fundamental information was non existent. This translated in lost time, down time and actually losing money (if you lost your job in one of those companies recently, the indolent SAs or Network administrators may be partly to blame).
You never know who the next guy will be, if he is less experienced or capable then the documentation will be very valuable, if he is more experienced or capable then you would have saved their time to do some real work, after all they (and you) have not being hired to do forensics.
How a professional can hide behind the "let's be real" nonsense is beyond the pale.
IANAL but write like a drunk one.
Complex is entirely subjective, and as such what is self explanatory to you may not be to others.
Yet another cavalier approach to doing the job properly, and I am not even half way the comments.
Dispiriting frankly :-(
IANAL but write like a drunk one.
All you are doing is wasting your companies money paying you salary for doing what they probably don't care about. The reason is because since there is no infrastructure for documenting it and keeping it current it will never be a part of the company. Without a clear chain of people up to the CTO monitoring that it is documented and kept safe you might as well use post it notes.
Sure keep your own documents but the company will just use your untimely death to buy new equipment, cut salaries and pay expensive consultants to fix the problem. Your death will cause someone to get a raise or fired or both.
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
Ah, you're not following the MACK(TM) Truck Rule.
The MACK Truck Rule (MTR for short) is a measuring stick which we use do determine if a solution is good for us. Basically, it's an objective measurement of the level of expertiese required to do something. Basically, the MTR has you ask yourself (Or your team) the following question:
If the person(s) responsible for a task was suddenly hit by a MACK(TM) truck, How much time would it take for somebody else, untrained, to complete that task if needed?
If that amount of time is unreasonable*, It doesn't follow the MTR. Notice the caveat for unreasonable; this is the subjective part. What' unreasonable for one may be reasonable for another. This needs to be decided for yourselves.
Documentation always helps difficult tasks pass the MTR. So can good support. I try to leave a readme in the place where the installer is for a difficult program. I'm now begining to use FreeMind to map out networks and servers. I have a good ticket system for all our repairs. Hopefully these things will make things easier the day I want to take a vacation.
--Pathway
I was wholly responsible for a large network. I usually had a few people working under me, so it was essential that we could find information about the servers.
Our documentation was pretty simple. Actually, very simple. We had an internal web site set up so techs could message each other on what they were doing. It was a simple chat type interface. Every line was stored in a database.
The company had several large sites, and tens of thousands of hosted domains. The internal web site had plain text files listing out the servers, grouped by city, then rack, with their name, their primary IP, and their function.
A second list was a copy & paste from the PDU's, showing what machines were plugged in where.
Both lists were linked from the web page for easy access, plus you could view them easily by logging into the server. As long as cat, less, or vi were working, they were easy to find.
At another company, we set up a more elaborate page. It was entirely PHP/MySQL. It recorded all the details of the machine. IP, hostname, virtual IP's, manufacturer, model, serial, city, rack number, rack position, number of rack units the server occupied, the full output of dmidecode/vmidecode, it's function, versions of server software installed, etc. This was searchable, and you could print information on a single server, rack, all racks in the city, or any searched criteria. In addition, since some of the staff were freakin' illiterate, a second result page showed the information appearing like stickers on stock images of the machine fronts, including empty spaces. If the information was accurately maintained, you should be able to take a printout of the rack, and compare it directly to the rack, and everything would correspond.
This second version made a very useful method to not need to create visio diagrams of every freakin' rack, and convince the people maintaining the diagrams to keep them updated with changes. Rather, if someone added or removed a machine, they were to enter the information into the warm fuzzy web interface.
There are programs to do similar things, but I never found one that quite fit the business needs, so a few days of programming gave the interface. Populating information was pretty easy with a perl script, dmidecode, and a few other basic command line functions and DNS requests.
Switches and routers were handled a little differently, but in a similar manner. Cisco 'show version' and 'show run' gave sufficient information to store everything about the way the switch ran, so in the event of a complete switch failure, a new switch could be dropped in place, the config could be copy & pasted into a terminal session, and it would be running exactly as expected.
Serious? Seriousness is well above my pay grade.
Any of the networks that I've worked with have always been for smallish companies (i.e. 80 employees) but often with very large offices. Things like museums, factories, mills, etc.
Without a doubt, the one thing that has helped me more than anything else has been when the person who came before me kept a very fastidious cabling system. By keeping the area as tidy as possible, and accurately (and informatively) labeling their cables and ports, it was very easy for me to work with the network.
Now, good labeling, and a tidy closet is not the same as quality documentation. But, it's probably analogous to well formatted code and useful code comments. (Where I think the OP is more looking for how to write the architectural specs.)
I have a nicely formatted template page with all those categories set out. I also maintain a page of IP address assignments and an inventory of harware specs of all the machines in the office (which is helpful in the cases of "We need to reproduce a bug that only happens on ____ processor with ____ video card" and of "We're getting new machines. Who is in most dire need of an upgrade?").
I write down everything in these, and find myself referring to them very often. My predecessor gave me a Word document with all his notes in it, which has been very useful, and I used that as a starting point for my pages. The wiki has saved me a ton of time, kept me organized, and serves as a great reference for me and for the inevitable next admin.
The only caveat is if the wiki (or the server it's on) goes down. This has happened once, and my instructions for fixing the wiki were... on the wiki, so extra troubleshooting for me. Thus, I find it good practice to maintain a hard copy of the wiki pages, especially the page that tells how to fix the wiki.
I'm running this on Redmine, which has proven to be bleedingly simple to use and administer, and much easier than trac, which we used before. It's especially nice having it on the intranet, as I'll just have a browser open to the wiki as I work on systems and refer to and update it as appropriate. It's very handy to document exactly how I performed a strange or experimental installation of some software that I'll want to replicate later without making myself crazy, and I'll take the extra few seconds to retype the commands I just used into the wiki from anywhere in the building, though I probably wouldn't do the same into a Word doc.
It's not so much the mundane day-to-day that I find that important to document. It's the weird fixes, the trouble spots, the command line parameters, the installation procedures, the changes that shouldn't have fixed it but did, and the horrific chain reaction situations that make one piece of software crash because a seemingly unrelated piece of software has the wrong version of the 64-bit library. Things that take 4 hours to figure out and 3 seconds to implement... those are the ones to document, and those are the ones that I'd be kicking myself 20 months later for neglecting to write down. In an afternoon, any schmuck could walk into the building and figure out which network cable goes where. Documenting the strange bits (and the frustrations), though, can get a malfunctioning mail server back up and running in 3 minutes instead of 3 hours (which, of course, is secondary to good administration keeping the server from going down in the first place).
-- I prefer the term "karma escort."
...write down the passwords ...make graphics of the wiring with different symbols for clients, servers, including labels which tell the ip and dns name ...for servers: write down their function, which services run on them and step-by-step guides on how to do maintenance work ...document as visual as you can and it makes sense
A wiki is what we used at my old networking company (Stability Networks) that specialized in Small Business contract work. Recently I have grown rather fond of Google App's Wiki. It's free, relatively powerful, and (most importantly) very user friendly. I use it at my current company.
Is there anything better than clicking through Microsoft ads on Slashdot?
Document only when necessary or ordered by your manager
Take pictures. When discussing a piece of gear a picture is worth a thousand words. Instead of "the blue thingie halfway down the rack" or "that black thingie in the corner behind the laser printer" progressive pictures of the room, corner, and black thingie are priceless.
Remember that documentation isn't for the outside consultant or even for the guy or gal who replaces you (ooh, is she hot?) It's for YOU so you don't have to remember so that if you ARE hit by a bus, this is like the "My own guide for me to help me do my job." Document as if you know nothing. If it's a strange piece of gear include a copy of the config OR where the config is backed up AND how to get the config into it OR a link to the mfg website that tells the same.
Pretend the person reading the guide you write is NOT an expert. This won't hurt you or the outside consultant or your replacement (wait, IS she hot?) but it will help anyone who needs it.
Finally, make sure it's well-documented as to where the documentation is! I've done gigs where I've wasted days reverse engineering something only to find that somewhere in the pile of charlie romeo alpha poppa was a set of good fully-written but never-mentioned docs.
Ehud /. mods are herd animals.
P.S. Often a printout of same in a three-ring binder with a cover "WiKi docs as per 2009-05-26 online at http://ourdocs.mydomain.org/" will have a dual purpose of providing DRP documentation (in case everything fails) as well as pointing to the real docs.
P.P.S. Ignore my being modded "troll", it's just that
I worked as a technician for a mom and pop shop and the most frustrating thing for me was not labeling a switch or the wall mounts. Most MS networks are not that hard to administrate. Usernames and passwords are a given and you should be doing that anyway.
Attempting (even facetiously) to blame SPAM on Windows is wrong. If every copy of Windows on the Internet somehow magically disappeared, the SPAM problem would not abate. Bot herders and spammers would simply shift their efforts to other platforms.
If your doubt this, consider what the winner of this year's PWN2OWN contest had to say about why it's easier to target Mac OS X.
BTW, this is not a troll, and I'm not a (Windows|Mac|Linux) evangelist of any kind. I just find kneejerk Windows bashing rather tiresome
There are templates, and maybe they work for some networks, but from what I've seen, they don't work for an 'organic' network. Especially one which has been shoehorned into working by someone competent, and then allowed to languish at the heightened state for a while before being taken over by a competent person. (It's like what happens when you hit a cluster of dandelions with a non-mulching mower, sorta.)
There are a couple basic things to document, in my opinion.
* Passwords. They go in the lock box which you and your boss have keys to.
* Configuration files. Print them out and archive them.
* Infrastructure topology and routing. BIG BIG BIG. This stuff can go left undiscovered for years and takes a long time to actually dig into, especially if you don't have a central management system.
* hostnames (and IPs if apropos) with associated services
* Critical service list (and interdependencies).. granted sorta like the previous one but a bit different)
* Things that are on the "do not touch" list - eg. legacy applications - which will cause problems or have problems with certain infrastructural changes (eg. I know of one terminal client which can not exist on a DHCP network)
* Peculiar configurations on workstations used regularly, eg. in a dept. (see the previous point)
* The location of all assets
* The physical/digital location of ALL OTHER DOCUMENTATION!
The above should be in a printed binder and reviewed every couple months for changes. It should short and to the poitn so that your incumbent (or you) can use it as a quick reference.
You should also already have some way to track licenses and control the software installed on workstations; it's part of your documentation.
There are only a couple additional things I'd document (unless you're feeling bored and want to give your employer too much of an incentive to get rid of you for some young and inexperienced blood) as a small network administrator, IMO, to get out from under the "negligent" banner. Do more as time and desire permits, I suppose.
Personally, I think it's very important to document everything remotely interesting/useful, and to make it accessible in a fashion that's more useful than figuring out the 'neat'/solution again. Keeping a daily 'script' so you can backtrace what you did is potentially useful short-term (for your incumbent in the event you get hit by a bus, or a new coworker). Keeping a daily log of what you did is also useful.
Keeping a wiki from here-on-out of problems as they crop up (as well as documenting the more important information retroactively) is what I'd call the bare minimum.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
Here is my advice for a quick an SUSTAINABLE documentation.
1) Create a Documentation folder on your central file server
2) Make sure that server is on a regular backup schedule to tape / alternate media
3) Create a macro or script that does the following each (at least once per month)
a. Telnet or SSH to network infrastructure using an account that is privileged as read only
b. Copy or capture the configuration file. In Cisco-esse it would be show run but you may want to do a show tech
c. If you have a central authentication server (AD / RADIUS / TACACs), copy the config file for these as well. If you do not have a central authentication server, then you should create strong root passwords - I recommend using an MD5 hash program on a pass phrase, and using the output of the hash. Write down the pass phrase and the hash type and result on a 3x5 card, fold, tape shut, put in an envelop that gets stored with other business critical documents at the very least, stored with the key control or in a 2 hr fire safe.
d. After you have all of the configuration files, zip the entire directory into one file.
f. Use SD card / DVD / alternate media and manually copy the zip file to physical media and put into a RED notebook binder. Make two copies. One to be placed by the ENTRANCE of the data room / server room. The other to be placed by the front desk or security and kept under lock and key. Off site data store of server tapes is a good idea as well.
This means that at the start of the you will need to burn a copy of the zip file twice and then replace the old files in the notebooks. If you schedule this properly, it will often consume no more than 30 min of your time.
Last thing to do is to create and IT disaster team. Have a once a quarter meeting that updates the IT staff and others outside of IT as to the location and response to a disaster.
This represents a stop gap, but low cost model that will allow most small to mid sized businesses to recover should there be an event that would damage operations (or you)
Totally agree about the wiki. I use a wiki because it's easy to write and fast to update.
All my network diagrams are done in graphviz in mediawiki which is also easy to write and fast to update once you get used to it.
As a bonus mediawiki keeps every old version of your documentation so you can easily prove what great work you have been doing.
I have been to work environments where people documented things... but one's person view of the network in his documentation is totally different than what someone else had, especially with regards to which switch went to what vlan. Documentation gets useless when four people have five world-views on what is going on, especially what is connected to what punchdown block.
The trick is to have centralized documentation on a server, but make sure the machine is both locked down six days from Sunday, and religiously backed up. Not just hang an external drive from it, but if possible, archive everything to DVD and keep multiple DVD copies offsite. If a wiki is used, back up the mySQL database, copy that file to the DVD, as well as the wiki's config directory. In a UNIX shop, I would consider a wiki, have Apache run SSL only, and require a username/password for any type of access.
Of course, document how the central documentation server is laid out, and have this documentation be in a standard format that your organization uses. Of course, have a hardcopy filed away in a physical filing cabinet just in case machines are down. The passwords for the central documentation server would be written down, inserted in a sealed envelope, and put in a place that only the CxOs and other corporate officers of the company have access.
Of course, one thing I stress about servers such as backup servers and documentation servers is to not just have network security, but to have some physical protection. Most UNIX variants are starting to have hard disk encryption. On the Windows side, I'd highly recommend a machine with a TPM and BitLocker. Other utilities like PGP or TrueCrypt is also good for this. The advantage of BitLocker is that you can set a PIN on the TPM, and the TPM limits brute force guessing. Of course, if the machine needs recovered, just stick in a usb flash drive right out of the tape safe, and it will boot just in case the PIN got forgotten. On this server, you want both security and recoverability.
As for things to document, network topology, machines, and perhaps a database storing machine passwords if the machine is secure enough. There are two other things:
First, document recovery mechanisms for machines. TrueCrypt's system encryption will have a user make a recovery CD (it can be turned off, but on by default.) PGP can offer recovery keys or passphrases. BitLocker will have a recovery key consisting of a 48 digit password. Windows EFS can have a data recovery agent key generated and the private part stored offline. Store all of these, especially of core machines in a secure, but retrievable place. This way, you can get access to a user's machine or a server if it was rendered unbootable, or the TPM on it was reset.
Second, document invoices, perhaps store them in a standard format your company uses, be it TIFF files, PDFs, or other.
The reason that documenting invoices is important are audits by the SIAA and BSA. There are a lot of people who, if fired for any reason, will immediately turn around and file a report with the copyright enforcement big guns alleging copyright infringement at the company. Soon after, you will have some attorneys requesting both a list of what software is licensed to what machines, and copies of invoices. Turn them away, they will be back with the constable, a motion of discovery, and a demand for the information they were requesting several hours ago. If you have the two, and the invoices show everything is licensed, you are free and clear, and likely won't be bothered by the guys again. If your records stink with no proof that anything was bought (and no, CD keys and "proof of licenses" jammed in some file drawer don't count), things start to go real painful real fast. So, having the ability to run an audit, print a list of machines and what they are running (OS and apps), then print the pertinate invoices showing that the machines are properly licensed is crucial for a business.
Finally, it doesn't hurt to have your own personal documentation trail on a secure part of your main workstation. This can help if a blamestorming fest occurs because a cow-orker toasted the Web server and starts pointing fingers at the other admins.
Store it in two or more physical medium (DVD/USB drive/printed on paper/whatever). These do tend to get lost and will never be found when you need them but too many people store it inside the network never realizing that you need them when the network isn't working.
There are a few things that are often overlooked and outright forgotten when documenting networks. I had to take over a few networks, let's see what I usually miss:
Every admin remembers to hand over passwords. Except for the routers.
Routers and other "managed black boxes" are notoriously being left out from the list of passwords. Fortunately, more often than not it's the standard password because "nobody has to touch them but me anyway" (ignoring that, if people only touched what they should, passwords would be moot...)
Every admin remembers to draw you a network layout. They don't tell you WHERE those switches physically are, though.
In large companies (read: Lots of room to cover, independent of the number of people working there), this can indeed be a problem. Especially when there's not one single server room where everything is collected, when you have switches and routers hidden in cupboards and other "innocent looking" furniture, cables that appear out of nowhere and disappear into walls, without an indicator where they surface again. Or what purpose they serve, first of all.
What HAS to be documented is the reuse of resources
That's the worst of the "undocumented changes". When you find a switch that shouldn't be there, you know you have to investigate, you know something wasn't documented. When you find a certain box sitting where it is supposed to be, you don't investigate. You expect it to do what it allegedly does. If it does not and has been "recycled" to fulfill another role, the whole documentation goes out the window. Because now you start questioning EVERY piece of hardware.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Draw a horrible diagram in Visio (or similar) of what's connected where without any indication how it actually works!
There are certainly a lot of pretty but unhelpful visio diagrams in the world. That's why Visio sucks. It's pretty, really pretty, but for network diagrams you don't need pretty you need accurate and easy to read and update.
I'll stick to mediawiki and graphviz and get my updates done in much less time, with built in versioning, and accessible over the intranet to anyone who cares.
Your story sounds very familiar to me, but at my office there is a significant difference: I am not alone, a small team is maintaining the network.
So our first choice for documentation was a wiki (MediaWiki in this case). In addition to the known benefits
there are some ways to display the content clearly arranged. For example take the wiki page of a network switch. A little picture lets you identify the model easily and a tabular contains one entry for each port and a link to the component which is connected.
We also put server racks into tables and linked the machines at the specified positions.
Of course we tried some other software packages for this task, but none was as flexible as the wiki solution.
If it is large enough (which it doesn't seem to be) you should divide stuff up in modules. Start from OSI layer 1 and work your way up.
If it's small, you probably wind up merging loads of stuff into one document in which a serious amounts of stuff is considered to be "the network" although it isn't.
/. to get this information. You're not the first person that has to do this. Must be a slow day here.
Having said this, there are places to go other than
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
All you are doing is wasting your companies money paying you salary for doing what they probably don't care about.
Doing a good job isn't just about doing what you are told. Just because management don't care about something doesn't mean you get to not care too.
Sometimes you have to do the right thing and often the right thing is helping the next guy who gets your job so that everything says running.
"don't print it or its out of date"
Good advise for all sorts of technical docs.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
I use a little KDE application called "basket". It allows you to embed any file in to a note, and to have a hierarchy of notes. This lets you organize thoughts and ideas (IP's and passwords and keys and images and whatever) almost like you would any other file. It's the same paradigm as a filesystem, just applied to (mostly) short notes with a very quick and simple point and click interface.
In addition to it's usefulness while sitting in the system tray, it can export to standard HTML that will give anyone with a browser a readable copy of what you have (complete with embedded files). Once a month I make an export and shoot it off to the relevant people Just In Case. I'm sure this could be automated through DCOP...
All of this is based off the KDE3 version of basket. I haven't used the KDE4 version much, but the one time that I did my data was imported *flawlessly* without any intervention or dicking around with anything. YMMV. (here's to KDE4 ever being as good as KDE3!)
Basket is *not* what you want if you need to have multiple people working on the documentation at the same time -- we have wiki's for that. It is effectively "mind-mapping" software, which means it works best when used by yourself for yourself. I keep a sub-basket for all of my client's information, and that's what I share with my clients. The way I write stuff down is specific to me, but all the information is there (it's what *I* copy-and-paste from) and anyone that can speak English is more than qualified to be able to get information from my exported basket. The rest of the information in it is for my eyes only, so I simply don't export it. It supports encryption, but I haven't messed with it yet.
My only complaints are that it hates Windows and that it doesn't have some kind of not-read-only web interface. Regarding the Windows port, I only briefly messed with the KDE4 version, and it failed horribly to even start. Past that I haven't bothered with it on Windows and don't much care to.
1. portscan everything on your entire network and spit it out into a text file
2. set up a wiki
3. paste the results of the portscan into the wiki
4. start writing about everything that showed up
i've actually done this before with a pretty high degree of success, pm me if you want some help setting it up
Read a lot of good posts and ideas so far here. From my perspective, the most cost effective solution for you and the business is, you need a backup engineer for in case you do get hit by that bus. Having a person knowledgeable enough about your network to keep it running in the event you are incapacitated for a length of time is by far the most beneficial, if for no other reason, because of the quick turnaround time they can come in and take over vs. company looking for another engineer, and the time it takes to learn the network and scrounge threw docs you created.
Very few documents are actually that meaningful if the engineer is halfway competent so as others have mentioned, no need to go documentation crazy. There are key docs I feel though that should be created and maintained and have been mentioned above.
1) Passwords, I cannot stress this enough, get all accounts privileged accounts and service accounts documented with passwords and secured somewhere (preferably off the network, such as a USB key with the data on it in a safe) as without this, it can be a very ugly scene.
2) Next, overall, logical and physical network diagrams are paramount. If done correctly can make troubleshooting a breeze, and a nightmare if not done correctly. One link that I like is a reference to a best practice guide about the Cisco 4000, 5000, and 6000 series equipment found here ( http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a0080094713.shtml#management_cfg ). Go to the network diagrams section and review the overall, physical, and logical section. Create your docs with this as a guide and any engineer who may have to troubleshoot the network will love you for it.
3) The answer to what 'other' documents should I create? Comes from you. Knowing what you know about your network, pretend you are coming into the network for the first time, and ask yourself, what I would wish I knew about this network? Make a list of your business critical functions where people would be screaming if the service was inaccessible. Document what would be useful info in a DR scenario of recovering the service. This leads me to the last doc I would recommend as useful only as an insurance policy for the business.
4) A procedural document of how to recover various business critical services. Again, key focus is on business critical, business users or clients will care less about non business critical services or be a lot more forgiving. This can assist greatly an engineer if good recovery procedures are documented, especially in area where customizations have been done (i.e. scripts and what not)
The other biggest important thing you should do is manage the businesses expectations. Talk with the business to get feedback as to What are the business critical services and document them. Next, get your Service Level Agreements ( SLAs ) agreed upon between you and them. And make sure you can meet them. If not, get a projects/tasks list together of what needs to be done so that either A) the business will fork over cash to meet agreed upon SLAs or B) they will accept the current SLAs.
The SLAs are important because it will force you to take a hard look at the network to see if you meeting their expectations. That is really what it all comes down to. When I.T. does not meet expectations is when the business gets all bent outta shape. Manage the expectations and get your SLAs agreed upon for restoration of services and you will be ok.
One more link that can help in ensuring you can meet SLAs is getting your RTO and RPO defined for you business critical services. Here is a nice easy link that talks about this that should help you.
( http://findarticles.com/p/articles/mi_m0BRZ/is_3_24/ai_n6017376/ )
Good Luck!
For example ITIL; http://en.wikipedia.org/wiki/Information_Technology_Infrastructure_Library
You don't have to try to implement all of the stuff at once, you'd be mad to try, but if you start at what sounds like the most important for you, like service portfolio and configuration management you will have solved your current problem.
p.s. my consultancy rates are very reasonable.
Deleted
You'll also want inventory, service contract information in place. Equipment under contract and equipment that's just out there. I used to see off lease equipment go into internal service since it was inexpensive. Of course, the users never wanted to put it under service, that would take funds. Everyone here knows what happened next.
If there are services or equipment where redundancy is required or was waived the next guy needs to know. When something's down, it'll be a big problem but when service renewal is up no one wants to open the checkbook.
Sounds weird maybe, but try Protege Owl (http://protege.stanford.edu). It comes out of biomedical research but, using the W3C OWL standard, is a brilliant tool for capturing complex knowledge of any kind. I use it to document a metro-wide fibre network (http://www.sabrenet.edu.au).
Some advice: avoid proprietary solutions like Visio..
Some questions:
1. if you are hit by a bus, you still care about your good references?!?
2. if they want to replace you with someone cheaper (i.e. you lose your job?): why bother leaving 'all sorts of details that outside consultants could understand too'? It's their problem end responsibility. Your professionality as network engineer might be to make sure to leave technical documentation, not oversimplify for future consultants.
When I was doing consulting/engineering services for SMB, the documentation I made was modeled after the actual OSI model.
So, you start with Layer 8, which is a basic description of your company's services, needs and requirements of the network, technical contacts, third parties involved in the network (security, telephony, ISP, hardware suppliers, specialized software) and their contracts.
Layer 7: Document the applications in use and how they are 'connected'. Create a global picture of what applications exist in the network, how they are connected, any dependencies on specific versions (e.g. frameworks, Java versions, databases, etc). Then, create a separate section for each 'application' that details the configuration of that application. E.g. the Active Directory gets its own section where you describe the domain name, relevant domain controllers, OU structure, blabla. For your web servers, document the site names, devices/servers responsible for the service, references to other documentation. Garnish with screen shots. Repeat for all applications.
Layer 6 may be hard to document because it's not always transparent, but include things like special software languages, broker services or anything sitting between your network and applications.
Layer 5, in a Windows-based network, should include documentation on the NT session security and compatibility level (authentication and signing requirements, if any).
Layer 4 should mention the transport protocols in use in your network (don't forget things like telephony), and any special settings made to the TCP/IP protocol stack required to make stuff work. Firewall access rules fit in this section nicely.
Layer 3 should include the addressing and routing schemes for your network. Server/distribution/access subnets, printer subnets, DHCP scopes, DNS servers, VPN connectivity, etc should all be described in this section.
Layer 2 should include the switch configuration documentation, VLAN configuration, frame types in use (again, don't forget telephony and wireless networks).
Layer 1 should include a building plan, documentation/certification of cabling infrastructure, patch cabinet documentation, maybe things like power infrastructure, physical access control. Sure you can make up the other stuff applicable to your network.
This is the global picture. The detailed configuration information for each server and device in the network should of course be noted, but that kind of information is useless without the big picture (which explains WHY the device has been configured thus).
I also never include security related information in generic documentation. A list of credentials should be stored safely in a separate document that sits in your boss' safe.
Hope this helps.
MMO Vampire Role Playing
Don't forget the data flows (port, protocol, direction, storage) and the data classification (what, where, why, level of sensitivity, protection, data owner etc.)
Yep.
I want to delete my account but Slashdot doesn't allow it.
My favourite technique for making sure documentation is done and updated, get the new guy to do it. Then he/she has to go all around the campus, locating servers and getting serial numbers form all sorts of odd equipment and making sure all of the support aggreements are current and the contact details for the vendors are accurate.
The other favourite is if I find new equpment that has been installed and is not labelled or documented, I get the installer responsible to audit all similar equipment to make sure there are no other ones missed out. After haivng to crawl around dozens of risers and labelling or confirming all switches etc are correct and documented, they don't often make the same mistake twice.
We also have a password management system which also allows details like how to install the management console or the URL to access a system for management to be stored.
My answer to any question about "What is the password for X", or "what the hell is the name of the server for X applicaiton" is "Its in the store" Then if it isn't, we add it 8) Only takes a few times for the newbies to start looking up the information themselves first.
The other key file is a massive Visio document with a summary page with a managment style overview, and then a document with everyhting in it in layers like an electircal diagram or building plan.
Lay in the workstaitons VLAN, the switch management VLAN, the Servers VLAN, link to things that are self contained like all of the Firewalls and DMZ configurations.
etc.
First: You must make everything as self-documenting as possible. Label every server, every cable, every power lead to within an inch of its life. And establish processes which say "when a cable is moved or added, labelling is updated accordingly". If you don't have a labelling machine, buy one.
That deals with basic "what's plugged in where" and is far more likely to stay up to date than a spreadsheet or wiki page.
Second: Whatever you choose, it must be something which can scale to your needs and which you can live with.
It will need regular updating - and quite frankly, very few people are able or willing to regularly update a single 200 page Word document complete with embedded spreadsheets, diagrams and photographs. A wiki - or even Sharepoint, if that's your thing - may be better. But if you do take the Wiki route, make sure you keep hard copies of the documentation which says "If the sh1t hits the fan, this is what you need to do to recover".
Others have said "don't bother, your successor won't read it" - I say balls. Documenting is more than just helping your successor - it also helps you remember what is set up, clarify how things work and as part of the process you start to look at things and think "hang on a minute.... this document I've written describes something quite absurd. Are we really doing that?"
Whether or not your successor reads it is really not your problem.
It depends largely on the organization, but more often than not, generating documents is often a goal on its own. Creating visibility, credibility. And most of those documents are only used at most once in a presentation - if you get lucky. It's called bureaucracy and it is probably as old as civilisation.
On its own it might seem a cynical statement, but it is not. People tend to learn that in this massive amount of information, nothing of value is to be found. So, they ignore it habitually.
Now, to break away from the cynicism, I would recommend
Try http://www.Lansweeper.com is your network is mostly windows.
It's free and gives a lot of useful information
Can be bother to write handover documents?
Easy!! Get your replacement to document the network for you, answer all the questions he has and the job is done.
Assuming of course that you do a handover...... but then that's your companies fault not yours.
Yes there is an escape route in there somewhere.
And if you can, maintain some sort of wiki. It is ideal for those kind of things.
For some long running gathering of information, I use the self contained (single file HTML + JS) TiddlyWiki, easy to keep on a USB stick...
Yeah, not everyone takes pride in their work. Those of us who do will do as good and complete a job as we can, because of what it means to us to, somewhat irrespective of external consequences. Those who don't will do only as much as will be noticed and as little as they can get away with, but deprive themselves of the satisfaction and accomplishments that the prideful get to feel. Not that that's a perfect consolation for the damages they cause.
The revolution will not be televised... but it will have a page on Wikipedia
Label both ends of every goddamn cable, especially those that run inside of walls, under raised floor, etc.
ITIL...ITIL...ITIL...
Why has no one mentioned this standard yet? Is it because it's European (British) and the US won't use anything not invented there?
The Europeans have already got a complete set of standards for deigning, buildng and operating computer systems properly. USE THEM!!!
99,99% of all known possible successors will just hotfix problems as they arise and blame everything on the predecessor. So just write up the things you need and tend to forget in a way you can use...
the cans of compressed air in every office supply store? inverted they throw out a very cold liquid that does exactly what you describe.
every day http://en.wikipedia.org/wiki/Special:Random
A network has lots of things sending each other their addresses at various levels. Get a tool that gathers and analyses that data, a web search will turn up tools of various levels of sophistication. The documentation you create should be "how to use the network management tools". You will never have time to keep low level documentation up to date so automate that level and focus on making sure your successor has pointers on how to use the tools effectively.
I have seen people suggest Visio, Excel and other tools that should not be on your list. Get an automatic system for doing this.
I do not know the size of your network but Microsoft, HP and IBM all have tools that range from bad through decent to good, depending on more factors than I would like to delve into here. If, for example, you have a large network, the later Tivoli solutions from IBM will discover your network, store your passwords, discover the applications on your network, register and report on dependencies (your CRM depends on subnet 172.130.*.*, your Oracle Financials needs port xxx and yyy open in FirewallA, FirewallB and FirewallD). Just as an example. I am not with IBM and I haven't dealt with Tivoli in about a year, but it worked really well for us. We had a huge network, tons of in-house apps and other things though.
As an example, we got a notification about a critical security patch for a specific version of our hardware, my spreadsheets were wildly out of date, but it took less than a minute with our network management software to get a list of all the HW that required this particular patch. You will never be able to do this with Visio and Excel.
For everyone that thinks no one is going to read the documentation so why bother. Please keep that attitude! I want your jobs. Several of my bigger clients came from them firing the idiots that were their then network administers because of lack of transparency in passwords and configuration information for the IT infrastructure.
Owners of businesses like to know that they can access their own infrastructure or get someone new to access it if the old IT guy gets run over in the street.
My computer company's standard policy is to compile a network diagram, list of all computers, servers, routers, network printers, time clocks, etc... with all user names and passwords as well as ipaddresses and ports needed to access admin and user configuration utilitys.
I have never lost a job or client by doing this. I have taken clients away from other computer companys and system admins by doing this.
A great tool to help document a network would be a device like Locate by eTelemetry. You plug it in, and connect it to a port that mirrors your authentication traffic, then point it at your switches and your LDAP server. It will load from LDAP, listen to the network and crawl your switches to map a real person to a switch port. If you know that user "Bill Gates" is in office 107, and that he's connected to port 21 on switch 10.1.1.10, you can build a pretty good topology map without getting out of your chair, much less tracing cables. Used it on several occasions, good tool.
Our profession is filled with idiots. You find yourself at a company, with no information, what do you do? Ask the company for the documentation left by your predecessor. If they say they don't have it, find out how to contact your predecessors. They can tell you or hang up in your face with a laugh. Been my experience even people who left pissed have forked over the information. And oh, if the company doesn't want to fork over the information to contact a person to help save them money, then I would seriously start a job search elsewhere, because a company who cares about their pride more than lossing 700k a day, doesn't have its priorities straight.
Personally I would settle for "I'm sorry about this. We were all really drunk at the time."
When I managed a bank network, I hired an firm to audit and document our network, policies, and procedures.
This served two purposes:
Using a third party, you get a better perspective on the design of your network from the outside. Many networks suffer from "this is the way it has always been done" syndrome, and a third party opinion may better reflect what an outside hire will want to see if you need to be replaced.
Regulatory compliance. If you are managing a network for a regulated industry (medical, financial, public/government enterprise...etc), you may be required by law to (regularly) have an outside firm audit your network. If this is required, you may as well have these guys do your documentation for you.
Peer review is a good thing in the software business - professional network engineers should also use it.
-ted
If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run. Ultimately, this won't be a good reference for me if they are trying to work out technical details for years to come.
If you are hit by a bus, you won't be worrying about references for a while. There are less painful ways to throw in the towel.
"Who controls the past controls the future. Who controls the present controls the past." -- George Orwell
A network diagram is absolutely a start. OPNET (www.opnet.com) actually provides a number of different utilities, namely VNE server which performs the network import, and IT/SP Sentinel which monitors stuff overnight. You can even run or create rules on the imported network in Sentinel using NetDoctor. There is also a plugin with OPNET that exports your network to a Visio diagram. For actual documentation, keeping everything on a Wiki or perhaps a Sourceforge Project would be a great way to go.
The OP makes a good argument because of the very way this "Ask Slashdot" started out. Let me rephrase the question and then you tell me if the OP is still overly cynical:
"When I started at this job, the network was a complete mess (compared to the way I would have set it up) and there was no documentation (not that I bothered to ask my predecessor) so I have spent the past X years straightening it out and making it run perfectly (according to my standards). Now, if I were to have to explain what I spent the past X years getting paid for, how do I write something that my successor will understand (since the only standards I followed, I made up as I went)?"
Honestly, there are plenty of logical, self-documenting proceses you can (and should) do if you are a remotely competent admin, that will result in a network that doesn't NEED a phonebook-thick document to understand. The first is a physical map of the building with connections annotated, and follow that up with a list of the critical pieces of hardware and the access passwords. This is something that makes the job easier whether you are starting or finishing at a site, and you should have little else to worry about so long as the rest of your network follows some sane technological standard (found in the first chapter of "blank networking For Dummies" where blank is Microsoft, Linux, Novell, etc.
However, if you fail to learn from history (as this Ask Slashdot poster has) you will certainly be doomed to reinvent it.
Document everything! I'm currently in the process of helping our systems admin document all of our servers, their purposes, all scripts running on them, specs, etc etc. We're trying to cover everything just in case one of us gets hit by a bus, we have everything documented. A lot of it seems pointless, but it could be useful to someone with little IT or Linux knowledge.
We just got a slower, old server and set up a mediawiki on it. Type all the info you need, upload png's of layouts, pdf's, whatever esle you like. We're urging our other departments to use the wiki and have everything up there, except root passwords... those stay in a safe.
Basic system documentation rules 1. Do not put doc on network.. so don't use wiki or all kind of CSM tools for that. network down = no doc. 2. Try to use scripts as much as possible to document your system and use CVS to keep track of change.. (automated, cron, report diff, etc) 3. Password and those stuff should be in a encrypted file (ok I know this is obvious)... 4. Keep track of past incident, problem, (internal and with vendors) (problem are recuring... so) 5. Avoid doing diagram. This is a real waste of time and they are only good for manager that want to show to other people how nice is the network in my cie. 6. Again... everything should be on your USB key (with a copy on the network).
if there's no one capable, and they're definitely not going to hire someone good (why would they?)
happened to me, was laid-off, basically because this clown of an admin thought he knew everything, was more expert than me.
the guy had one linux box at home, that was his claim to fame. me, 15 years enterprise experience, solaris admin for over 10 years.
yeah, he can do the job, screw him.
so, no docs for them, nothing, screw them. they'll never figure stuff out.
all my code, gone (it was stuff I had written before joining the company)
and they've screwed it up royally, oh well, theey're too stupid to know... i ain't gonna tell them...
How about a program called JPasswords? Or create a very simple secure php page with database of passwords. We had dozens of secure passwords (of various levels) and users could only see certain passwords. So your DBA can see all the DB passwords but not the domain/root passwords.
You probably already know what you need to document on your network - it will be whatever it takes to get things back up and running, and it depends not only on your operating system, but also on the hardware. I manage a R&D network of mostly UNIXes, and due to the R&D part of it, I have learned how to get by on minimal hardware and how to rescue systems wen they have been driven over the edge - again. There is no such thing as too much documentation, is what I've found, but it has to be well structured, easily accessible and easy to update.
There are some things you should obviously have written down: the physical and logical configuration of your network, the physical position of all hardware (which should always be clearly labeled), and administrator passwords should be written down somewhere safe, unless you have a password strategy that is easy for you to remember, but hard to guess for others (I'll elaborate a bit below).
Maybe it is just me, but I have found it absolutely necessary to put clearly visible labels on all machinery and make a list of them as well; when you have more than 40 servers plus all the little gadgets that form the understorey of a wild server habitat to keep check on, you just can't remember. And the idea that a serverroom should or could be a well-ordered place is naive, in my experience. I have a small label even on each network cable just in case.
The thing about passwords - some say that we should use passwords that are basically digitised, white noise, and that they should be changed every week or so; but who can remember that kind of things? So you end up with little lists and notes that are always out of date and are too easily forgotten in places where others might see them. I have done away with that - instead I use words I find easy to remeber, but which I have reasons to believe others won't. Some time last year it was words like the perhaps too well know "1337H4X0R", but recently I have found that there is a wealth of insanely obscure words to be found in older dictionaries for little know languages - it's a bit of a hobby I have; did you know that there are several hundred largely unrelated languages in Papua New Guinea alone? Not to mention Inuit dialects, Native American languages, etc etc. I haven't tried myself, but how fast could one guess a pasword like "umiarssualivinnguaq" ("small harbour" in Greenlandish)? It's probably not the first one most people would think of. The advantage is that these words have been used by real people; this means that 1) they have a real meaning, and 2) they are pronouncable (well, in principle), two things that make them easy to remember.
But I think the most important thing is not WHAT, but HOW you document. I have over the years evaluated a few documentation-/monitoring strategies, and the one I have settled on is in many ways the simplest: I use a Wiki and store all the home-made documentation there. It is simple to maintain and easy to access. Forget about MS doc formats and PDF or any other complicated format - a good Wiki is what you need, especially if it allows you to read it even from a text-only browser, because sometimes that may be all you have access to.
I was recently in a similar situation and I was the one who had to figure it all out due to lack of documentation. The main things that I did were to map the network and create updated diagrams. I did this by using a bunch of utilities both commercial and open source.
Create a list of UIDs and PWDs and maintain them in a program like PasswordSafe. http://passwordsafe.sourceforge.net/
Map all the switches using a program like netdisco. Depending on your equipment, it can find the uplinks and map the network for you. Otherwise, fill in the neighbor information on your own. http://netdisco.org/
Setup monitoring with Nagios and set the parent/child relationships using nagios. Make sure the map is accurate. Monitor all critical network services such as routers, dns, wins, email, proxy, fw, etc. http://www.nagios.org/
If you're not going to graph service data with Nagios, do it with Cacti. That will provide historical/trending data that is important for future network related decisions. http://www.cacti.net/
Create high level network overviews using Visio. Solarwinds LANsurveyor Express is very useful for automating network maps. http://www.solarwinds.com/products/LANsurveyorExpress/
Make sure you have good backup configs of all devices. A tool like Kiwi CatTools will automatically backup the configs for your devices and even alert you to when configs have been changed. It's great for change management purposes. http://www.kiwisyslog.com/kiwi-cattools-overview/
It really depends on the size and scope of your network. Currently, I run an ISP network, so if you are interested in documenting the infrastructure, I try my best to let the network document itself:
- RANCID for network device configuration
- different coloured cables for different purposes (with a legend on each rack, or on each device)
- Visio (or equivalent) online and printed documentation for router/switch interface connections
- Reverse DNS
- consistency with hardware and software versions/platforms (where possible)
- templates, so that common tasks are as copy/paste-able as possible
- information sharing. Write up a minimalistic report each month documenting an overview of your previous months efforts and give it to your boss. This will slowly but effectively create a documentation trail for change management, but it will get you in the habit of gauging your own performance
- make notes, even just silly quick ones. Most of the time, they are impossible to find later, but you know you wrote it down somewhere
- keep a personal blog and document periodically what you've recently learned or achieved. This not only provides a minimal amount of documentation, but it helps reinforce the experience gained
- USE THE DESCRIPTION FIELD wherever there is one. I find this to be one of the most effective methods of documentation
You are doing the right thing here. Even though you know that your company will 'cheap out' if you ever leave, documentation is the professional thing to do.
Good luck!
Steve
What is in your job description? Is documenting the network part of it? If it is, then clearly you need to document it as they are paying you do to that, even if you are to be outsourced. However, if it is not, then you should think long and hard about your options, and remember IANAL. 1) Document it at work anyways. 2) Document it at home making sure work does not even know it exists. 3) Do not document it at all. If you have a real fear of being outsourced, do number 2 with instructions that upon your death (i.e. the bus) it is delivered to your work. You could also hand it over when you leave of your choosing, this will help you with references in the future. If you are fired or outsourced, and asked about it in the future, the answer is "I normally would have, however it was specifically not in my job description". And as you didn't document it on company time, there is nothing wrong with that answer. I do not know the legality of the above, remember IANAL. I have all ways left companies because of my choosing and it has allways been part of my job description.
I dont care how you do it but for the sake of your replacement please do. I am a Network Administrator. Im pretty fresh out of college with little work experience. I got this job as Network Administrator and took it VERY happily and felt very lucky to have it (and still do). However the previous NA left with nothing but a "see ya later." About all I got handed was passwords and what they thought was on the servers. Not to mention the backups at every site were failing, WSUS was disabled, and the MAIN MAIN AD server was in total shambles and crashing daily which took a month to figure out. I started last decemeber and am just now starting to have things cleaned up. Please dont do that to anyone.
"I .. have been guilty of being too busy with the doing of it to document the changes and systems that were put in place. Now as I look back, I'm worried that I am the only one who will ever know how this network works. If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run"
Excuse me for saying so, but if you've implemented a system that is only understandable by your own good self, perhaps you're in the wrong profession. What is it about the system that required constant intervention and keeps you too busy to document it. From what I've seen, and I've seen a lot, most anyone would want from a system, is an exchange type server and a bunch of shared folders, and some kind of offline/mobile device. Most of which could be provided by a basic LAMP stack and a BlackBerry.
How to document, in each directory on the system that you are customizing, put a log file detailing the changes you have made. One line per entry as in: 'date: message: username'. That way, over time you will be able to track and document changes. You or anyone else for that matter.
davecb5620@gmail.com
There is something a bit disturbing in the threads related to passwords. Everyone talks like they have just one or two monolithic administrator / root passwords for a network of even moderate size.
I am curious. How many passwords do you have on your networks?
I run a relatively small network, but put different passwords whenever possible on diffrent systems. Yes, some passwords have more rights than others, but no one password is the complete "keys to the castle".
Just having one root password systems wide is like the administrator version of writing your password on a post-it next to your monitor. If someone can crack that password (or find it by accident), then they own everything. Do you really use the same password on a web server exposed to the internet, as a file server in the back office?
Make them work for it. If one system falls for whatever reason, they still need to crack their way in to the next system. Security with depth and layers.
By the way I learned to do this the hard way. When administrating several systems at once with multiple terminals open, I discovered that once in a while you might loose focus and execute the wrong command on the wrong system. Beyond security, it might just protect you from yourself.
Living in Chile
Networks are mostly just a complex program operating across a number of different machines. So, why not just create a flowchart of the network to describe its features and the create sub-flowcharts for machines connected to it?
Once you have a complete picture, documenting it should be easy.
8==8 Bones 8==8
More like 5 digit.
Nerd rage is the funniest rage.
The passwords required to run the network equipment at my site are ONLY written down. I have 2 notebooks, one here, one across the street in a different building, in physically secured rooms. The passwords exist in NO OTHER PLACE.
I'd like to see your remote exploit that steals them. Even your onsite attack is going to be tricky. In fact, why don't you tell me the password for our firewall? Good luck.
No wonder our field and many of our professions have such a bad reputation.
I have read only a few posts and two (moded up 5) say pretty much to ignore the issue.
In several networks I have worked with fundamental information was non existent. This translated in lost time, down time and actually losing money (if you lost your job in one of those companies recently, the indolent SAs or Network administrators may be partly to blame).
You never know who the next guy will be, if he is less experienced or capable then the documentation will be very valuable, if he is more experienced or capable then you would have saved their time to do some real work, after all they (and you) have not being hired to do forensics.
How a professional can hide behind the "let's be real" nonsense is beyond the pale.
Exactly my thoughts. I also was thinking "Boy, there are a lot of people in IT that just plain old don't want to do a fundamental part of their jobs".
But here's another incentive- when the higher-ups in their suits send over the outside IT consulting company, I think most IT managers would prefer that the report to the CEO says "Your IT guy is really good. Everything is well documented & we were able to quickly assess your entire network". Or perhaps you would like the consultants to tell the executive board "Sorry guys, we won't be able to get that report to you for another 6 months. Your IT guy has the network so messed up & nothing is documented, so you'll need to pay us a lot of money to do his job & figure out what you do & don't have. Let's start by recommending a new IT manager who understands how to document things".
Or perhaps next time the budget is up for review, you may need to justify an expense. Say for something like some servers or network equipment. Documentation tends to help get funds to your group.
Items necessary for good network documentation
1. Identification of servers, workstations, printers, routers, switches, etc.
a. IP addresses
b. NetBIOS/Host names
c. MAC addresses
2. Description of each device on the network, including make, model, serial number, and printouts from system inventory software (such as Belarc Advisor)
3. Network topology diagrams, including placement of servers, routers, switches, firewalls, IDS, etc.
a. Physical and logical diagrams
b. Layer 3 networking diagrams, including backbone and WAN links
4. Internet provider information
a. Description of link(s)
b. Contacts and support numbers
c. Terms of service
5. List of supported network operating systems (Win2K Server, NT4, NetWare 5, Linux, etc.)
6. List of supported client operating systems (Win2K Pro, Win98, MacOS, Linux, etc.)
7. List of supported network protocols (TCP/IP, IPX/SPX, AppleTalk, NetBEUI, etc.)
8. DHCP server settings, including scopes and options
9. Network security settings
a. Firewall configuration (including TCP and UDP ports open)
b. Router access lists
10. Troubleshooting history/administrator's activity log
a. Common problems and resolutions
b. Installation history
11. Network baseline information
a. Traffic flow and network utilization
b. Bandwidth utilization
c. Percent of collisions
d. Average server and workstation CPU utilization
e. Average server and workstation memory utilization
12. Fault tolerance mechanisms in place
a. Disk redundancy (e.g., RAID arrays)
b. Tape backup plan, including rotation and off-site storage
c. Clustering and failover systems
13. Physical location documentation
a. Building map
b. Room numbers
c. Availability of access keys
d. Unusual configuration information
14. Policies and procedures
a. Naming conventions
i. Workstations and servers (NetBIOS and host names)
ii. Network equipment (e.g., routers and switches)
iii. Active Directory
iv. DNS
b. Points of contacts (IT director, administrators, help desk, etc.)
c. Disaster recovery plan
i. Vendor phone numbers for support
ii. Remote access plan for administrators
iii. Higher-up administrator or consultant on call
iv. Virus prevention/recovery plan
d. Copies of maintenance plans, warranty agreements, and tech support contacts
e. Software licensing information
f. User rights policies, including Internet and e-mail usage
I put all of my passwords in a file on a USB drive locked with a password of the cto's choice. They are in an odf spreadsheet file encrypted with the same password. All cables, switches, wall, and punch ports are labeled. If someone takes over my position and can't figure it out from there then the cto got what he paid for.
Having to work for a living is the root of all evil.
No, this is the only way to create useful documentation (provided it's for "internal use only" and an unspecified readership). It's no use (re-) writing "network admin for dummies". There's only one thing worse than no documentation - outdated documentation.
Ok, lots of debates on what do do with passwords which doesn't address the OP's question.
Create an internal wiki and begin documenting. Add some high level flow charts for overall clarity.
Indicate that sensitive details like passwords are stored in such-and-such safe or whatever.
Provide links to the websites for any software being used or techniques / best practices.
This way the predecessor can easily add to this wiki/edit it when he/she takes over.
Check out http://www.ratemynetworkdiagram.com/ for diagramming ideas.
Doesn't work so well if decent encryption is involved.
Important data might end up inaccessible.
Or you might no longer be able to sign certificates that are used in the organization - you'd have revoke the main cert and create a new one.
that red stapler, ya, I'm gonna have to take that...
Things I would have liked to had on hand rather than having to make up / discover:
* subnets and addressing schemes, and the DHCP config doing it ... whatever there is
* yes, the commonly used shared admin passwords
* diagrams of the physical network and location of any wireless AP
* brief descriptions of critical services and the servers they run on
* ibid on applications, web sites, mail domains, fax farms
* patch / antivirus / backup schedules
* backup policy, procedure, media location and any other DR plans
* any user policy that's been circulated
* known exceptions to the rules
Hope that gives you some ideas. As a tech the format of your documentation can be pretty simple, so you can use your favorite tools ($EDITOR, wiki, dia, gimp ...) For your own respect or a larger organization a nice print out in a binder is pretty standard.
<script>alert("I never liked JavaScript, really; it just seemed a bad idea.");</script>
1) Assuming you have the telemetry stuff (inc. passwords) you can get the rest by pulling and parsing configs+Sh ver/sh Hardware etc. and sending someone around w/a cam to take pictures of all the devs in the racks to make sure you got them all, then config-control (using SCCS etc.) the configs+show commands and parsed database to track changes as they occur...easy stuff if have a PERL weenie....
You don't want someone like me coming in and looking at your documentation and declaring you incompetent, it can cost you your job.
What else would you say/do? "This documentation is great!! You don't need me at all! Hire a couple of freshly minted Microsoft admins for peanuts and Bob's your uncle." No, the natural tendency for you/your org is to discredit past efforts to maximize billable hours.
Don't take that the wrong way though. What you describe is necessary and valuable in some organizations. I've seen too many projects blow through scope/scale/cost estimates by simply discrediting others when it wasn't justified.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
I use a combination of a portable wiki/notefrog & network notepad. Network Notepad allows for a simple visual reference (with the CDP portion as well) and the portable wiki I carry on my flashdrive. I've started using Notefrog and one day may move all my data to it.
Open-Audit is a great tool for helping document your network.
I had to do that and below is a list of tools that I have used: 1. Splunk: Not only does this tool consolidate the logs but with the change management module, all changes to routers, switches and firewalls (Cisco devices) is tracked 2. Visio or a comparable product: Draw network diagrams 3. PINS or a similar application: Use it to store all your passwords in an encrypted file and also keep a printed copy of them in an envelope in the safe 4. SolarWinds IP Address Tracker or similar application: Scans your network subnets and keep a list 5. RackMonkey: Use it to document your assets in racks as well as their layout 6. Sydi for Windows server documentation: Use this tool to document all your servers 7. Alfresco, SharePoint or a comparable document management system: Keep all your server build, etc. in a common repository 8. Change Management: Implement a system for tracking changes and ensure it includes signatures. Would be worth it to either attach it as PDF to your IT Work Order system or to your document management system 9. IT Work Order system: Enter all your work in a work-order system and it would be helpful it is support file attachments 10. PC Inventory and license inventory: Implement a solution such as Lansweeper to keep track of all your PCs/laptops as well as software compliance 11. Perform audits to ensure everything is fine
While you could very well go about manually documenting every piece of the network, and hope it remains relevant and up to date in the future, this could take weeks and add significant overhead to your role in keeping current. I recommend looking into the many auto-discovery tools available from vendors like HP, BMC, Computer Associates, etc... They aim to store everything in a single database (CMDB) and track any changes or additions by scheduling delta-discoveries whenever you deem fit. The initial setup can be a lot of work, but since you know most of the information required by the auto-discovery tools for accessing system information (usernames, passwords, IP subnets, common services, ports...), it should be pretty straight forward for you. In a larger organization where this information is spread around various groups, it can be a lot more challenging.
HP has a product, formerly by a company called Mercury, that I find works quite well. It would at least be a good place to start looking... Link here. Good luck!
"You are not a beautiful and unique snowflake."...Tyler Durden
You are discriminating against people who are differently alive! You're such a speciecist!
Undead : Yes.
Un-person : No !
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
How a professional can hide behind the "let's be real" nonsense is beyond the pale.
Hey, I would *LOVE* to be wrong with "let's be real" except that you don't know how many times I've said: "You'll find it in the documentation I left for you" only to find that, despite me leaving VERY EXPLICIT INSTRUCTIONS about how important the docs are, with each element in the system identified and tabbed as such, only to find out that they NEVER BOTHERED TO MOVE IT from the spot on the shelf where I left it, even after reminding them that it's in the docs repeatedly.
You do this a couple of times, and then you discover that there's no point at all to producing concise, descriptive documentation that nobody ever reads.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
I would suggest you looking into a cable management system. These systems allow you to document the physical network and any moves, adds, and changes. The information can be documented in the application so it lives and is more accessible. http://www.bradyid.com/netdoc, ulticam (www.ulticam.com), to name a couple products. MarlonD
Every company seems to have a different definition of what a "Network Manager" is responsible for, so "documenting a network" is too vague a phrase. Are you referring to network equipment only, all of your IT infrastructure, or some middle ground? Furthermore, depending on the size of your infrastructure, it may make sense to consolidate or further break down the bits in to larger or smaller chunks (adding more detail or diagram/document X, or breaking diagram/document X in to three smaller diagrams/documents.)
In my experience, the best documentation is accurate documentation. Since most IT operations are in a constant state of flux, your documentation repository will need to be updated frequently (and these updates will have to become part of your company's work-flow if your documentation has any chance of keeping up). I've seen many systems come and go, and strongly suggest that you adopt a wiki of some sort. If you choose one with a changelog (most have them these days), I suggest that you grant write access to everyone you grant read access to. (Frequently the consumers of the information can make important/useful updates - let them!)
1) Physical Inventory - manufacturer, make, model, location (rack/u)
2) Network Diagram - Your network diagrams should include all network infrastructure (communication lines, switches, routers, SLBs, etc.). If you are responsible for the servers and storage as well, then include them too. For each piece of equipment, include hostname(s), IP(s) and the high-level function (web/db,mail,etc.). I usually use Visio for this, but finding a wiki that supports network diagrams natively would be even better.
3) Service Notes - Make notes for each "service" available on the network.
Example:
Service X
-Service Overview
>Big picture overview - what does this service do?
>How important is this service (think outage)?
>Who is responsible for this service?
-Hardware List
>list of all machines that make up this service
>machine's role/sub-role/hostname/IP/console port/etc.
-Access Notes
>include information about how to access (log in to) the machines
>(physical access, console access, network access)
-Start/Stop Notes
>information about service dependencies, start/stop scripts, etc.
-Installation/Build Notes
>include hardware type, OS, list of packages to install, config files, etc.
-Service Monitoring
>notes about what is/should be monitored, and where to find logs/etc.
-Service Testing
>how to test each service/sub-service
>include notes about required test accounts/etc.
-Backup Notes
>backup instructions/details/notes/etc.
>recovery instructions/details/notes/etc.
-Other Notes
>include information about common problems, planned upgrades, etc.
4) Passwords - There should be a physical copy locked in a safe somewhere, that a select few have access to. The specifics will depend on your organizational structure, but the post by christianT about sealing them in an envelope is right on track.
5) Escalation Procedures
-List of all services, and their associated priority, and escalation procedures
6) Disaster Recovery - whatever one needs to know to restore service in the event of a disaster
I'm sure that I forgot many things... this was just a quick brain-dump.
- phrend
Take a look at the site: http://www.network-documentation.com/
It has documentation templates from a now defunct site called networkdna.org
I use these documents to document the various client networks that I'm responsible for, and they work quite well. You can modify these to fit your own needs.
Quit playing Monopoly with Bill.
Linux - of the people, by the people, and for the people.
Disclosing and securing all those admin passwords is certainly an important part
of what you have to do, but it's by no means the whole problem.
You can read about privileged password management - which is really what you're getting at:
here: http://en.wikipedia.org/wiki/Privileged_password_management
or here: http://id-archive.com/docs/privileged-password-management.pdf
The bigger problem is the network diagram -- what subnets are out there? What servers are on them? What do they run? What services do they provide? Who/what consumes each service? That's a lot of information and it's constantly changing.
You not only have to capture it all, but you should keep updating it as you make changes to the systems. I think
someone mentioned a wiki in this thread, and that's an excellent approach - low barrier to making updates.
If you're a masochist, you can always try and follow the DoD Architecture Framework which defines multiple views of architectures (including networks). Once finished, there shouldn't be any question of what your network is, what it does, and how it does it, but you'd probably need an army of peons to put it together.
I do security
Just draw a neat map.
There is no way to cheat towards useful documentation. I think there are two rules:
Rule#1 - Please, in the name of all that is Holy, do *NOT* create a *^&@( Wiki. Write documentation. A Wiki is *NOT* documentation.
Rule#2 - Learn how to properly use M$-Office or OpenOffice. Seriously, this will make create and maintaining good and thorough documentation. In OpenOffice (sorry, don't know a darn thing about M$-Office) you can create a master document (ODM) that aggregates other documents and that will automatically create a table-of-contents, index, etc... if you use the styles correctly. It is *WONDERFUL* to use. And each section as its own file is much easier to maintain that one MASSIVE document and the auto-indexing makes it pretty easy to look stuff up (which also very much helps in keeping docs up to date). I have no doubt that M$-Office has the same features packaged somehow.
Documenting, especially creating documentation someone can actually use, is just plain hard work.
Using "Common Sense" is being either to arrogant or to ignorant to ask people who know more about something than you.
Don't network monitoring tools usually have good ways of drawing the physical layout out the network? We use Nagios and InterMapper. InterMapper uses SNMP to gather more information about each device and both provide a view of the physical connections.
Some of the busier networks get crammed on the view, but I think the "crammed view" is a necessary evil for the more saturated subnets.
The things I have found difficult to recreate when taking over an existing network are: passwords; especially for routers, firewalls, VPN's, websites, FTP accounts and such; hardware manuals for older servers and equipment, licensing information for absolutely everything (BSA may come a'knockin); hardware histories, warranty info and anything that isn't set up in a standard way. I have seen some networks where the previous consultant created some of the most convoluted, cockamamie scripts imaginable, yet they worked... now good documentation for those would have saved me more than a few headaches and sleepless nights.
Homestead is a 3D real-time network visualizer, displaying hosts & packet traffic. Features: multiple sensor support, gather hostnames & services, configurable subnetwork layout, record/replay packet traffic, filter packets by host, protocol or port.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
i thought so!
Ask Me About... The 80's!
I actually called my network documentation this. The first paragraph is an example of some of the horrible things that could happen to me, and then it goes on to explain what all the machines do :P
The network diagram is done in inkscape, with much zooming -- at 100% you see a network overview; if you zoom in far enough on each box you can see all kinds of notes about it (services, IP addresses, etc.) in very tiny text. And when you're trying to find something ("Which goddamn box has 198.222.40.6 and since when does it do mail??"), inkscape lets you search the text.
As for passwords, my predecessor kept a meticulous password list that he didn't make accessible when he left. Mine's not very accessible either, but the manual has a page that explains how to get it if I die.
Isn't that tautological?
Anyway, we did most of our network via Visio (it had some search and document sort of function for mapping it). Not sure if there are any Linux tools like that.
I currently am an assistant to the systems manager at a department in my university, and we document stuff all the time for the very reason you bring up. Really, the main things to document are: - Server setup/maintenance - Network layout/settings - Solutions for important/frequently occurring issues - If you install computers using an image, document how this is done (we use Ghost where I work) - Descriptions of software used on network clients There are many other things that you can document, of course. But the above list contains what I consider to be most crucial.
I documented, inviting everyone in the department to join me in documenting their work, until I started documenting myself out of job. My successor, reading my documentation, thought I was brilliant. (Understandable.) I was often accused of being too communicative in emails, i.e. too many emails telling people what was going on. I remain mystified as to how one could be "too informed" about what was going on with the network. I never missed my project deadlines.
Bottom line: Documentation is good, if anyone cares.
If you went to your boss and said here is how to replace me, would he. if so document the network and keep it in paper form in the bottom door of your desk marked if your reading this iv been killed or fired
Technology will default in society to its most rudimentary level:::stupid computers for stupid users:::