Slashdot Mirror


Documenting a Network?

Philip writes "Three years ago I was appointed as a network manager to a barely functioning MS-based network. Since then I've managed to get it up and running — even thriving — but have been guilty of being too busy with the doing of it to document the changes and systems that were put in place. Now as I look back, I'm worried that I am the only one who will ever know how this network works. If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run. Ultimately, this won't be a good reference for me if they are trying to work out technical details for years to come. It looks like I'm going to have to document the network with all sorts of details that outside consultants could understand too (no, I don't want to be the outside consultant), especially since it's likely that my replacement will have less technical expertise (read 'cheaper'). Are there any good templates out there for documenting networks? Is anyone who has done it before willing to share some experiences? What did you wish your predecessor had written down about a network that you inherited?"

109 of 528 comments (clear)

  1. I know... by EdIII · · Score: 5, Funny

    What did you wish your predecessor had written down about a network that you inherited?

    The Passwords.

    1. Re:I know... by Drinking+Bleach · · Score: 5, Insightful

      Whether the comment was intended to be funny, I find this to actually be a serious issue...

    2. Re:I know... by 2Bits · · Score: 5, Interesting

      This may sound funny, but I recently had the same experience. I took over the position of CTO of an electronic payment company, and after one week, I figured a lot of critical systems are missing root password, including Linux, AIX, HP/UX and SCO Unixware. No one knows the password, it's been changing hands so many times, and the people who were responsible for those machines have left, without leaving the passwords behind.

      Those are critical systems that must run 24x7. We had to rebuild the system on new machines, re-route transactions to the new machines, and shutdown the old ones to recover (single user mode).

      And that's a platform handling over 400 billion in transaction per year. Scary. But that's the easiest problem I have inherited, mind you.

    3. Re:I know... by Architect_sasyr · · Score: 2, Interesting

      I note that GP was very careful to mention "systems" as a plural a lot. There is a cluster near me at the moment which nobody knows the root password to (until a recent local kernel root exploit at least) because of a very similar situation. Thankfully we got them back with a lot less trouble than this one, but there it is.

      --
      Me failed English...
      FreeBSD over Linux. If my comments seem odd, this may explain...
    4. Re:I know... by Anonymous Coward · · Score: 4, Funny

      So that's what happened during the bank collapse!

    5. Re:I know... by stonedcat · · Score: 3, Insightful

      At which point you won't be getting the passwords anyway since you fired him/her? Good idea. ^_^

      --
      You can't take the sky from me.
    6. Re:I know... by somersault · · Score: 3, Insightful

      So what happens if said predecessor gets hit by a bus, has a heart attack or a stroke and can no longer tell you the passwords? Or, worst case scenario, the whole IT team gets taken out in a road accident on the way to a team building session for example? I've read a few "deserves to be fired" comments on /. and usually tend to agree (or occasionally get embarrassed because I think hmm, that's me!), but in this case you are being a fool.

      Of course, if you are dead then you won't care if they have the passwords, but some of us actually like our places of work and even our colleagues, and want our place of employment to be able to chug along even in our absence.

      --
      which is totally what she said
    7. Re:I know... by Nutria · · Score: 5, Insightful

      If the predecessor does write the passwords down, he deserves to be fired.

      That's knee-jerk stupidity, and you should be ashamed of your non-thinking fundamentalism.

      Passwords do need to be written down, and stored in "escrow". I put the list of passwords in an envelope, lick seal it, sign and date the seam, and then seal it again with clear packing tape. Give it to the boss to put in his safe.

      Yes, it's easy to open, but you'd know whether someone tried to tamper with it.

      --
      "I don't know, therefore Aliens" Wafflebox1
    8. Re:I know... by c0y · · Score: 3, Interesting

      So what happens if said predecessor gets hit by a bus, has a heart attack or a stroke and can no longer tell you the passwords?

      Boot single user and reset them?

      Yes, I realize that doesn't scale, but it works fine in a lot of small environments.

    9. Re:I know... by drolli · · Score: 4, Funny

      Check the top drawer of the your desk where the paper clips are lying normally.

      Oh? It is locked and you dont know where the key is?

    10. Re:I know... by Jurily · · Score: 5, Funny

      That's knee-jerk stupidity, and you should be ashamed of your non-thinking fundamentalism.

      I just assume that any event capable of destroying my ability to transmit said passwords to my successor also destroys any ability to give a damn about my job. Problem solved.

    11. Re:I know... by AmiMoJo · · Score: 2, Funny

      No no, you want to keep everything in your head, that way you can't be replaced by someone cheaper!

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    12. Re:I know... by noundi · · Score: 2, Insightful

      Here we run mainly Windows anyway, the only password that is really important to keep safe is the domain administrator password.

      Nope, local admin password on the PDC is far more important than domain admin.

      --
      I am the lawn!
    13. Re:I know... by Anonymous Coward · · Score: 2, Insightful

      If you actually *need* the password to the root account, UR DOIN IT RONG. Seriously. You should be sudo'ing everything.

      So to get yourself in the wheel group, you might initially need the password of another sudo'er, but you don't need root. I like to randomize (and forget) my root password after adding my own account to wheel.

    14. Re:I know... by somersault · · Score: 3, Informative

      How exactly do you get a local admin account on a domain controller? I didn't think there was any such thing.

      --
      which is totally what she said
    15. Re:I know... by ta+bu+shi+da+yu · · Score: 5, Funny

      If you think that is ironic, then note that the story poster is worried his ability to get a job if he dies in a bus accident.

      --
      XML is like violence. If it doesn't solve the problem, use more.
    16. Re:I know... by sumdumass · · Score: 3, Interesting

      Or how about a new CTO pissing both admins off and one walking out as the other was fired for no good reason.

      I've had to walk in behind something like that several times and reset the passwords or load the password hashes into some cracker in order to find the passwords. A lot of times, you can pull them from workstations the old admins used and they are easier to crack then the newer MS servers. The funniest thing is that the CTO usually wants me to stay on full time and gives me a dirty look when I won't do it because he made life so miserable that two other people walk off the job under his supervision.

    17. Re:I know... by plover · · Score: 4, Interesting

      Yes, it's easy to open, but you'd know whether someone tried to tamper with it.

      Try spraying the envelope with refrigerant. The paper becomes translucent when wetted and you can sometimes read what's inside, and then it dries without a trace (unlike wetting it with water, which swells up the paper fibers leaving the telltale signs of tampering.)

      Learned this one from a history of the U.S. Black Chamber.

      --
      John
    18. Re:I know... by whiting · · Score: 2, Insightful

      Think about what you would want to know if you walked into your network cold. (don't document the passwords, but make a provision for passing them on).

      Don't document every detail. Point to reference material, but you shouldn't be documenting theory, just your implementation.

      Also, I hate to take the pessimistic view, but based on my experience, even if you do a stellar job, there's only a small percentage chance that anyone but you will read the document. But you can sleep well at night knowing that they can figure out your handiwork (network design) from your writting if they do decide to RTFM.

    19. Re:I know... by ultranova · · Score: 4, Insightful

      If the predecessor does write the passwords down, he deserves to be fired.

      Either he writes the passwords down, or he uses weak passwords that a human mind can remember.

      Besides, a password is a security token. A piece of paper or a little plastic card with the password printed on it or a USB stick with SSH key or whatever saved on it are also security tokens. They aren't inherently less secure than memorized passwords; you simply have to secure the physical object by, for example, locking it in a safe.

      "Passwords should never be written down" is an idiotic rule, right there with "never use goto".

      --

      Forget magic. Any technology distinguishable from divine power is insufficiently advanced.

    20. Re:I know... by Anonymous Coward · · Score: 5, Insightful

      if they can't replace you they can't promote you

    21. Re:I know... by Minwee · · Score: 5, Funny

      I just assume that any event capable of destroying my ability to transmit said passwords to my successor also destroys any ability to give a damn about my job. Problem solved.

      So what was it like working for the City of San Francisco, anyway?

    22. Re:I know... by theSpitzer · · Score: 2, Insightful

      You're a fucking retard. No one uses Windows in a server environment.

      Except for entire Colleges and many Universities. Other than that, and a million and a half other places, you're correct.

    23. Re:I know... by llamapater · · Score: 2, Insightful

      does no one on this site know how to restart a machine off a bootdisk and zero out a password with vi seriously =/

    24. Re:I know... by demonlapin · · Score: 2, Interesting

      So write in silver paint pen on black paper.

    25. Re:I know... by Amouth · · Score: 2, Informative

      do remember that the Directory Restore password is diffrent than the domain admin password..

      from directory restore you can change the domain admin.

      --
      '...if only "Jumping to a Conclusion" was an event in the Olympics.'
    26. Re:I know... by damien_kane · · Score: 2, Interesting

      Boot single user and reset them?

      If the password can be worked around, what does it matter if it gets written down in the first place?

      Because if the password is written down, then in (most) windows shops you can login remotely as administrator (or get administrative rights while logged in as a non-priviledged user).
      GP did say Boot single user, which entails physical access to the system.
      While effectively any system to which there is network access can be compromised (eventually), every system to which you have physical access can be compromised (and that, rather trivially).

    27. Re:I know... by drinkypoo · · Score: 2, Interesting

      If you are primarily a Linux shop you might use Coda or NFSv4, but if you still have Windows clients you might also run Samba. Actually, Samba can serve CIFS and not just SMB; CIFS has extensions for Unix features. So it's not a bad choice for serving files to Unix, but it's no NFSv4. (In some ways, that is a good thing.)

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    28. Re:I know... by Acer500 · · Score: 4, Insightful

      Something I found to be VERY good policy that was implemented at my former position (as Network Admin) was to hand to the boss (CEO, CIO or whatever) a sealed envelope with EVERY relevant password (most importantly the admin password :) ), to be held in the company's vaults (if your company has such a thing of course, or similar).

      Whenever an important password was changed, I would hand over the new envelope :)

      --
      There are three kinds of lies: lies, damned lies, and statistics.
    29. Re:I know... by v1 · · Score: 3, Informative

      If the predecessor does write the passwords down, he deserves to be fired.

      You can't always take it for granted. I've been on the cleanup-end more than once. Sometimes it's "engineered job security"/"they don't dare fire me", sometimes it's "I don't have time for that", sometimes it's just plain forgetting about a piece of hardware you set up the first week you started work there, and sometimes it's a legacy password issue. ("nobody's been able to login to that box since Phil left in '05") The rare treat is finding a mystery box that nobody knows what it does nor has any idea how to login to it. Too many managers don't understand the danger and consider it a waste of time or a bad risk to try to fix problems like that.

      Sometimes it's an uphill battle when taking over, too. You want to document the settings on all the routers, but two of them are "legacy" password issues, and the router only supports hard reset (clears password, AND all settings) so you can't get the settings from it once you reset it, and you need the settings from it in case it gets reset. That's never fun, but you will find yourself in that catch-22 occasionally, and it's hard to blame someone for not fixing it because it's utterly unpleasant to deal with. (hint: get another router and program it how you think the mystery box is set up. swap. test. immediately swap the mystery back in. adjust the settings on the new one and test. Swap back out. repeat until you get it right, and don't reset the mystery immediately, keep it onhand for at least a month in case some uncommon thing requires a setting you haven't yet discovered)

      Occasionally you can get lucky - contact the hardware vendor and see if the box has an undocumented soft reset. ("open it up and short together the two pads left of D-15, and you can login for 5 minutes with no password")

      --
      I work for the Department of Redundancy Department.
    30. Re:I know... by ThrowAwaySociety · · Score: 2, Informative

      Yes, it's easy to open, but you'd know whether someone tried to tamper with it.

      Try spraying the envelope with refrigerant. The paper becomes translucent when wetted and you can sometimes read what's inside, and then it dries without a trace (unlike wetting it with water, which swells up the paper fibers leaving the telltale signs of tampering.)

      Learned this one from a history of the U.S. Black Chamber.

      Does anyone not use "security" envelopes anymore? (The ones that are printed with a dense line pattern on the inside?) I didn't know they sold plain envelopes anymore, except for greeting cards.

    31. Re:I know... by BattleApple · · Score: 2, Informative

      Someone in this thread mentioned password safe from sourceforge.. I just installed it, and it's pretty nice. If you double-click an entry, it copies the password to your clipboard, so the only time you have to type it is when you first enter it. You can also view a subset of a password as a reminder

    32. Re:I know... by christianT · · Score: 3, Insightful

      I was taught that the proper way to store admin passwords for an emergency situation (such as admin gets hit by bus) was to
      1. Write each password on an individual piece of paper
      2. Seal each piece of paper in an envelope
      3. Store the envelopes in a Safe/Deposit box that a limited number of people have access to, (company owner, CTO, CEO, IT Manager)
      4. Put policy in place that requires passwords to be changed and re-recorded any time the envelope it is stored in is opened.

      It may seem low tech, but it is probably the best solution for a small to medium sized business.

      That's my USD$0.02

    33. Re:I know... by harp2812 · · Score: 2, Insightful

      Wouldn't a password vault or encrypted text file on a non-networked pc be much simpler? (Not to mention more likely to actually be updated on a regular basis?)

      --
      I've found that nurturing one's Zen nature is vital to dealing with technology. Violence is pretty damn useful too.
    34. Re:I know... by flydpnkrtn · · Score: 3, Informative

      Reboot the DC into "Directory Services Restore Mode" (this is on Server 200 and above) and the local Security Accounts Manager is used again, not AD.

      This is actually a way of resetting the admin password on a domain, if you ever need to. Boot from the Linux password reset CD (http://home.eunet.no/pnordahl/ntpasswd/bootdisk.html), blank the Administrator password (which resets the _local_ admin password), then reboot into DS Restore Mode. Log in and then copy cmd.exe over on top of logon.scr and reboot into "normal mode" (with AD functional).

      Wait until the "screensaver" pops open and you have a command prompt that just opened that's running with SYSTEM privledges. From there you can run mmc.exe (or whatever else you like).

      Oh and of course, this is exactly why you don't allow physical access to servers....

      In reference to the PDC statement, I'm not aware of any way to use local accounts on NT 4 or below, but I think he meant just DC instead of "PDC."

    35. Re:I know... by Mr.+Slippery · · Score: 2, Insightful

      Wouldn't a password vault or encrypted text file on a non-networked pc be much simpler?

      Since when is a computer -- subject to various sorts of software and hardware malfunctions -- much simpler than a freakin' piece of paper in a locked box?

      --
      Tom Swiss | the infamous tms | my blog
      You cannot wash away blood with blood
    36. Re:I know... by legirons · · Score: 4, Funny

      If you think that is ironic, then note that the story poster is worried his ability to get a job if he dies in a bus accident.

      maybe he could work as a voter in Florida?

    37. Re:I know... by Kompressor · · Score: 2, Informative

      Another nice way to get a SYSTEM privileged command prompt is via AT:

      Microsoft Windows [Version 5.2.3790]
      (C) Copyright 1985-2003 Microsoft Corp.

      C:\>whoami
      DOMAIN\administrator

      C:\>time /t
      01:05 PM

      C:\>at 13:06 /interactive cmd
      Added a new job with job ID = 1

      Wait for a minute, and a new command window will pop up.

      Microsoft Windows [Version 5.2.3790]
      (C) Copyright 1985-2003 Microsoft Corp.

      C:\WINDOWS\system32>whoami
      nt authority\system

      C:\WINDOWS\system32>

      This will work on XP and Server 2003 for sure, and doesn't work on Vista. Probably also doesn't work on Server 2008. Depending on how effective your domain administrator is, you might even be able to do this from a regular user account. Be careful who you let run the AT command ;-)

      Have fun!

      --
      kmem russian roulette: Aquillar> dd if=/dev/urandom of=/dev/kmem bs=1 count=1 seek=$RANDOM
    38. Re:I know... by Sancho · · Score: 2, Insightful

      Someone will still need the password to decrypt the store.

      Public-key cryptography to the rescue!

      You can encrypt the file in such a way that any of a set of private keys can decrypt it. You can also encrypt it in such a way as that a combination of keys are needed (i.e. 5 keys used in encryption, and you must have 2 in order to decrypt.)

      You can also send that machine's logs to all 5 owners, so that an intrusion by any one will be more likely to be noticed (and it will be harder to tamper with the logs.)

      You could probably even automate the entire encryption process. Any time a password changes, automatically re-encrypt with the public keys and store on the server. That way, you remove the human element, which could screw up encryption or signing, or forget to update the file.

    39. Re:I know... by slamb · · Score: 3, Insightful

      Public-key cryptography to the rescue! You can encrypt the file in such a way that any of a set of private keys can decrypt it. You can also encrypt it in such a way as that a combination of keys are needed (i.e. 5 keys used in encryption, and you must have 2 in order to decrypt.)

      Didn't Bruce Schneier once say "If you think cryptography can solve your problem, then you don't understand your problem and you don't understand cryptography"?

      I don't think this solution is nearly as good as the envelope system. In particular, I don't expect the company owner or the CEO of the average small to medium business to have the expertise keep their private key files and password accessible to them and no one else. That means that either you're actually relying on two out of the other three keys being available (if those two fail to keep their private key files accessible to themselves) or you have poor security (if those two fail to keep their private key files accessible to no one else), whichever is worse.

      In contrast, all of the trusted people probably know how to keep a physical key reasonably secure and accessible, and if you use a deposit box at a bank there may be identity checks as well.

      In general, if your solution requires detailed technical knowledge from non-technical people, your solution is broken. Pick something low-tech instead.

    40. Re:I know... by slamb · · Score: 2, Informative

      Frankly, I wouldn't expect the CEO to have the keys. I'd expect it to be largely in the hands of senior IT staff.

      Sounds like that's the root of our disagreement - when I think of a small business, I think of one that doesn't have five IT people, much less senior IT people.

    41. Re:I know... by mysidia · · Score: 2, Informative

      Documenting your network is not about teaching people how to configure X vendor devices, it's about telling them the identity of X vendor, the identity of the devices, how they are in general configured.

      I.e. what are the statically assigned IPs go to, which servers have which roles, which server is DHCP, which server is DNS, etc, what subnets exist, whose on them, what each device actually does on the network, e.g. why isn't it in some closet or in the trash can instead of plugged in and running?

      And then there are things like design and planning... if you want to allocate a new subnet, where should it (as a matter of planning) start?

      What's the special procedure to do X ? e.g. in the case of adding a subnet for a new department, the documentation might indicate things like which DHCP server should be used, how DHCP should be configured, should a certain existing server be used (with dhcp relay), or should a new server be racked up, in general...?

      Documentation also generally includes diagrams, things like visual maps that serve as a guide to fully understand how things are setup and working, AND to understand in a manner that permits an understanding of what changes might need to be made...

      + ability to make new changes smoothly.

  2. Do what the guy before me did by TornCityVenz · · Score: 5, Funny

    Use Post-its.

    --
    I Need someone to rebuild a Digitech Digital Delay pedal for me....for me...for me...for me.
    1. Re:Do what the guy before me did by FooAtWFU · · Score: 4, Insightful
      The nice thing about post-its is that they can be updated very easily when you're fiddling with the device. That doesn't work if you're configuring it remotely, though. :|

      The next step up from a Post-It, though, is a snazzy label-maker. My portion of the company uses these extensively to document our development lab (we do some NMSy stuff). Of course, it's not a production network, and standards are a little different.

      --
      The World Wide Web is dying. Soon, we shall have only the Internet.
  3. Good News by N3Roaster · · Score: 5, Interesting

    Your successor will never find any documentation that you leave behind (or if you show it to them they won't bother with reading it) and by the time they notice it they'll have already screwed things up to the point where the documentation will be obsolete. This means you can save yourself the trouble of doing the documentation unless that documentation is going to make you more effective while you're there.

    --
    Remember RFC 873!
    1. Re:Good News by BrokenHalo · · Score: 5, Insightful

      ...or if you show it to them they won't bother with reading it

      This is more to the point. Most network admins have the attention span of a flea and won't read past the first sentence of anything you write; actually, I could probably expand that statement to include most people generally. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

    2. Re:Good News by MaskedSlacker · · Score: 5, Funny

      You just thought I wouldn't catch a reference to Cicero's De Finibus Bonorum et Malorum.

      Original Lation from which it was derived: ...neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet, consectetur, adipisci[ng] velit, sed quia non numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel illum, qui dolorem eum fugiat, quo voluptas nulla pariatur?
              At vero eos et accusamus et iusto odio dignissimos ducimus, qui blanditiis praesentium voluptatum deleniti atque corrupti, quos dolores et quas molestias excepturi sint, obcaecati cupiditate non provident, similique sunt in culpa, qui officia deserunt mollitia animi, id est laborum et dolorum fuga.

      English:

              Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find fault with a man who chooses to enjoy a pleasure that has no annoying consequences, or one who avoids a pain that produces no resultant pleasure?
              On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain.

      What was that about my attention span?

    3. Re:Good News by Tiger4 · · Score: 5, Funny

      ...fellatio uber alles...

      WTF???

      --
      Behold, this dreamer cometh. Come now, and let us slay him... and we shall see what will become of his dreams.
    4. Re:Good News by GeorgeS · · Score: 4, Funny

      Not if they did their jobs right :)

      --
      "I'd rather have a bottle in front of me than have to have a frontal lobotomy."
  4. get some help by Jean-Luc+Picard · · Score: 5, Insightful

    Sounds like a very easy way to over work and over stress your self, get some help one way or another. Summer is coming and I'm sure there are plenty of Comp Sci/Network Engineer/IT students that could of help. It may not be a bad idea if you make a plan of some kind before you go head in.

  5. Documenting teamwork by Anonymous Coward · · Score: 3, Interesting

    1. Simply begin documenting, its a work in progress..never finished. 2. Select a worthy staff member from your org, with the brain cells (and desire)to start learning networking, and begin to train him/her on what you are documenting. 2.a refrain from selecting the network thinks-he-knows-it-all type, instead pick anyone else with the skills listed in 2.

  6. Alternatively... by Anonymous Coward · · Score: 2, Interesting

    Professionalism be damned, we're in the middle of a recession and you're wanting to make yourself dispensable? Go ahead by all means, but pass your boss my number :)

    1. Re:Alternatively... by cyber-dragon.net · · Score: 4, Funny

      English only please... I don't understand this word "vacation" you have mixed in with the others.

  7. What about? by DaMattster · · Score: 4, Informative

    A good tool like dia which can allow you to create a network diagram. When it comes to documenting a network, a picture can be worth a thousand words. Or you could also use MS Visio as it is, perish the thought, a good tool. A good, detailed diagram can come in very handy as a reference tool for your own use in case of a failure.

    1. Re:What about? by drachenstern · · Score: 2, Informative

      In other words, the tool you recommend is Etherape?

      Any other tools you would use to grab a snapshot?

      @the submitter, what about a tool like spiceworks?

      --
      2^3 * 31 * 647
    2. Re:What about? by operator_error · · Score: 2, Informative

      Etherape is gorgeous. I wish I had time to understand what it all means, but still I love firing it up and staring at the network traffic.

      Another tool that I've enjoyed much more personal success with is cheops-ng. For documenting a networking, cheops-ng and a decent icon collection provides a pretty snazzy view of what NMAP can see. (NMAP being another tool I haven't had time to fully grok, yet)

      http://cheops-ng.sourceforge.net/screenshots.php

    3. Re:What about? by RiotingPacifist · · Score: 4, Insightful

      A summary diagram (dia/visio) is still pretty key, but nmap (or specifically the latest versions of zenmap (the gui frontent to nmap) provides a good detailed view of network topology and if you include the capture file, you can annotate specific computer details in the notes section (ofc you have to assume your replacement will be familiar with zenmap for that to be useful though)

      --
      IranAir Flight 655 never forget!
    4. Re:What about? by xSauronx · · Score: 2, Informative

      has anyone ever used The Dude?
      http://www.mikrotik.com/thedude.php

      i just got a work-study job with the campus adming at the community college i attend. hes been there almost a decade and has no network monitoring system, so he has no idea when something goes down until he gets a complaint or cant get something to work himself. i thought an interesting project during my time with him might be to see if hed let me help implement a network monitoring system, and ive seen a couple of people use The Dude before and it seems pretty capable.

      --
      By and large, language is a tool for concealing the truth. -- George Carlin
  8. schematics by ClaytonianG · · Score: 5, Informative

    Basic network documentation:

    I've found that starting out with the very basic physical layout and working your way up in complexity is greatly beneficial.

    i.e. start out documenting network cable runs including cable type. follow it by switch layout. follow that by routers and vlan setups. follow that by the servers that provide basic network functionality(e.g. DHCP, etc...). If this is a windows network, that would likely mean detailing the domain controller setups. From their systematically document the systems in order of importance to the business, etc...

    Also, visual diagrams are extremely helpful.

  9. Let's be real by mcrbids · · Score: 5, Interesting

    Short answer: don't worry about it too much. Put together enough that it looks like you've done something then go have a beer.

    You could have the most amazing docs the world has ever known - with passwords and clear instructions - ad the odds are about 20% that the next guy will even read them.

    The next guy will figure that he/she knows much more than you as evidenced by the fact that they are there and you are not. And, the cheaper they are (read: inexperienced) the more likely this is to be the case. When things go wrong, they will blame you anyway.

    So document away, but for YOUR sake so that if/when you are called in after the new guy horkens everything, you can have an easy time putting it all back together. But don't wait for the call... people will put up with almost anything when pride is on the line.

    And go have a beer.

    --
    I have no problem with your religion until you decide it's reason to deprive others of the truth.
  10. Start with disaster scenarios by bjcopeland · · Score: 5, Informative

    For me, visio's are great and everything, passwords too, but really the most valuable thing you can do is document single points of failure, outdated software/hardware, etc., license keys/expiration dates, cert expiration dates, personal support contacts you have and all vendor relationship details as well are essential. Do you use change control? If you do, go back and comment your changes, if not, do the best you can at explaining why things are the way they are. Get some open source software that is good at indexing data and create a searchable knowledge base from the information above. Don't concentrate on docs that can be found on the web at first because any admin worth their salt will know where to look for how to's, etc. Focus on the why's, the where's and disaster recovery.

    My two cents...

    1. Re:Start with disaster scenarios by enoz · · Score: 3, Informative

      Get some open source software that is good at indexing data and create a searchable knowledge base from the information above.

      A Wiki?

  11. Here is what I would get by Anonymous Coward · · Score: 5, Informative

    1. Viseo overview of the network drawing with complex areas drawn out specific detailed viseo's (even a scanned sketch or paint drawing is better than nothing)
    2. A spreadsheet with circuit ID's mapped to router and interfaces.
    3. Document the trunk interfaces as well as the LAG's (Link Agrregation Groups, port channels, whatever you want to call it)
    4. TACACS passwords / domain logins in a secure location (or radius or diameter or whatever you use)
    5. Data center capacity as a function of 1. Rack Space, Cooling Capacity, Electrical Load.
    6. Write brief knowledge articles describing any problem areas and explaining a history of anything you think would be hard to figure out easily. No need to go hog wild, just re-brand the RCA documentation you have. You do have Root Cause Analysis right?
    7. Network protocol hierarchy map. Where are your major redistribution points, what is your routing strategy etc.
    8. If you have a voice network document all your DID's, PRI trunks, Gatekeepers, Dial Plan, and any translations you use on h.323 gateways or how MGCP or SIP is configured. If you have a complex call center you should probably pay to have it professionally documented in down to the minute detail.
    9. SSID's and BSSID's for any wireless you may have as well as passwords, 802.1x authentication methods, along with linking documentation.
    10. Make the documentation part of your CMR process (Change Management Review) and incorporate it into the time allotted for a change.

    I know these are just rough ideas and you should get many more ideas from all the smarter people on here than myself, but whatever advice you get I would say you would need to have the documentation update able via subversion, or some document control system and have some kind of review process for it, even if it means getting together over pizza with some of the other groups and asking them about their environment and getting pointers and possibly help on documenting it all. Documentation is a full time effort and IMHO there is no such thing as too much documentation. You would be surprised how good documentation can aid you in problem resolution down the road or aid vendor support in helping you resolve a major outage. The three basic principles of network care are document document document. :)

    Cheers,

    Anonymous Coward.

    1. Re:Here is what I would get by Hecatonchires · · Score: 4, Insightful

      circuit ID ? in that a farmer's synonym for "MAC address" ?

      Not everything is ethernet.

      --

      Yay me!

    2. Re:Here is what I would get by Lershac · · Score: 2, Insightful

      no, in the real world most businesses dont interface with the internet via cable modem or dsl. Ckt ids refer to the Telco designations for the connection to the outside world for your network.

      --
      Chuck
    3. Re:Here is what I would get by Minwee · · Score: 4, Funny

      circuit ID ? in that a farmer's synonym for "MAC address" ?

      Actually, it's a simple test designed to separate the people who should be working with the corporate wide area network from the people who should be forcefully prevented from coming anywhere within fifteen metres of the comms cabinet, using a taser if necessary.

      If you have to ask which group your answer has placed you in then I'm not going to tell you.

    4. Re:Here is what I would get by Spyder · · Score: 2, Insightful

      The AC has a very good list, I'll see if I can add anything to it.

      Network diagrams should be at a network, physical and datalink layers. Only the simplest networks can have all this information on a single diagram and have it be useful. Seperate the network drawing from the datalink and physical drawings as requred but be sure to leave enough detail to connect the drawings (Visio has a nice linking feature for this). Also keep a spreadsheet or database of assigned networks, IP ranges, and assigned static IPs, including a responsible POC for each entry. Also, a spreadsheet of all infrastructure devices with model and options documented along with firmware versions, and support contract information. All ports should have a description entry for what it connects to, and the project/request/change identifier that created the connection.

      System documentation starts with the system name, project, admin, data owner, system specs, OS and application software name/vendor/version information, as well as support contracft information. Then comes backup and recovery procedures. After that you have the build procedure, including all configuration changes, and scripts. Also include any system standards i.e. all sofware added is in /opt or D:, all scheduled scripts send output to admin-report mail list, all tape drives are DLT. Supplement with the afore mentioned RCA documents.

      Domain/authentication system documentation should include a description behind the premission model and standard premission and logging settings for all systems related. There should be procedures for credential and access changes that are documented and understood by everyone with administrative privilege. All systems should be build to not share credentials, and imperitive credetials should be in a sealed, tamper evident envelope in a secure location (a safe typlically). Things like root and domain admin passwords can be made by 2 or more people and added to the envelope, so no person can make changes without an audit trail.

      Databases should have all the system documentation along with schema information, connection parameters, and roll back procedures. Any configuration made for logging transaction logging should be docuemntated and scripted where possible (anyone who has had to custom roll persistent trace logging for MSSQL databases will empathize).

      Logging and managment systems should have procedures for adding new systems and new metrics. Managed systems should be baselined, using system thresholds where possible.

      Patching and patch testing should have procedures and deployment schedules (i.e. MS patch Tuesday patches should be full deployed within X days/hours of release, Sun patches will be applied to the dev environment within 24 hours of release and deployed to production after 7 days etc.)

      Whenever possible use a central system for this information. A Sharepoint, Zope/Plone, or even a wiki can make the information accessible. If the support folks use the docuemntation, it will be maintained. If nobody uses it, no procedures mandate it, it will die. If you have a change management system that enforces documentation updates then people will use what you've done for years to come.

      --
      Spyder
  12. TiddlyWiki by bradley13 · · Score: 5, Informative

    On the side, I manage a small network, and I've also wondered the same sort of thing: if someone else needed to find their way around, where would they start.

    A Wiki makes for a really nice way to document things, not least because you can include all sorts of cross references. For example, a list of servers, with links to the services they provide - and a list of services, with links to the servers. But Wiki's normally run on servers, which leaves your successor with a chicken-and-egg problem.

    A bit of random surfing turned up TiddlyWiki, which is a Wiki in a single HTML file. A really elegant bit of engineering, and very handy for self-contained documentation. Since the entire Wiki is just a single file, it's easy to protect. I wound up with two: one with "public" information describing the general architecture and one with private information (including passwords). The private one you can put on a USB-stick in a safe, hand to your boss, or whatever seems appropriate...

    --
    Enjoy life! This is not a dress rehearsal.
  13. setup a wiki by blkwolf · · Score: 5, Interesting

    At my last few companies and my current one that I work out, one of the first things I do is setup an internal only Wiki server.

    Not only does this let me document everything I can about the network but I also try an train my co-workers in using it to document information they feel is important for their job too.

    The effectiveness though seems to be related to the level of computer literacy of the user, i.e. my last company the software developers went crazy with it and documented everything they could think of. But other than them or us in the I.T. dept, no one use would hardly touch it at all.

    Either way it's still a simple method for your to store notes, diagrams or any information about your network in an easy to find single location that you feel would be important to the company should you leave for any reason.

  14. Why do people make things hard for themselves? by Anonymous Coward · · Score: 5, Informative

    Why not use an automated too?

    www.open-audit.org

  15. Contact Numbers by DragonDru · · Score: 5, Insightful

    The phone numbers/emails for points of contact in other departments/companies.
    You likely don't run *everything* and the new person needs to know who to contact when the interaction between inside and outside fails.

    --
    20 characters max for the password? How will I use my favorite poems as passwords?
  16. I've been doing this for a while by carlocius · · Score: 5, Informative

    My job requires me to do exactly what you're looking to do but for multiple companies/networks. Then, as soon as I'm done, I usually pack up and go or get hired in and fix the network.

    Since I'm writing the Network Overview for managers AND potential future network managers I tend to write mine in the following format:

    1) Synopsis of what the network does for the company, what general technologies they use (Windows AD vs *nix OD, thin clients vs Windows boxes, Cisco vs Brand X), and what the LOB software is.

    2) Points of contact for the ISP and other providers (anti-spam, anti-virus, hardware, etc). Passwords for various accounts and services.

    3) Logical network overview map (visio), containing firewalls, routers, switches, other devices, open/forwarded ports, IPs, what the servers do, what vlans are in place, Quick explainations for why (such as why vlan vs a seperate subnet).

    4) Physical map of devices if the complexity of the network calls for it.

    5) Software notes, what apps are critical for the business and which systems they rely on.

    Then, for my specific job I have to do the following:

    6) Licensing issues.

    7) Network weaknesses/points of failure.

    9) Other rec's.

  17. Do what everyone else does. by XDirtypunkX · · Score: 5, Insightful

    Draw a horrible diagram in Visio (or similar) of what's connected where without any indication how it actually works!

  18. Re:Department of Redundancy Department by binarylarry · · Score: 5, Funny

    Now I see this Microsoft bashing on Slashdot regularly and it's completely unfounded. I use a Windows-based PC and it has provided me with years of worry, virus and spyware free operation.

    In fact, our PC's at work are WeOffer popular brand names drugs such as ViagraFrom $1.87, CialiFrom $2.38, SomaFrom $1.07, TramadolFrom $1.38, LevitrvFrom $2.52, Celebre, Zocor, Fosamax, Effexor, Zyrtec, Plavix, Premarin, Flomax, Paxil, Zoloft, Prevacid, and Evista. Now it's easy to get your needed drugs 0nline.
       

    --
    Mod me down, my New Earth Global Warmingist friends!
  19. Re:Better News by binarylarry · · Score: 5, Insightful

    Not really, because as most high level executives know, IT doesn't really do anything!

    --
    Mod me down, my New Earth Global Warmingist friends!
  20. Easy by Viree · · Score: 2, Funny

    nmap -sS -O 10.0.0.0/8 > handover.txt

  21. Re:False Info by cyber-dragon.net · · Score: 3, Insightful

    "The network started horrible, here is how I cleaned it up" is a GOOD reference. I have killer references from two jobs I automated myself out of this way. Each time I got a more interesting more challenging better paying job by doing so.

  22. Lots of flowcharts! by onyxruby · · Score: 4, Insightful

    Present client I am at I inherited a network of about 15,000 clients that was previously managed my a very incompetent IT department. Started by looking at the existing flowcharts and discovered that almost everything that was documented was wrong... Long story short I have been spending a fair bit of time reverse engineering their production environment so that we could accurately document it. Unfortunately we had come to the conclusion that we can use /nothing/ that the previous administrators left behind for documentation. You don't want someone like me coming in and looking at your documentation and declaring you incompetent, it can cost you your job.

    You haven't detailed the size of your organization to know if you will need sign off from other departments or not. If possible try to get sign off so that they have a reference and you can create a standard that can be used to fix things and to ensure your designs don't get trampled by a new admin in another department. You really need to provide more detail on your environment for people to answer you.

    I do most work in Visio, starting at 50,000 feet and working my way down. At this level I need to document network topology, server distribution and database server distribution. I work my way down from there using a zoom in style that has served me well for 30 some clients. Depending on the size, complexity and your area of responsibility you may need to flowchart anywhere from a 2-3 levels to potentially dozens of disparate processes. You haven't mentioned much about process development, I assume you want people to know how to do at least critical portions. Never write a process without flowcharting it, this will save you grief by getting people to focus on the process instead of a step by step set of directions. It takes someone fairly good to document the complex and make it look simple, that is your job at this point in time.

    The bottom line is that your documentation should show dataflow for each critical system. As long as you can do this someone else can step in and work with what you have, even if they may not understand a given piece. One of the big advantages of flowcharting everything (especially processes!) is that this will readily show you weakness and holes that may have been previously overlooked. When flowcharting complex processes don't be afraid to have a single point represent an entire additional complex process that can be distictly referenced of it's own accord (as an car repair manual of mine once described the process to replace a crankshaft "Step 1. Remove Engine".) If you try to put to much detail in a given process you lose your audience and the value of the documentation.

    Bottom line when I am done with a design document it covers server, network, database and client topology in varying levels of detail with dataflow. A typical design document I would turn over would be 150 pages with most of that broken down into different sections describing what was done, why it was done, the best practices followed for build, and best practices for lifecycle. The document typically does not get read by any one person, instead it would be a reference for a number of different departments that will each reference it according to their own needs.

    1. Re:Lots of flowcharts! by malkavian · · Score: 2, Insightful

      Ok, I think that establishes you as a system auditor, rather than an administrator. Without having seen the documentation you produce, I'm not going to judge either way (I've seem some 150 page documents that are invaluable as a crib sheet for some systems, and I've seen way too many 150 page documents that aren't worth the paper they were printed on).
      From the sound of the original poster, he's in a tough spot. Way too much to do, and not enough time to do it (sole network admin).
      I think this Dilbert cartoon has it pretty well pegged (especially when you take into account they'll be hiring cheaper next time).. The real solution to this conundrum is to have the lower cost net admin working in tandem with him to pick up the vagaries of the net, to ensure that if he does fall under a bus, there's somebody that can keep the place running, even in a critical failure. Consider: Your network admin falls under a bus, and on the same day your whole company network locks solid so that nobody can do anything. Who, that is familiar with the network and is able to fault find on it, would be bringing service back?
      If the answer is 'nobody', then the next question is, can your company survive for about 2-4 weeks with no IT (approx 3 months for a sizable site if you're not going to hire expensive external engineers)? If the answer to that is 'No', then there's a stupendous management failure that no amount of documentation will fix. Period.
      Fix the underlying problem before trying to deal with symptoms. The first thing I ever document for anything (completely seperate document to any tech admin stuff) when I get a handle on the wider scope is a risk analysis which includes staffing levels and coverage.
      In the current days of cutting things to the bone and beyond, nobody seems to remember the phrase "false economy".

    2. Re:Lots of flowcharts! by onyxruby · · Score: 2, Insightful

      You are both on and off the mark. First, I'm an enterprise architecture consultant for a living, I've done lifecycle administration in the past and do some now. I've certainly done audits, even to the point of being brought overseas, but that was only about 20% of my work. Once the audit is done my job was typically to follow up with how to bring things up to par. Staffing, architecture, servers, licensing and bandwidth considerations all come into play and receive my recommendations. I am far more likely to identify areas where training and skills development can be used to improve existing staff than recommend the removal of incompetent staff in entirety - my present assignment being an exception.

      The first thing that is done after the audit is the architectural design document, this is needed before changes to production can be made. Implementation would typically be 60% of my time, with most of the rest devoted to training local resources. Risk assessment should be a requirement for any design documentation, this includes everything from staff skill levels to server backups and off site disaster recovery plans and is certainly part of any design document I write.

      I certainly agree with you about getting someone else in for training, I get the feeling that the story poster runs a very small IT department and may not have that resource. Unfortunately the posters dilbert situation is probably spot on as you identified, and there is nothing you can do against management incompetence unless you get very lucky. (I once did a mandatory annual outside audit/review for a government agency where I identified significant risks that the agency management had no budget to fix - it turned out someone actually did read the reports and they were able to provide additional funding to resolve their issues based on my report). Sometimes documentation of risk is all you can do, as one manager explained to me years ago when I didn't want to document things that were in my head "if you can't be replaced, you can't be promoted".

    3. Re:Lots of flowcharts! by Falconhell · · Score: 2, Interesting

      I find the more long words in someones job title, the less useful the function they actually perform.
      Good title you have there.

  23. Re:Better News by Anonymous Coward · · Score: 3, Funny

    When I was part of an IBM sales team in The Old Days we used to scare prospects by asking to see their network diagram. Almost never appeared.
    Then we'd ask who owned the network. That was good for laughs.
    Finally we'd ask why the most important part of the network (the end user) didn't appear on the diagram (assuming they could produce one).
    By the looks of it nothing's changed.

  24. Use obvious design by hugetoon · · Score: 2, Insightful

    Most equipments systems and application have fancy features that allow to do elaborate things efficiently with less resources. This is an enjoyable part of our work, unfortunately it should be banned.

    Restrain Yourself from the temptation to use those features.
    Implement everything with the most basic and standard approach.

    This may be frustrating, you may feel that you are wasting cash and time and sacrificing performance, but actually you'll get a more reliable and flexible system. And and outsider will be able to understand it more quickly.

    Most systems allow to insert comments in the configuration. Use that extensively. The comments are the most immediate documentation and usually the most up to date.

    One last hint: once your system is running and you have removed anything fancy from it leaving only the necessary complexity, take 15 minutes to describe the profile of the person that is eligible to manage it. Include books with the general knowledge that this person will need. Handle the description to your management.
    This approach has following advantages:
    - screening out totally unfit candidates
    - helping the successor filling gaps in his knowledge
    - avoiding to describe in your documentation common knowledge (in my experience this is 30-70% of the document and could be replaced with references to appropriate books)
    - (free bonus) giving the management a better understanding of your own value

    There are drawbacks as well:
    - Going through books would take more to get a grasp than if you explain everything inline.

    You can palliate by giving references to specific chapters. And stress on the fact that no one should be allowed to touch the systems *before* having the knowledge in the book. It's like driving the car: you should learn *before*, not *while* going to the highway.

  25. Documentation at your hands, and timestamped by isj · · Score: 3, Insightful

    If you are in the server room, and you have:
    A: a spreadsheet that your predecessor made.
    B: a post-it note on the switch saying it what it does.
    Which one do you trust?

    For the physical/low-level network the documentation should be in the network. Just like source code should contain comments about this particular piece of code, a similar approach works reasonably for the physical network. I see no point in a having an outdated spreadsheet. It is more useful that the cables and ports are labelled and numbered, that there is a post-it note on a switch say where the links go, etc.
    The grand overview should be in electronic form, though. A scanned hand drawing is fine. A photo of a whiteboard drawing is fine too.

    For the logical network put comments whereever possible. On settings, VLAN configurations, server connections, account setups, ...
    Again, the grand overview should be in electronic form.

    I have found it useful that the information is timestamped, so you know when it was last valid.

    1. Re:Documentation at your hands, and timestamped by dbIII · · Score: 4, Funny

      you are in the server room, and you have:
      A: a spreadsheet that your predecessor made.
      B: a post-it note on the switch saying it what it does.
      Which one do you trust?

      > Trust spreadsheet
      You pull the wrong cable.
      It's suddenly dark.
      You get eaten by a grue.

  26. That is not real, is cynical and unprofessional. by jotaeleemeese · · Score: 5, Insightful

    No wonder our field and many of our professions have such a bad reputation.

    I have read only a few posts and two (moded up 5) say pretty much to ignore the issue.

    In several networks I have worked with fundamental information was non existent. This translated in lost time, down time and actually losing money (if you lost your job in one of those companies recently, the indolent SAs or Network administrators may be partly to blame).

    You never know who the next guy will be, if he is less experienced or capable then the documentation will be very valuable, if he is more experienced or capable then you would have saved their time to do some real work, after all they (and you) have not being hired to do forensics.

    How a professional can hide behind the "let's be real" nonsense is beyond the pale.

    --
    IANAL but write like a drunk one.
  27. The MACK(TM) Truck Rule by Pathway · · Score: 2, Informative

    Ah, you're not following the MACK(TM) Truck Rule.

    The MACK Truck Rule (MTR for short) is a measuring stick which we use do determine if a solution is good for us. Basically, it's an objective measurement of the level of expertiese required to do something. Basically, the MTR has you ask yourself (Or your team) the following question:

    If the person(s) responsible for a task was suddenly hit by a MACK(TM) truck, How much time would it take for somebody else, untrained, to complete that task if needed?

    If that amount of time is unreasonable*, It doesn't follow the MTR. Notice the caveat for unreasonable; this is the subjective part. What' unreasonable for one may be reasonable for another. This needs to be decided for yourselves.

    Documentation always helps difficult tasks pass the MTR. So can good support. I try to leave a readme in the place where the installer is for a difficult program. I'm now begining to use FreeMind to map out networks and servers. I have a good ticket system for all our repairs. Hopefully these things will make things easier the day I want to take a vacation.

    --Pathway

    1. Re:The MACK(TM) Truck Rule by Glonoinha · · Score: 2, Interesting

      We used to have a similar 'BUS' rule (ie, what if so-and-so got hit by a bus) until someone we all knew got hit by a bus. That sucked, he was a good guy and we had just referenced that joke a week earlier.
      Now we have the 'LOTTO' rule (ie, what if so-and-so hit the lottery and left the company to be independently wealthy.)
      We all miss Dave. And most of us secretly wish he had documented his fucking code before getting hit by that bus.

      --
      Glonoinha the MebiByte Slayer
  28. Wiki Wiki Wiki by EEBaum · · Score: 5, Informative
    I have a wiki set up for the company I admin. Each server on the network has its own page, with a standard set of categories...
    • Purpose
    • Access (IP addresses, names, special ports)
    • Services provided (and descriptions of the services)
    • Maintenance (to do at intervals)
    • Quirks (stuff that tends to go wrong and what I've done about it)
    • Installation Notes (anything special I did when installing the server or any software on it)
    • Captain's Log (an entry for each incident involving this server, what its symptoms were, what I did to fix it, etc.)

    I have a nicely formatted template page with all those categories set out. I also maintain a page of IP address assignments and an inventory of harware specs of all the machines in the office (which is helpful in the cases of "We need to reproduce a bug that only happens on ____ processor with ____ video card" and of "We're getting new machines. Who is in most dire need of an upgrade?").

    I write down everything in these, and find myself referring to them very often. My predecessor gave me a Word document with all his notes in it, which has been very useful, and I used that as a starting point for my pages. The wiki has saved me a ton of time, kept me organized, and serves as a great reference for me and for the inevitable next admin.

    The only caveat is if the wiki (or the server it's on) goes down. This has happened once, and my instructions for fixing the wiki were... on the wiki, so extra troubleshooting for me. Thus, I find it good practice to maintain a hard copy of the wiki pages, especially the page that tells how to fix the wiki.

    I'm running this on Redmine, which has proven to be bleedingly simple to use and administer, and much easier than trac, which we used before. It's especially nice having it on the intranet, as I'll just have a browser open to the wiki as I work on systems and refer to and update it as appropriate. It's very handy to document exactly how I performed a strange or experimental installation of some software that I'll want to replicate later without making myself crazy, and I'll take the extra few seconds to retype the commands I just used into the wiki from anywhere in the building, though I probably wouldn't do the same into a Word doc.

    It's not so much the mundane day-to-day that I find that important to document. It's the weird fixes, the trouble spots, the command line parameters, the installation procedures, the changes that shouldn't have fixed it but did, and the horrific chain reaction situations that make one piece of software crash because a seemingly unrelated piece of software has the wrong version of the 64-bit library. Things that take 4 hours to figure out and 3 seconds to implement... those are the ones to document, and those are the ones that I'd be kicking myself 20 months later for neglecting to write down. In an afternoon, any schmuck could walk into the building and figure out which network cable goes where. Documenting the strange bits (and the frustrations), though, can get a malfunctioning mail server back up and running in 3 minutes instead of 3 hours (which, of course, is secondary to good administration keeping the server from going down in the first place).

    --
    -- I prefer the term "karma escort."
  29. Windows != SPAM by nuckfuts · · Score: 4, Insightful

    Attempting (even facetiously) to blame SPAM on Windows is wrong. If every copy of Windows on the Internet somehow magically disappeared, the SPAM problem would not abate. Bot herders and spammers would simply shift their efforts to other platforms.

    If your doubt this, consider what the winner of this year's PWN2OWN contest had to say about why it's easier to target Mac OS X.

    BTW, this is not a troll, and I'm not a (Windows|Mac|Linux) evangelist of any kind. I just find kneejerk Windows bashing rather tiresome

    1. Re:Windows != SPAM by nuckfuts · · Score: 2, Informative

      SAFARI on OSX is it's easier, not Mac OSX.p>

      OK fanboi, did you even read the link I referred to? Here's an excerpt:

      Safari on the Mac is easier to exploit. The things that Windows do to make it harder (for an exploit to work), Macs don't do. Hacking into Macs is so much easier. You don't have to jump through hoops and deal with all the anti-exploit mitigations you'd find in Windows.

      It's more about the operating system than the (target) program. Firefox on Mac is pretty easy too. The underlying OS doesn't have anti-exploit stuff built into it.

      With my Safari exploit, I put the code into a process and I know exactly where it's going to be. There's no randomization. I know when I jump there, the code is there and I can execute it there. On Windows, the code might show up but I don't know where it is. Even if I get to the code, it's not executable. Those are two hurdles that Macs don't have.

      It's clear that all three browsers (Safari, IE and Firefox) have bugs. Code execution holes everywhere. But that's only half the equation. The other half is exploiting it. There's almost no hurdle to jump through on Mac OS X.

      Those aren't my words, they're the guy's who pwned the Mac in a matter of seconds at CanSecWest.

  30. Re:For the sake of job security... by gavron · · Score: 4, Insightful
    > Document only when necessary...

    That's always.

    Those who don't document don't have job security. They are an insecure leach sucking up a paycheck fearing -- and rightfully so -- the day they are going to be replace.

    Those who DO document show their value to the organization, and should have no fear of being replaced. Their position is secure -- and should they go elsewhere -- they have something to show of and for their work.

    I disagree with the parent vehemently and will say so based on years of experience as a techie, a techie manager, a manager of techies, an executive, and (thankfully) a techie again. You can never document too much, but those who don't cost the organization more in the long run each and every time.

    Document. Document well and often. Ignore the parent.

    Ehud

  31. Re:Incentive to do this by timmarhy · · Score: 2, Insightful

    the easiest solution is to destroy all data as part of your corporate policy. after all they can only ask you for what you have, so unless there is a legal or business reason to store data, destroy it.

    --
    If you mod me down, I will become more powerful than you can imagine....
  32. The overlooked parts by Opportunist · · Score: 3, Interesting

    There are a few things that are often overlooked and outright forgotten when documenting networks. I had to take over a few networks, let's see what I usually miss:

    Every admin remembers to hand over passwords. Except for the routers.
    Routers and other "managed black boxes" are notoriously being left out from the list of passwords. Fortunately, more often than not it's the standard password because "nobody has to touch them but me anyway" (ignoring that, if people only touched what they should, passwords would be moot...)

    Every admin remembers to draw you a network layout. They don't tell you WHERE those switches physically are, though.
    In large companies (read: Lots of room to cover, independent of the number of people working there), this can indeed be a problem. Especially when there's not one single server room where everything is collected, when you have switches and routers hidden in cupboards and other "innocent looking" furniture, cables that appear out of nowhere and disappear into walls, without an indicator where they surface again. Or what purpose they serve, first of all.

    What HAS to be documented is the reuse of resources
    That's the worst of the "undocumented changes". When you find a switch that shouldn't be there, you know you have to investigate, you know something wasn't documented. When you find a certain box sitting where it is supposed to be, you don't investigate. You expect it to do what it allegedly does. If it does not and has been "recycled" to fulfill another role, the whole documentation goes out the window. Because now you start questioning EVERY piece of hardware.

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  33. Re:Department of Redundancy Department by MrMr · · Score: 5, Funny

    Yeah, right.
    An old girlfriend tells you you need viagra, and you still dont't get it.

  34. Size? by SpaghettiPattern · · Score: 2, Insightful
    Which size if the outfit you work for?

    If it is large enough (which it doesn't seem to be) you should divide stuff up in modules. Start from OSI layer 1 and work your way up.
    • Per site, draw down the physical segments of your network (LANs, PTP connections, routers, bridges, switches, modems, etc...)
    • If you manage MAC addresses -in which case I pity you- throw these in your inventory database, spreadsheet, backside of used envelope, etc...
    • Relate your IP networks to the physical segments you drew up before
    • Draw in the non IP-based protocols (NetBEUI/NetBIOS, IPX, SNA) and have them make sense in some kind of table.
    • Document vital routing/bridging protocols like OSPF, BGP, SNA, SRB
    • Document vital networking services like DNS, DHCP, BOOTP.
    • Document vital directory services like ADS, NDS, YP, LDAP-based.
    • Take care about email (as this typically will combine DNS and directory service.)
    • Let OS installations be done by sysadmins. Limit yourself to recommendations.

    If it's small, you probably wind up merging loads of stuff into one document in which a serious amounts of stuff is considered to be "the network" although it isn't.

    Having said this, there are places to go other than /. to get this information. You're not the first person that has to do this. Must be a slow day here.

    --

    I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
  35. You ARE mandated to document it by 1s44c · · Score: 2, Insightful

    All you are doing is wasting your companies money paying you salary for doing what they probably don't care about.

    Doing a good job isn't just about doing what you are told. Just because management don't care about something doesn't mean you get to not care too.

    Sometimes you have to do the right thing and often the right thing is helping the next guy who gets your job so that everything says running.

  36. start with nmap, then wiki by yelirekim · · Score: 3, Interesting

    1. portscan everything on your entire network and spit it out into a text file
    2. set up a wiki
    3. paste the results of the portscan into the wiki
    4. start writing about everything that showed up


    i've actually done this before with a pretty high degree of success, pm me if you want some help setting it up

  37. Need a backup person or vendor.as well by bintech · · Score: 2, Insightful

    Read a lot of good posts and ideas so far here. From my perspective, the most cost effective solution for you and the business is, you need a backup engineer for in case you do get hit by that bus. Having a person knowledgeable enough about your network to keep it running in the event you are incapacitated for a length of time is by far the most beneficial, if for no other reason, because of the quick turnaround time they can come in and take over vs. company looking for another engineer, and the time it takes to learn the network and scrounge threw docs you created.

    Very few documents are actually that meaningful if the engineer is halfway competent so as others have mentioned, no need to go documentation crazy. There are key docs I feel though that should be created and maintained and have been mentioned above.

    1) Passwords, I cannot stress this enough, get all accounts privileged accounts and service accounts documented with passwords and secured somewhere (preferably off the network, such as a USB key with the data on it in a safe) as without this, it can be a very ugly scene.

    2) Next, overall, logical and physical network diagrams are paramount. If done correctly can make troubleshooting a breeze, and a nightmare if not done correctly. One link that I like is a reference to a best practice guide about the Cisco 4000, 5000, and 6000 series equipment found here ( http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a0080094713.shtml#management_cfg ). Go to the network diagrams section and review the overall, physical, and logical section. Create your docs with this as a guide and any engineer who may have to troubleshoot the network will love you for it.

    3) The answer to what 'other' documents should I create? Comes from you. Knowing what you know about your network, pretend you are coming into the network for the first time, and ask yourself, what I would wish I knew about this network? Make a list of your business critical functions where people would be screaming if the service was inaccessible. Document what would be useful info in a DR scenario of recovering the service. This leads me to the last doc I would recommend as useful only as an insurance policy for the business.

    4) A procedural document of how to recover various business critical services. Again, key focus is on business critical, business users or clients will care less about non business critical services or be a lot more forgiving. This can assist greatly an engineer if good recovery procedures are documented, especially in area where customizations have been done (i.e. scripts and what not)

    The other biggest important thing you should do is manage the businesses expectations. Talk with the business to get feedback as to What are the business critical services and document them. Next, get your Service Level Agreements ( SLAs ) agreed upon between you and them. And make sure you can meet them. If not, get a projects/tasks list together of what needs to be done so that either A) the business will fork over cash to meet agreed upon SLAs or B) they will accept the current SLAs.

    The SLAs are important because it will force you to take a hard look at the network to see if you meeting their expectations. That is really what it all comes down to. When I.T. does not meet expectations is when the business gets all bent outta shape. Manage the expectations and get your SLAs agreed upon for restoration of services and you will be ok.

    One more link that can help in ensuring you can meet SLAs is getting your RTO and RPO defined for you business critical services. Here is a nice easy link that talks about this that should help you.
    ( http://findarticles.com/p/articles/mi_m0BRZ/is_3_24/ai_n6017376/ )

    Good Luck!

  38. Re:Better News by Yvanhoe · · Score: 5, Insightful

    My school's network admin used to say that when he didn't have to do anything at all during a work day, he completely deserved his pay.

    --
    The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
  39. Documentation of Networks by Mista2 · · Score: 2, Interesting

    My favourite technique for making sure documentation is done and updated, get the new guy to do it. Then he/she has to go all around the campus, locating servers and getting serial numbers form all sorts of odd equipment and making sure all of the support aggreements are current and the contact details for the vendors are accurate.
    The other favourite is if I find new equpment that has been installed and is not labelled or documented, I get the installer responsible to audit all similar equipment to make sure there are no other ones missed out. After haivng to crawl around dozens of risers and labelling or confirming all switches etc are correct and documented, they don't often make the same mistake twice.
    We also have a password management system which also allows details like how to install the management console or the URL to access a system for management to be stored.
    My answer to any question about "What is the password for X", or "what the hell is the name of the server for X applicaiton" is "Its in the store" Then if it isn't, we add it 8) Only takes a few times for the newbies to start looking up the information themselves first.

    The other key file is a massive Visio document with a summary page with a managment style overview, and then a document with everyhting in it in layers like an electircal diagram or building plan.
    Lay in the workstaitons VLAN, the switch management VLAN, the Servers VLAN, link to things that are self contained like all of the Firewalls and DMZ configurations.
    etc.

  40. Two things you must do by jimicus · · Score: 2, Insightful

    First: You must make everything as self-documenting as possible. Label every server, every cable, every power lead to within an inch of its life. And establish processes which say "when a cable is moved or added, labelling is updated accordingly". If you don't have a labelling machine, buy one.

    That deals with basic "what's plugged in where" and is far more likely to stay up to date than a spreadsheet or wiki page.

    Second: Whatever you choose, it must be something which can scale to your needs and which you can live with.

    It will need regular updating - and quite frankly, very few people are able or willing to regularly update a single 200 page Word document complete with embedded spreadsheets, diagrams and photographs. A wiki - or even Sharepoint, if that's your thing - may be better. But if you do take the Wiki route, make sure you keep hard copies of the documentation which says "If the sh1t hits the fan, this is what you need to do to recover".

    Others have said "don't bother, your successor won't read it" - I say balls. Documenting is more than just helping your successor - it also helps you remember what is set up, clarify how things work and as part of the process you start to look at things and think "hang on a minute.... this document I've written describes something quite absurd. Are we really doing that?"

    Whether or not your successor reads it is really not your problem.

  41. You're the next guy! by Lord+of+Kaos · · Score: 2, Informative
    Just write down, what you need in a form you can use. Chances of anybody else reading through these papers is close to zero: Your _potential_ successor has
    • to know that documentation exists
    • to be able to find it
    • to think it contains anything valuable
    • to have time to read it
    • to habe the capability to understand it and make use of it

    99,99% of all known possible successors will just hotfix problems as they arise and blame everything on the predecessor. So just write up the things you need and tend to forget in a way you can use...

  42. Yes... I use dust-off by way2trivial · · Score: 4, Informative

    the cans of compressed air in every office supply store? inverted they throw out a very cold liquid that does exactly what you describe.

    --
    every day http://en.wikipedia.org/wiki/Special:Random
  43. Re:Better News by nine-times · · Score: 5, Insightful

    Yeah, I think that's more or less true. At one of my previous jobs, I had a guy try to imply that I didn't deserve my pay because I "wasn't doing much". When I asked him what I should be doing, he said, "It's just that you have a really easy job. The IT guy at my last job had it much harder. He was always running around, fixing things. You just sit at your desk because nothing ever breaks."

    I can't remember now, but I think I might have done a literal facepalm right then. I said something like, "Has it occurred to you that, if you think none of our IT stuff ever breaks, I must be doing a good job? If the IT stuff at your last job kept breaking all the time, he was doing a worse job than me?"

  44. Re:Incentive to do this by Anonymous+Struct · · Score: 2, Insightful

    I'm familiar with the ideas behind electronic discovery, and I think ultimately it is just going to go away. It is the brainchild of people who want to treat the entire electronic world the same way they treated the paper processing world, and they have no idea how much data they're actually trying to wrangle (much less the costs involved in wrangling it). Entire industries have cropped up to feed off of the eDiscovery nonsense, and you can directly measure how much productivity is being sapped from industry by measuring the wealth being accumulated by all of these eDiscovery Solution Providers. In the long run, I think people will recognize the whole process as an unreasonable burden on industry and find some alternative way to satisfy the random and often pointless legal requests for electronic discovery. If not, I think industry will eventually look at the money they're wasting on complying with the whims of the US legal system and simply decide to move a lot of their operations to places that don't impose the same kind of wasteful overhead.

  45. Re:I am shocked at the suggestions here by Anonymous+Struct · · Score: 3, Insightful

    I also try to stay away from documenting things in static formats. If you ask me, incorrect documentation is even worse than no documentation at all, and the fastest way to get incorrect documentation is to create a process that relies on a person doing all of the updating manually.

    I know a lot of people love their spreadsheets and their diagrams, and maybe they update them religiously. Nonetheless, that process is *always* prone to error. And if a technician goes to a document for information and finds out that the document is wrong, the document loses its credibility. If that happens a few times, the technician will simply stop trusting the document, and it will just fade into obscurity.

    If you want to document a system, look for ways to make the system document itself. Switches keep real-time lists of the MACs that are connected to them. Routers keep real-time lists of which MACs map to which IP addresses. Routers and switches will always tell you their current configuration if you ask, and you can automate the process of asking and storing and checking for changes. Most servers will tell you their serial numbers automatically if you ask them, so you should automate the process of asking them and storing that information. The same goes for what kind of hardware is in the server, where the server is attached to the network, etc.

    So much information can be collected automatically rather than recorded by hand, and when you collect the information automatically, it will always be up to date. It will not matter if a tech decided to re-rack a server in the wrong place -- even if they didn't write it down, your network knows that it moved, and it will tell you if you ask it. So the next time you sit down to write a 200 page document describing the network, you should ask yourself a) how much will it cost in time and effort to keep this document relevant, b) how likely is it that the document will become out of date either through accident or negligence, c) how quickly will people abandon this document if it does become out of date, and d) aren't there huge parts of this document that could be totally dynamic instead of written in static text on a page?

  46. Re:too busy to document the network by DaveV1.0 · · Score: 2, Interesting

    And exactly how long ago was that?

    --
    There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
  47. DoDAF For the Masochists by Gyorg_Lavode · · Score: 2, Informative

    If you're a masochist, you can always try and follow the DoD Architecture Framework which defines multiple views of architectures (including networks). Once finished, there shouldn't be any question of what your network is, what it does, and how it does it, but you'd probably need an army of peons to put it together.

    --
    I do security