Slashdot Mirror

"Priced inflated beyond worth" but they keep selling them. Does not compute. You don't seem to understand that Cisco is a software company and not just selling you a box filled with ASICs.

Great story of bad network administration by the way. You don't take config backups (using a free tool like rancid) and don't know that your hardware has a software license key that needs to be transferred, but somehow your ignorance is Cisco's fault.

I really do not understand how people like you stay employed in IT.

Doesn't anybody do that anymore?

Rancid is arguably the contemporary equivalent. At the user end, you get all the convenience of revision control and versioning for your configurations; but the actual 'make-it-so' layer that turns the configuration you define into a properly configured device is handled in the background by a scripted process that logs in, makes config changes, collects data, and so on.

It is mostly aimed at fancier switches, rather than cheapie endpoint devices; but adding device support through modules is doable and might be worth a look in this case(especially if the SNMP-foo of some of the devices is very weak, as a poster above claimed).

I've seen dozens of methods at different companies, but I've only ever seen one that works and it works really well. Many of the top ISP's use a variant of it.

Let the network self document.

What does that mean? Well, typically it means some discipline in how descriptions are written. For instance ISP's will use a standard customer identifier on all ports. An enterprise might just use hostname. From there, tools like Rancid can poll router and switch configs, store them in a version control system, and mail out changes to the entire staff. Rancid is great to use, because it reduces the human work load down to entering a single line for each device (name and OS type), and making sure that the device accepts logins.

Now that all the configs are archived and you have the one true list of devices it's trivial to take that list of devices and feed it to other tools. One of the first might be NetDisco which probes the devices with SNMP and builds adjacency tables, tracks MAC addresses, and so on. From it's database you should be able to locate anything on the network in seconds.

Now that there is a complete picture of the network, it's time for a little scripting. Take the output of Rancid and/or Netdisco, and use it to for instance build an MRTG configuration file, or a list of things for Nagios to probe. It's fairly easy to take the NetDisco adjacencies and run them into a tool like GraphViz to produce a network diagram.

I know of at least two ISP's using this basic formula, and it works really well. Going to an internal web site they can bring up diagrams, usage graphs, MAC tables, IP information and all sorts of other things about any device in the network in seconds. Once devices are in the system it is 100% automated, turn on a new port and it is magically graphed, MAC tracked, and added to the diagrams. Turn it off, it magically goes away. Everything is in version control so old state can be reconstructed. The only human manual intervention is adding/removing one line to the Rancid config when a device is turned up or turned down. I have even seen folks automate that with Netdisco (but, I think that can be problematic, as it's almost circular).

Spreadsheets, Visio diagrams, and the like are always out of date. Someone will always make a change and forget to update it. Some places are only a little out of date, most places are downright wrong. Self documenting is achievable, and always 100% current.

Racktables
Rancid

Document everything you can.
Backup configs, make sure you save them frequently when things are working.
Get a good network management/monitoring package which uses SNMP to monitor the equipment.
Take as many classes and training sessions as you can.
Purchase vendor support for equipment. Cisco TAC is invaluable when the excrement hits the oscillating device. When the network is down, and the boss comes into the server room to ask when it's back up, it's much more comforting to hear that the vendor is helping you investigate the issue than to hear you have no idea what the problem is or when it might be fixed.
Build a lab to test/learn new protocols/ways of doing things. Have a couple servers in there, as well as the same type or smaller versions within the same family. If you're running Cisco 3945 routers in production, a lab with 1720s running 10 year old code doesn't help you troubleshoot production issues or test code upgrades.
A good podcast which covers CCNA/CCNP level topics with examples:
http://www.ciscohandsontraining.com/
How to backup your devices:
http://www.shrubbery.net/rancid/
Netdisco, good tool for network discovery and host tracking
http://www.netdisco.org/
Join and read network mailing lists. NANOG, Cisco-NSP, Juniper-NSP are a good place to start. http://puck.nether.net/mailman/listinfo/ to subscribe to several of those.
Beyond that, good luck. Speaking as someone who has been doing systems/network administration for close to 15 years, you will learn something new every day. If you don't, you're not trying hard enough.

How about RANCID? http://www.shrubbery.net/rancid/ It doesn't give a shit how you edit your firewall config. Version control does not preclude using a GUI. While a CLI should always be there, I have no problem adding a GUI to just about anything.

Not a cure-all by any means, but one more trick for the toolbox. Very useful during a maintenance window. Obviously Cisco specific.

(tftp/scp/etc new-config to router)

router# reload in 2
router# copy flash://new-config run

(something along those lines, this is off the top of my head, basically copy your new config to the running config)

if it works, wr it to startup config, if you get disconnected, wait 2 minutes for the router to reboot and automatically load the previous startup-config. Adjust the time as necessary depending on change/complexity.

Also use something like RANCID or KiwiCatTools to help handle managing your configuration changes.

But the best trick of all is using a full blown router emulator like gns3.

It's a MIPS emulator that loads unmodified IOS images. You can build complex scenarios and even attach them to NICs on the host PC. I've built labs with several routers attached to bridged NICs in VMWare guests. So you can literally start, say, a webserver on one vmware guest and access it across your gns3 "network". You can also bridge it to physical NICs -- you could have a 7206vxr router running on an old PC!

Plenty of limitations. Namely it can only run a specific set of IOS images for specific models and you have to use an NM-16ESW to simulate switching since switching is done in ASICs.

It doesn't save you from doing stupid things; but putting your device configurations under revision control, using something like Rancid can make rolling things back easier, as well as generally encouraging sanity around device configuration.

We have had/do have similar issues and have not found a single solution. For windows host inventory, we're utilizing Microsoft Systems Center Configuration Manager (previously we were using System Management Server) For network device inventory (managed routers and switches) we take a two-fold approach: Rancid for configuration (and therefor inventory) and NeDi for network discovery and inventory. For IP address Management we tried a few apps (phpIP and IPPlan) but I found issues with both...so i wrote my own and we use it now: Collate:Network. I had written something similar to Collate:Network for hardware/software/user-assignment management called Collate:Inventory but it never caught on so it mostly sits idle now waiting for someone to ask me to get off my butt and start adding new features. To a certain extend I think this mish-mash of tools works well for us. Each tool is good at what it does (at least the ones I work with are...i don't really use the Microsoft tools personally) and so we usually get what we want out of them. The problem we run into is that there are so many tools to manage that they sometimes don't get the attention they need to stay up-to-date on our environment...though i don't know if a single monolithic tool is the answer either.

I would agree with your assessment on Opsview http://opsview.org/ . It is working well for me so far. I recently built did a nearly painless build (via apt-get install blah) of it on a Debian box, and they also have a VM available.
I'm not sure why the NMIS / MRTG combo doesn't do the trick for your trending / graphing needs -- I've used plain old NMIS http://www.sins.com.au/nmis/ (which opsview includes) to do a lot of the things I have done in the past with Cacti. If there's other stuff you're getting out of cacti these days, I'd be interested in hearing that. These are all basically frontends to RRDtool, if memory serves.

Opsview has a clean (IMO) interface (no goofy Windows-like dropdown like groundworks), and does monitoring (agentless or agent-ful/agenty), trending, psuedo-useful but mgmt-pleasing network visualization (via nagviz), alerting, custom hoopla, etc..

My additional need has been configuration management for network devices, which is where RANCID http://www.shrubbery.net/rancid/ comes in. Rancid also allows a lot of nice (expect-based) mass-configuration of network devices (e.g., changing snmp passwords globally). Command-line required. There is also a (somewhat weak) 'looking-glass' plugin that comes with NMIS (and I think opsview) so that you could tie in viewing of RANCID configs from the same NMIS/opsview dashboard.

My only complaint with opsview at the moment is that the integration with MRTG and NMIS isn't very tight. You just click over to their dashboards. On the plus side, device/host configuration is shared, which is fantastic. (Also, you don't have to install them separately, which is actually a pretty big win.) Another good thing -- if you're talking 5000 devices, agents and distributed monitoring are there for you.

I've used NMIS to good effect, really love this tool. Also don't forget something like rancid either, it will save your life at some point.

http://ftp.shrubbery.net/rancid

Will store your configs including SW versions in a CVS tree. You could easily parse this stuff and dump it into a DB.

Tripwire for routers? talk about reinventing the wheel: try the open alternative that's been around for a few years: RANCID http://www.shrubbery.net/rancid/

Plus, RANCID supports a whole heap of other devices (other than Ciscos), including Juniper boxes, ADC EZ-T3 muxes, Alteon, Extreme, Foundry, etc. It requires perl, expect, and CVS, and it's dead easy to setup and use.