Slashdot Mirror

Wireshark was already mentioned, so I'll list some other tools I've found useful:

Mtr is better than traceroute. It has ncurses and graphical versions.

For persistent ping tests, I can recommend SmokePing.

Any modern network should have SNMP monitoring capability in the switches and routers. Ask permissions to get read-only access on the devices and there's a wealth of information to be gathered. From basic information like port status, packet/byte counters, to more advanced like topologies learned by MAC learning and neighbor discovery protocols (CDP, LLDP). Or you can just buy one for the class. 100M 24-port managed switches are not that expensive and a Linux server can be used as a SNMP-enabled router (Install and configure snmpd).

To actually act on that data.. You can try one-off tools like Cacti for traffic monitoring, and NetDisco for device and topology discovery. Or a huge does-it-all tool like OpenNMS.

Managed network devices can also dump traffic, either using "monitoring ports" (that mirror traffic from other ports), sflow (sampled stream of packets, unless 1:1 sampling, only useful for statistical traffic measurements) or nflow/ipfix (aggregated flows).

I'm especially fond of nflow, in addition to previous tools. Nflows can be used to analyze, post-mortem, who contacted and where and how much data was transferred at what kind of approximate pattern. This kind of data can be dug out from a full dump, but it's usually infeasible to dump _everything_ to disk. I've used flow-tools.

Two network discovery and self documenting systems. NetDisco is (if you can set it up) fantastic for tracking changes made per port.
http://www.netdisco.org/
http://www.observium.org/wiki/Main_Page

I've seen dozens of methods at different companies, but I've only ever seen one that works and it works really well. Many of the top ISP's use a variant of it.

Let the network self document.

What does that mean? Well, typically it means some discipline in how descriptions are written. For instance ISP's will use a standard customer identifier on all ports. An enterprise might just use hostname. From there, tools like Rancid can poll router and switch configs, store them in a version control system, and mail out changes to the entire staff. Rancid is great to use, because it reduces the human work load down to entering a single line for each device (name and OS type), and making sure that the device accepts logins.

Now that all the configs are archived and you have the one true list of devices it's trivial to take that list of devices and feed it to other tools. One of the first might be NetDisco which probes the devices with SNMP and builds adjacency tables, tracks MAC addresses, and so on. From it's database you should be able to locate anything on the network in seconds.

Now that there is a complete picture of the network, it's time for a little scripting. Take the output of Rancid and/or Netdisco, and use it to for instance build an MRTG configuration file, or a list of things for Nagios to probe. It's fairly easy to take the NetDisco adjacencies and run them into a tool like GraphViz to produce a network diagram.

I know of at least two ISP's using this basic formula, and it works really well. Going to an internal web site they can bring up diagrams, usage graphs, MAC tables, IP information and all sorts of other things about any device in the network in seconds. Once devices are in the system it is 100% automated, turn on a new port and it is magically graphed, MAC tracked, and added to the diagrams. Turn it off, it magically goes away. Everything is in version control so old state can be reconstructed. The only human manual intervention is adding/removing one line to the Rancid config when a device is turned up or turned down. I have even seen folks automate that with Netdisco (but, I think that can be problematic, as it's almost circular).

Spreadsheets, Visio diagrams, and the like are always out of date. Someone will always make a change and forget to update it. Some places are only a little out of date, most places are downright wrong. Self documenting is achievable, and always 100% current.

Document everything you can.
Backup configs, make sure you save them frequently when things are working.
Get a good network management/monitoring package which uses SNMP to monitor the equipment.
Take as many classes and training sessions as you can.
Purchase vendor support for equipment. Cisco TAC is invaluable when the excrement hits the oscillating device. When the network is down, and the boss comes into the server room to ask when it's back up, it's much more comforting to hear that the vendor is helping you investigate the issue than to hear you have no idea what the problem is or when it might be fixed.
Build a lab to test/learn new protocols/ways of doing things. Have a couple servers in there, as well as the same type or smaller versions within the same family. If you're running Cisco 3945 routers in production, a lab with 1720s running 10 year old code doesn't help you troubleshoot production issues or test code upgrades.
A good podcast which covers CCNA/CCNP level topics with examples:
http://www.ciscohandsontraining.com/
How to backup your devices:
http://www.shrubbery.net/rancid/
Netdisco, good tool for network discovery and host tracking
http://www.netdisco.org/
Join and read network mailing lists. NANOG, Cisco-NSP, Juniper-NSP are a good place to start. http://puck.nether.net/mailman/listinfo/ to subscribe to several of those.
Beyond that, good luck. Speaking as someone who has been doing systems/network administration for close to 15 years, you will learn something new every day. If you don't, you're not trying hard enough.

Sounds like you need Netdisco. It was originally designed to track assets on educational campus networks, and if your devices support a link-local discovery protocol (CDP, LLDP, FDP, etc) and SNMP than chances are they can be wrangled to work. It's complicated to install, but it's the best tool I've seen for really keeping track of a massive amount of devices on the network.

I was recently in a similar situation and I was the one who had to figure it all out due to lack of documentation. The main things that I did were to map the network and create updated diagrams. I did this by using a bunch of utilities both commercial and open source.

Create a list of UIDs and PWDs and maintain them in a program like PasswordSafe. http://passwordsafe.sourceforge.net/

Map all the switches using a program like netdisco. Depending on your equipment, it can find the uplinks and map the network for you. Otherwise, fill in the neighbor information on your own. http://netdisco.org/

Setup monitoring with Nagios and set the parent/child relationships using nagios. Make sure the map is accurate. Monitor all critical network services such as routers, dns, wins, email, proxy, fw, etc. http://www.nagios.org/

If you're not going to graph service data with Nagios, do it with Cacti. That will provide historical/trending data that is important for future network related decisions. http://www.cacti.net/

Create high level network overviews using Visio. Solarwinds LANsurveyor Express is very useful for automating network maps. http://www.solarwinds.com/products/LANsurveyorExpress/

Make sure you have good backup configs of all devices. A tool like Kiwi CatTools will automatically backup the configs for your devices and even alert you to when configs have been changed. It's great for change management purposes. http://www.kiwisyslog.com/kiwi-cattools-overview/

If you use anything bigger/better/faster/stronger than simple un-managed switches, such as Cisco, HP, Extreme (and so on), you will find that the switch exports its MAC/CAM table via SNMP. This means that you can poll MAC tables of managed switches in different switching domains (separated by routers). If you are interested, tools like http://www.netdisco.org/ will do this for you quite nicely.

We use Solar Winds and SNMPC together, however, What's up Gold is useful. Both Solar Winds and SNMPC are very powerful tools that can monitor large, widespread networks.

Although a PITA to set up, Netdisco http://www.netdisco.org/ is a pretty awesome Open Source solution.

strike

Netdisco is an open source switch management solution. Shows you MAC, IP and NetBIOS information per port, draws graphs and allows you to change VLANs and enable/disable ports with logging.

http://www.netdisco.org/

When considering how to secure the ports, I think you have to find the balance between security and functionality. If you lock down each MAC to a specific port, how much time will you spend managing it? Whenever there is a connectivity problem, will you have to fight with the other groups assuring them that it isn't the network?

As a final thought, you generally get out of a network management system what you put into it. With a network as large as yours, there isn't a silver bullet to fix all of your problems. Whether you customize, roll your own or use vanilla off the shelf software, you need to figure out what makes the most sense for your business. Good luck. It sounds like you need it.

You want something like NetDisco. It will go out and discover every switch on your network and can figure out how they're interconnected. You can then query ports for VLAN, speed, and duplex settings, among other things. Finally, you can toggle ports on and off. It can even produce graphs of your network layout using GraphViz. You'll need a *nix box, Perl, Apache, and Postgresql, as well as managed switches that support SNMP.

If all of this scares you, then go out and buy Visio, but all you'll get out of that is pretty pictures.

check out: http://netdisco.org/
It also draws a layer 2 map and records device IP, MAC, and switch port locations which can be very handy.

http://netdisco.org/ Does network discovery and layer 2 topology.

I forgot to mention, we used ettercap to detect attacks.

Ettercap:
http://ettercap.sourceforge.net/

Netreg:
http://www.netreg.org/

Netdisco:
http://netdisco.org/

I used to work in a similiar environment. We set up Netreg - unregistered machines can only access a website with a form explaining the terms of use and with links to AV software and patches. By clicking on the agreement they consent to having their port disabled if they are found to be in violation of the policy (i.e. if they are flooding the network/spreading viruses). We set up Netdisco (http://netdisco.org/) to handle the actual disabling of ports. For awhile we had 3 categories in Netreg, unregistered, registered, and infected. We would move people found to be infected into the infected category which put them on their own restricted subdomain that could only access a "YOU ARE INFECTED" page with cleaning tools and info on how to get help from us, but it was a nasty cludge of a hack in Netreg to make this work and we ended up giving up on doing it this way after having lots of problems with it and ended just using Netdisco to disable them.

It's not the perfect solution but it did a fair job while I worked there.