Slashdot Mirror

We have VPNs to each data centre and client site and administer them over SSH generally. Some systems (eg ones dealing with customer details like credit cards) we have a single external facing host with Yubikey authentication to reach that network, and we use SSH port tunnelling to reach other hosts.

Wireshark was already mentioned, so I'll list some other tools I've found useful:

Mtr is better than traceroute. It has ncurses and graphical versions.

For persistent ping tests, I can recommend SmokePing.

Any modern network should have SNMP monitoring capability in the switches and routers. Ask permissions to get read-only access on the devices and there's a wealth of information to be gathered. From basic information like port status, packet/byte counters, to more advanced like topologies learned by MAC learning and neighbor discovery protocols (CDP, LLDP). Or you can just buy one for the class. 100M 24-port managed switches are not that expensive and a Linux server can be used as a SNMP-enabled router (Install and configure snmpd).

To actually act on that data.. You can try one-off tools like Cacti for traffic monitoring, and NetDisco for device and topology discovery. Or a huge does-it-all tool like OpenNMS.

Managed network devices can also dump traffic, either using "monitoring ports" (that mirror traffic from other ports), sflow (sampled stream of packets, unless 1:1 sampling, only useful for statistical traffic measurements) or nflow/ipfix (aggregated flows).

I'm especially fond of nflow, in addition to previous tools. Nflows can be used to analyze, post-mortem, who contacted and where and how much data was transferred at what kind of approximate pattern. This kind of data can be dug out from a full dump, but it's usually infeasible to dump _everything_ to disk. I've used flow-tools.

I worked at a wireless ISP that serviced roughly 200 customers that were completely unreachable by traditional means. The location was set in the mild to medium forested areas of East Texas. We had a 30Mb pipe that worked quite well for our network we never saw it start to "peak" or be overtaxed. Being that we were on the 900Mhz spectrum, the fastest anyone could run at was 1.5Mb/s - 2Mb/s.

Here area some of my thoughts regarding setting up your own ISP.

1) It is completely doable. However, there are two roads to take. You can do it on the cheap, or you can do it the way that will stand time. My company chose the method that stood throughout time. What I mean is, we were not using off-the-shelf radios. We rolled out the network using the 900Mhz Motorola Canopy equipment. We used outdoor rated cable that had separation of twisted pairs and grease filled interior to prevent water issues.

Our main competitor, who worked on the north and west side of the city went the opposite route. He chose to use cheaper 2.4GHZ equipment, primarily PTP bridges.

2) The technology is out there, you just have to find it at a price that you are willing to pay. When I was servicing the radios, they would cost roughly $350 new from Motorola just for the endpoint Subscriber Module. We instead purchased refurbished models for almost half the price at $200-225. The Access points and other major equipment will set you back, IT IS NOT CHEAP.

3) Backbone and network structure. We may have over engineered our network, but we felt it was necessary to keep subscriber information private. We had a small cisco switch that at each access tower that would assign VLAN to each subscriber module. On the internal side of the switch, the VLANs were removed and went into a bulk VLAN that was specified for that tower. No other subscriber could see any other one without first going to "The Internet". We also created a Management VLAN, so we could service and access the management interfaces on each of the Backhauls and APs. Latency across the network averaged about 50-150ms.

4) Please for the love of all that is holy, do not, run your own Email server. It is a absolute pain in the ass. I was the person who was in charge of ensuring that the systems in place stayed running. This meant, DHCP, DNS, HTTP, Email Services, and Management interfaces.

Remember Virtual Machines are your friend. Buy one or two hefty servers and backup the VMs to each other. That way if you have an outtage, you can get the VMs back up in running in about an hour.

DHCP - Since we had a bit of a robust network, we had different subnets for each of our towers. In total we had about 18 subnets that each had different purposes. This tool helped like the charm that it was. http://phpdhcpadmin.sourceforge.net/ At the time the logout system was broken, however, I patched the code to disable the login/logout functions and wrote a script that would automatically give me the next available IP address.

DNS - No fancy tools here, I mostly just let it roll and didn't touch it. I only touched DHCP when we added a hosted website.(which later went to rackspace)

HTTP - Simple, run Apache, set and forget.

Email Services - Complete Pain In The Ass. No really, I'm not joking. At the time, the powers over me, decided that we would give our customers up to 5 email addresses. So I setup a linux server in that ran Postfix, Dovecot, ClamAV, Squirrel Mail. It provided IMAP, POP, SMTP and SSL(if wanted). At the time, when I arrived the server was already in place and running. However, fast forward, 3 months, and someone decided to run "updates" on the server. Breaks all of the packages, settings, the whole shebang. Not a fun week at all.
Besides that, there were also issue with SPAM. We would constantly get blacklisted by various servers.

Management Interfaces - This was where the heart of out network lay. I have one word, Cacti, http://cacti.net/ For wireles

Best way I've found to measure growth is to have a running history of traffic on each router. You don't need a $billion to do it. There are some decent enough FOSS tools out there to do it. MRTG or Cacti will work nicely and integrate with SNMP.

For a smaller network, you could run a span port and graph your own data with a shell script, or hook up NTOP. which will give you real-time views of traffic but you would need to implement something to save those reports daily.

You suggest some good tools, but they primarily measure network utilization rather than capacity. The question isn't "how much data is my network handling now" but "how much data could my network handle at peak"?

Best way I've found to measure growth is to have a running history of traffic on each router. You don't need a $billion to do it. There are some decent enough FOSS tools out there to do it. MRTG or Cacti will work nicely and integrate with SNMP.

For a smaller network, you could run a span port and graph your own data with a shell script, or hook up NTOP. which will give you real-time views of traffic but you would need to implement something to save those reports daily.

I work heavily with Cacti, and I do not recommend this book. For its price, you would just be better off reading the manual at http://docs.cacti.net/

I use Cacti, with THold and weathermap plugins.

But then I'm biased.

I use Cacti, with THold and weathermap plugins.

But then I'm biased.

You can run a one wire network which uses 2 wires. There is a range of devices you can read information from, http://owfs.org/. For example you could run a temperature sensors in each room. Combined with a tool like http://www.cacti.net/ you can log an ongoing temperature graph. Combined with X10 http://www.linuxha.com/ you now could act on the information you receive. for example if the room reaches a certain temperature you switch on the fan. racker79

I was recently in a similar situation and I was the one who had to figure it all out due to lack of documentation. The main things that I did were to map the network and create updated diagrams. I did this by using a bunch of utilities both commercial and open source.

Create a list of UIDs and PWDs and maintain them in a program like PasswordSafe. http://passwordsafe.sourceforge.net/

Map all the switches using a program like netdisco. Depending on your equipment, it can find the uplinks and map the network for you. Otherwise, fill in the neighbor information on your own. http://netdisco.org/

Setup monitoring with Nagios and set the parent/child relationships using nagios. Make sure the map is accurate. Monitor all critical network services such as routers, dns, wins, email, proxy, fw, etc. http://www.nagios.org/

If you're not going to graph service data with Nagios, do it with Cacti. That will provide historical/trending data that is important for future network related decisions. http://www.cacti.net/

Create high level network overviews using Visio. Solarwinds LANsurveyor Express is very useful for automating network maps. http://www.solarwinds.com/products/LANsurveyorExpress/

Make sure you have good backup configs of all devices. A tool like Kiwi CatTools will automatically backup the configs for your devices and even alert you to when configs have been changed. It's great for change management purposes. http://www.kiwisyslog.com/kiwi-cattools-overview/

I've used Cacti. It's based on RRDTool. It's pretty and also useful.

Cacti.
Ntop.
Nagios.
MRTG.

1. Deny IRC traffic at your firewalls. If there is a business need for IRC then setup a IRC proxy, or inline authentication. This simple step will stop many of the bots out there from phoning home.

2. Enable reverse path detection on your network devices. This forces your internal routers to check whether the source ip address that the bot is sending, is available out the interface that your comprimised host exists on.

3. Enable DHCP snooping on your edge switches. By configuring this feature the switchport that your host plugs into passively observes what IP address was given to your computer. If traffic is spoofed (a common occurrence for botnets) the switchport effectively shuts your host down.

4. Monitor your network. There many free and commercial products that will make it clear that your traffic profiles have changed. Some good free tools for this are Cacti - http://www.cacti.net/, Nagios - http://www.nagios.org/ and NTOP - http://www.ntop.org/

5. Utilize update antivirus technology, hopefully one that reports to a central console. These are simple steps, that frankly most people do not use in their networks. If they would the botnet issue would be greatly minimized.

Offtopic: I've played around with Zenoss, and I can tell you that it isn't a good enough NMS to replace http://www.opennms.org/ OpenNMS, and doesn't graph well enough to replace http://www.cacti.net/ Cacti.

• Cacti (an RRDtool front-end -- if you don't know what RDDtool is, you don't need this :) )
• Munin, and
• OSSEC.

I've looked over someone's shoulder at the latter - it seems pretty good, it runs on SNMP - I tinkered with NAGIOS five years ago and found it good, but a little dangerous if you didn't read the docs before firing it up (back then, anyway, it auto-discovered the local network by strobing everything in sight with Nmap scans)... but I've no experience of any of these in production. I've been asked to build out a new office network, which will be a template for future local offices, and getting the monitoring right is going to be crucial, so any actual experience of production use gratefully received!

Ahhh, I wasn't aware that they were even part of the same project.

The DD-WRT website is very scant on details and only seems to provide a decent explanation of what it is if you already know everything that it does.

I really wish it had a complete list of features on there. After installing it, I tried to figure out (for several hours) how to do snmp monitoring so I could add it to my cacti graphs only to realize that it had that capability in there already. A simple google search would have shown the same thing, but that info should have been readily available on their site and not hidden away in the forums like it is.

Another really cool thing about dd-wrt is that it does have ssh/telnet for doing manual tweaking, although I wasn't immediately able to figure out how to edit anything since it all seems locked/ there's no available disk space.

There is a number of alernative Open Source monitoring software available. Nagios, Cacti and my favourite ZABBIX. ZABBIX is much less resource hungry comparing to Nagios and especially to OpenNMS.

I haven't seen Munin myself, but I'll go check it out cause I'm in the middle of configuring various monitoring myself.
 
Here I've got FreeBSD boxes with the ports of Nagios and Cacti running, very easy to get setup through ports. Cacti I admit has a learning curve, but the forums are fantastic. This post in particular is a good starting point http://forums.cacti.net/about15067.html
 
If you are shy of trying to create your own scripts, just look for one that is similar and edit it. There is example code in the forums for creating a perl script to query the Nagios Windows client from a *nix box for perfmon counters, that works fantastic for me - I look up various troubleshooting tools, figure out what perfmon counters they check and write a script to graph that myself, did that for our Exchange boxes by using some of the same perfmon counters that the Exchange Troubleshooting tool from MS does. Once you edit templates and scripts a few times from the forums, you'll be able to create ones from scratch a lot easier.
 
Also in Cacti the generic SNMP template is great if you can figure out what OID you want to graph, helps to have some experience with MIB's or snmpwalk though.
 
One more thing is that I have in the past worked with some high end cost a lot of bucks monitoring systems, including the Solar winds toolsets and I am using Cacti/Nagios not because they are free, but that I think they are better and considerably more flexible, maybe they don't look as pretty, but they get the job done.

We have a medium sized setup and for us, Cacti works great. http://www.cacti.net/

MRTG has always done the job for me but I have been looking more and more at cacti, it's worth a look over here http://www.cacti.net/

There is a Threshold Plugin for Cacti, it requires the Plugin Architecture, which is set to be rewritten and integrated in the next major release of Cacti. I think it works extremely well, but I may be biased considering I helped write it.

How is this different from cacti?

All of these are open source, built on LAMP, and run great on Windows.

HW & SW inventory: Winventory (http://winventory.sourceforge.net./
Trouble ticketing: Eventum (http://eventum.mysql.org/wiki/index.php/Main_Page ). The Anonymous Reporting Form is a time saver.
Cacti (http://www.cacti.net./ Graphs all parameters on your servers and routers.
Documentation: TikiWiki (http://tikiwiki.org/tiki-index.php). It has articles, FAQs and LDAP integration.
FreeMind (http://freemind.sourceforge.net/wiki/index.php/Ma in_Page). Mind maps are a FANTASTIC tool for documentation and you can publish them easily on a web server (get 0.8.1 beta3).

These are free, and get the job done.
SysInternals's tools (http://www.sysinternals.com./ Process Explorer and TCPView are the most useful, and there are many other great utils.
KiXtart (http://www.kixtart.org./ The best language for login scripts, and just about all your scripting needs on Windows.
Network Notepad (http://www.networknotepad.com./ Draw your nework diagrams here, and then publish them on your web server.

ntop
Nagios
MRTG
Cacti

By touching it. There's always an assistant named Murphy looking over my shoulder, but she usually waits until I'm in the shower or leaving on vacation before "helping".

Your question is really "How do I introduce layer 1 and 2 problems into my home LAN, since all layer 3 routing is limited to a NAT box with a single default route?". The lower layers are a good place to start, since half of all your problems come from there, save the routing problems for a future ask/. question.

Others have already pointed out the joys of having dueling DHCP servers, subtly mis-configured DNS servers, overlength cables and the like. Keep an eye out for others throwing out bad ethernet cables with broken catch-tabs, frayed insulation, sharp kinks or intermittent wiring, and put them into critical places in your network. They may not fail right away, but will wait until you host a lan party at your place or you have a few hours to get a report done. Her name is Murphy, she's a bitch and she'll gladly pay you a visit when you least want her around.

Start to learn what kind of traffic is on your local network. Get ethereal, snort and ntop running, and see what the packets look like. Chances are you'll find some things that look suspicious, you'll learn a lot by figuring out how DHCP handshakes work, how often ARPs happen, what other protocols are on your net besides IP. Since you are running a BSD, you can pretty safely put the box on the outside of the firewall (it probably is the firewall) and watch all the constant crap scanning the internet. That's a great way to learn how to tune firewall rules by hand, and you will break things along the way.

To really start to learn how layer 2 networking almost works, grab some old cisco kit off of eBay. I've seen 2900 switches for US$20. Plug something slightly pro into your network, start simple, just get a cheap used cisco/hp/3com switch off eBay that can do 802.1q vlans, spanning-tree, and snmp. Your BSD ethernet card can be configured to do .1q, so there is a lot of learning there by creating multiple separate vlans, one for each machine. A single router and switch with 802.1q vlans can make some pretty complicated networking topologies without massive amounts of wiring. Then you can break your network by plugging a crossover cable into two ports and watching spanning tree open up one of them. Bonus points if you create a topology where by creating a spanning tree loop, your main gateway or server port is the one that goes into blocking mode (you need a minimum of two switches to do that).

To break things in subtle and non-obvious ways, try changing your address ranges from the usual 192.168.0.0/24 to something unusual like 172.31.255.16/29, doing the netmask/subnet/broadcast calculations in your head for practice. Then misconfigure the netmasks on each device, notice how one machine can ping another, but not the other way around. Try building multiple separate segments rather than multiple subnets on a single wire, this will force traffic to use your router, and really show netmask problems more clearly.

To really break things, instead of using reserved RFC1918 addresses behind your NAT box, use a public network range like 66.35.250.0/24. Sure, it will break one major site, but you shouldn't be wasting your time there :-)

Since you already have a BSD running, do you leave it on 24/24? If so, its time to start loading up the real tools like cacti, nagios, and smokeping. It helps if you have an SNMP capable switch on your network, but configuring your own SNMP can be quite a learning experience as well. With graphs showing what is happening on your net and the internet over time, you will start to see the cycles of congestion every evening and maintenance times every sunday at wee hours. The most frustrating problems in networkin

I don't think the tools he mentions are that interesting at all. In fact, many of the tools have nothing directly to do with sysadmin activities at all. I love abcde too, but what does have to do with Linux administration. GNU Screen does the same things as the default terminals that come with KDE (kterm) or Gnome (gnome-terminal), specifically one window tabbed consoles and detaching. The use of emacs or VIM or even nano is preference/needs based and could be the focus of an entire article, as it has been done before.

The only tool that was mentioned I found that I thought was worth mentioning was rsync. This is simply because it is not as well known as other tools. Although, its use has spread. I find people using scp or something else, manually or scripted, for things that would be handled much more efficiently with rsync.

To be fair, dig was worthy of mention, for the simple fact that many people who have become used to using nslookup don't know that another tool exists, perhaps even on their machine, that they may find more advanced. Also, pwgen could be useful, but not crucial by any means. I guess some of these tools may solve some annoyances people may have, but not necessarily working to improve or make the role of a sysadmin dramatically easier.

It would have been more useful to mention some monitoring tools or some web applications. For instance, tools that I would mention that help syadmin activities include the following:

- Cacti (http://www.cacti.net/ is a PHP application that uses RRDtool/SNMP to monitor server performance and usage (like disk space, CPU, load average, logged in users, and a lot more).
- phpMyAdmin/phpPgAdmin/phpLDAPadmin
- syslog-ng/php-syslog-ng (http://freshmeat.net/projects/php-syslog-ng/)
- Gregarius (http://www.gregarius.net/) is an a nice web RSS reader. I use this to keep up with the latest version of software releases, a lot of times using the RSS feeds available for all SourceForge.net projects, or otherwise using a feed that may be available on the software's website. Tiny Tiny RSS is another nice RSS reader (http://bah.spb.su/~fox/tt-rss/)

Since I work on a university campus, web applications help a lot b/c I can be in any lab or at home or on any PC and know exactly what is going on with all systems and even the development of the software that is used on them (RSS). But web applications are usually considered useful because access to them is easy and sometimes leave less room for human error.

Step 1) Map the network both logically (which networks, what is the routing, etc.) and physically... the "tug test". Label everything, and put it all in a spreadsheet. Tools are nmap, pen and paper, and a label printer. Access to the routers, or being friendly the the router admin is a must.

Step 2) Isolate the problem protocols and hosts. Be on the lookout for appletalk, IPX, or old netbios. All very chatty protocols. Look for old hubs and replace them with switches. Look for comprimised boxen. Try to VLAN things logically (by department, or usage which ever is best for the environment). Tools are snort, ethereal, ntop, and syslog (any managed switches should be sending to a syslog server (I've used syslog-ng))

Step 3) Trend as much as you can. Even before the network is cleaned up, start to collect statistics from the switches, and/or hosts on your network. Any gateways should be monitored as well. This will let you see if there are problems corelated to a particular time of day, if your're going over your bandwidth etc. Tools are MRTG, or for more in depth try Cacti http://www.cacti.net/

There is much more after you get to this point, but people will be much happier the faster you get here.

Good luck

Run Cacti on different monitors. View some screenshots here

Run Cacti on different monitors. View some screenshots here

I second the IT Watchdogs products. We have a WeatherGoose hooked up to Cacti. Works like a charm.

How about moving to a Unix-like operating system that doesn't require you to compile everything? Having the computer recompiling this, that, and the other uses much more energy than letting it idle and downloading binary packages

I've used other distros, but I eventually, after much hit-and-miss settled on gentoo - and its compile everything methodology for the following reasons

If you compile everything from source, then you know that everything works. In the past I had problems because of the way other distros package their system... they forced their choices onto me, which meant that, for example, I was unable to drop cvs and replace it with cvsnt.

Plus, by compiling everything, you know that the maintainers of the software have given you everything you need to compile you system... with a binary-only approach, the say will come when you want to tweak the system, so you decide to grab the source, mod it, then compile it... and that's when you discover the maintainer missed something, or it won't compile due to dependencies with your system.

So, I'd rather burn a few millwatts of power compiling, rather that blindling grabbing a binary distribution and installing it

Also, its not as if I'm constantly rebuiling my system... on averge I do a complete rebuild every 6 months (from Gentoo stage1)... and the systems that are up and running get minor updates every month or so... looking at my logs - thanks to Cactus (http://www.cacti.net/), my avergae CPU usage is so low, the blips when doing a major compile hardly show.

Finally, its kinda cool to compile your own Linux system from scratch!

I agree about Cacti...it was stoopid simple to install and setup. The same can be said about Cricket.

If you are looking at Big Brother, I would probably recommend you look at Hobbit Monitor instead. Hobbit is Open Source and unconstrained as is BB. They are developing a client-side piece to replace BBNT, as well. Hobbit extends the good things we like about BB and adds some other things we would have really liked to have SEEN in BB.

It seems to me, also, that BB hasn't enjoyed much development activity for awhile now. Every tool has its place...we use BB/Hobbit and Cacti both, each for their own advantages and good points.

-PONA-

Opennms

Mon

Big Brother

For system information polling I'd go with:

Cacti hands down this is the best polling system out there and it's simple to setup and run.