Slashdot Mirror
Managing Huge Networks with Open Source Tools?
An anonymous reader asks: "I work for a large multinational firm with a network that spans the globe and am responsible for evaluating the software we use to monitor our network. Our department has a lot of money, and we're usually willing and able to spend it on good commercial software. Recently though, I find myself evaluating and approving more and more open source software. We are actually in the process of replacing some of our commercial tools with software like Nagios, LooperNG and syslog-ng. We are also evaluating MRTG, RRDTool, ntop and a host of other tools. The problem is that there's just too many of them, most of which are not maintained anymore. Here's my question: What other open-source tools do you use to monitor your networks? I not just looking for names, but how long you've been using them for, how easy / hard is it to administer and I guess how well it scales as the network grows. More importantly, are their respective projects still alive and kicking?"
For my large network at home, I have a little device with lights that blink when the network is working. I think the guy at the store called it a switch or something.
He who laughs last is stuck in a time dilation bubble.
...unless you have the resources to maintain it yourself.
You will find gaps in the utility of any set of tools you use, but a maintained project will be likely to have a community of people who can verify any patches you submit and provide tips on using the software in the most effective manner.
Stop the world; I need to get off.
I created and use JFFNMS (Just for Fun Network Management System) to monitor my network at work.
Its also used by a lot of people to monitor big, and medium networks.
Its fully mantained, and customizable.
- Smells Like Open Source Code
We used to use Nagios here to monitor a large number of services on a large number of servers. We eventually agandoned it and replaced our "is the server up" monitoring with simple scripts that call fping. The problem with nagios is that the process model starts to fall apart at several hundred monitored servers/services and we really did not want to dedicate a farm to monitoring.
A slick tool is Gkrellm, which has real-time graphical status for memory/temperatures/net/disk. Can be run in "server mode" (so no need for X on the monitored server). Lots of plugins are also available, from SNMP to ping tools. The project is well alive. Don't know if it floats your boat, though, as you're mentioning huge networks.
Feel ready to own one or many Tux stickers?
there are three security based live cd's
d
= hakin9_live
based on quick search of 'secur' on distrowatch's live cd page
http://www.distrowatch.com/dwres.php?resource=c
here are the results
f.i.r.e.
http://biatchux.dmzs.com/
hackin9
http://www.haking.pl/en/index.php?page
plan-b
http://www.projectplanb.org/
i think live cd's are of great use when figuring out networks etc, b/c its very easy to reboot with the cd and get on and working on the problem
back in the day we didnt have no old school
Snort[www.snort.org]...Sure you've heard, of it...IDS and more
ethereal[www.ethereal.com] , "sniifer" of course is also good.
nmap[www.insecure.org/nmap/]...very usefull as well, various penetration testing utilities.....
rrfw.sf.net is a nice gem of an app. It can automatically discover a lot of high end snmp equipment, and set itself up for monitoring that equipment. It's a mix of XML and Perl, and is really easy to add support for more hardware.
FlowScan/flow-tools/RRDTool/CuFlow.
Well, don't forget it as much as get something better, Cacti. Cacti is a frontend to MRTG & RRDTOOL and offers a lot of awesome improvements, such as a web frontend to add devices, device "profiles" to enable a common monitoring set for things such as Cisco routers, servers, etc and a whole lot more. We used MRTG here at our (Windows only) network, and I'm slowly moving it all over to Cacti for all of the above plus a lot more.
Looking for hardware (Currently need: Large Etch-a-Sketch) Have one? See my journal!
In Soviet Russia the insensitive clod is YOU!
I bought Open Source Network Administration by Kretchmar to answer this question. I was looking for open source tools to be used in a service provider environment and was unpleasantly surprised at what was revealed in this book. However, since it seems you're looking for enterprise-ish stuff, I highly recommend this.
You'll find many of the tools within to be quite useful during both day-to-day operations and troubleshooting as well as long term planning on your network. The author does a fairly decent job of walking you through a basic installation of each tool.
Slashdot reviewed it here.
Here are most of the tools discussed in this book.
"A clear conscience is usually the sign of a bad memory."
An advantage you have is the financial resources to make support arrangements for open source software. These arrangements can be standard, such as purchasing support from a company that expressly supports the OSS project. Or, it could be less traditional, such as finding the authors and making arrangements with them in advance. If you have the financial resources to do so, I would strongly suggest making support a criteria of selection.
From looking at what we've achieved with these I would say that you will need to be careful trying to scale them to large networks. They can start huge numbers of processes each minute, when monitoring many servers.
It depends what you're monitoring, of course - in our case we are monitoring maybe 20-30 operational parameters on each server. If we were only monitoring a single parameter then we could probably look at around 1-2000 machines from a single P4-based monitoring box, without any real problems. Using a 2.6 kernel on the monitoring box would also dramatically increase the scalability of it all.
Scalability issues bite similarly with rrdtool: numbers of parameters monitored per server can ramp the load on the monitoring machine(s) quite quickly. Again that is process load, not CPU load though, and a 2.6 kernel will be significantly better in this area. It can also be resolved by scripting the collection process better - not just running some collect-the-statistics routine from cron every minute.
If you're looking at monitoring 1000's of systems though, maybe you have enough of a budget to be able to plan around these issues.
I'm sure that ultimately all monitoring apps run into issues with how many (parameters * servers) each monitoring system can monitor too.
You say that your department has lots of money and you are happy to waste it on commercial software. You also say you like open source but are worried about it being maintained and supported.
Sounds to me like you could create some jobs for a couple of talented programmers. Everyone wins.
This post contains benzene, nitrosamines, formaldehyde and hydrogen cyanide.
We are use looperng to do most of our snmp collection and have it integrated to netcool which is our commercial platform. we were using it since the earlir days of looper (not ng).
It has scaled very well since now we have almost four looper collection stations collecting traps from over 6000 elements.
Netcool is our main platform but we also use NNM and ehealth.
We tried nagios when it was known as netsaint and found it to have poor performance at the time.
http://www.opensims.org
Though alternatives do exist.
If you've got a cluster or otherwise "clumpy" network, ganglia is the ideal end-user-visible monitoring tool - lots of pretty and informative graphs, multicast based so no heavy network load, no particularly sensitive information unless you choose to reveal it.
For filesystem security, samhain is mature, secure and imho very nice, though the good web-frontend is non-free.
For network security, nessus scan on a daily basis.
www.opennms.org
If you have the budget why not fund a project to make sure it doesn't stop being developed. This would make good use of that money and help everyone.
Not sure how helpful this will be in huge environments, I live in the small to midsize market, but here are some tools that I have found useful in the past:
Not exactly a monitoring tool, but definitely the most versatile all around auditor I have ever found: Nessus.
Ettercap is a good sniffer.
The MRTG tool has been a godsend when I have had managed devices to deal with, and I have heard very good things about the RRD tool and Cacti.
Tripwire is freely available for Linux and the BSDs, though the Win32 version has not been open-sourced.
One tool I have not been able to find in F/OSS is a Windows event log monitor (though believe me I'm still looking).
I have had good results from OpenNMS.
You would be amazed at what you can get out of a few things like Perl, MYsql, SNORT, GNUPLOT, UCD-SNMP and and apache server. Be creative, and create your own network management system. I have used HPOV, SNMPc, Cabletron Stectrum, CiscoWorks and others to manage 1000+ node, global networks. The later being the worst software i ever laid my finger tips on. The BIG boys are good but you can do the same with some simple programming and creativity. Plus maybe you'll have the chance to share your work with the rest of the world. Good Luck.
I've recently taken the position in a large multi-national company to monitor our devices as well. We are only about a year into it. We also use a mixture of OpenSouce and commercial tools. The thing I've noticed mostly with most software today is that they are becoming more and more a mix. For instance our main managment software is currently Netcool. Netcool is commercial but it utilizes Apache, Tomcat and alot of Perl/CGI. On the commercial side it uses Java, and thier own proprietary DB called Omnibus as well as a Sybase Communications protocal. The end result it is extremenly flexible and works across all platforms I've tried with little trouble. I really don't think the future of software will be as black and white as OpenSource vs Commercial. Its really going to be a mix of both. The benifit is lower costs to the customer because of less development effort for the vendor. I like Netcool alot because its very nature is to be as flexable as possible. Its possible to use Netcool for other puposes that the develpers never thought of. Our HP NNM system is good but is nowhere near as flexible simply because it can only go as far as HP wants it to go. Hence it is only used for network discovery and SMNP trap collection. Bottom line is that the best choice in software today is the software the includes some OpenSource code.
Both tools give us a much better view of our network, and what our various devices are doing.
Man watching 6 MSCE's around a sun box, looks alot like the opening scene's of 2001:space odyssey...
A great tool that I've got some experience with is big brother
The monitoring won't blow your mind, but the notifications are pretty cool. It looks like they've decided to go with a less than free license though. I don't know if you're a crusader for those sorts of things. They do offer source code, and many platforms for their client software.
Technology Consulting & Free Downloads
What tools are popular for IP address management? Most of what I've seen is pretty basic. Are there any tools that are good at combining device inventory managment with address management and assigning addresses to new devices according to configurable rules?
For example, if I need to add 50 new web servers and each web server needs a redundant pair of NICs sharing one IP plus. Plus allowing multiple IPs per device or multiple devices per IP (e.g. VRRP, HSRP).
Spreadsheets seem faster than web based systems if it takes multiple queries to get all the info you need. A flexible query system and integration of address management with server monitoring seems like it would be very useful, but I haven't come found anything yet.
Seriously, it's hugely useful. It's very nice to be able to show management that you not only have a config backup system for your network devices, but your backup system is also doubling as a change control system. It's at http://www.shrubbery.net/rancid . I tend to use something like webCVS with it, to let folks browse through the CVS configs (you will, of course, want to use authentication to restrict access to webcvs).
I did not see anyone mention Cricket at http://cricket.sourceforge.net/. Our ISP uses Cricket w/ RRDTool to monitor bandwidth on all of their infrastructure. We track our Internet usage this way. It appears to be a pretty good tool for mid to large networks. Just another good OSS tool.
For handling traps I use SNMPTT with Net-SNMP. It allows for traps to be converted to more meaningful messages using variable substitution.
I've seen it used in small and large environments.
http://www.snmptt.org/
http://www.net-snmp.org/
http://www.stemsystems.com
This appears to be perl-based and Damian Conway, Randal Schwartz and Aeleen Frisch are/were advisors.
But, it doesn't seem that much has been done in over
2 years. They did have a writeup in SysAdmin mag, back when they first started.
Pain is merely failure leaving the body
iftop, nload, tcpdump, ngrep, hping2,fping, mrtg-rrd, RRDGraph, tracepath, bing, arping/ rrdtool /rrdworld/smokeping.html
not tested yet (smoke ping)
http://people.ee.ethz.ch/~oetiker/webtools
Developed at Dartmouth.edu, now spun off for profit. Cheap and it works real good. I have spoken with many Tivoli/openview users who say Intermapper works far better on thier networks. -- nobody and I speak for one another