What Would You Want In a Large-Scale Monitoring System?

Krneki writes "I've been developing monitoring solutions for the last five years. I have used Cacti, Nagios, WhatsUP, PRTG, OpManager, MOM, Perl-scripts solutions, ... Today I have changed employer and I have been asked to develop a new monitoring solution from scratch (5,000 devices). My objective is to deliver a solution that will cover both the network devices, servers and applications. The final product must be very easy to understand as it will be used also by help support to diagnose problems during the night. I need a powerful tool that will cover all I need and yet deliver a nice 2D map of the company IT infrastructure. I like Cacti, but usually I use it only for performance monitoring, since pooling can't be set to 5 or 10 sec interval for huge networks. I'm thinking about Nagios (but the 2D map is hard to understand), or maybe OpManager. What monitoring solution do you use and why?"

Comments

Before I get flamed...

[jwilki1]

I am going through this right now and am using and have used all the above mentioned solution. We are leaning towards System Center Operation Manager. http://www.microsoft.com/systemcenter/operationsmanager/en/us/default.aspx If you had told me 6 months about that it would be the way to go, I would have said over my dead body, but it has come a very long way in terms of usability and ease of setup.

Re:OpenNMS

[mu51c10rd]

I use OpenNMS as well. I actually migrated off of Nagios to OpenNMS. Tried out Zenoss and Cacti as well. While any of these are better than OpenView IMHO, I liked OpenNMS's full suite of functionality without having to pay for the 'commercial' version.

ZenOSS all the way

[Midnight Warrior]

We use ZenOSS exclusively at work and have enjoyed every minute of it. Pro's include:

• 2D map with status of all nodes or submaps, organized by network
• Application monitoring, with more advanced maps available for purchase (Oracle, JBoss, Cisco) for those things you already paid a lot of money for
• Performance monitoring via SNMP or other data sources using RRDtool internally which includes graphs linked to each other during zoom in/out or panning
• Nagios plugins already do some of the heavy lifting
• Built-in support for watching Windows servers (any metric accessible via WMI)
• Access control using at least LDAP and Active Directory
• Secondary data collectors for those networks which are too big for just one central source
• Highly customizable through Python
• It has so, so much more than pathetic commercial solutions like OpenView

Cons:

• You have to keep your eye on the back end database
• It still takes a long, long time to tune it to remove noise events
• If you don't know Python, it can be tough in a few places
• Proper support is not cheap

Re:A more interesting question

[abigor]

The big questions are:

Will your solution need to support snmp v3?

Do the devices you talk to have published oids?

Do you need source code to extend it?

If yes to these, OpenNms is a great bet.

I Hate War Rooms

[afabbro]

I really don't like the "War Room" video wall concept. I suspect such walls are made to look cool rather than to monitor.

What you want in large-scale monitoring is:

• The ability to map complex relationships. I don't want 50 alerts that I can't reach host X, host Y, etc. I want one alert that I can't reach router A. Even better, I want to map things so that I can say "end user application XYZ is not accessible in Kansas due to X being down".
• I want my monitoring solution to understand HA and service degredation. I want programmable rules about what happens when X is down or Y is down.
• I want many options for escalation. If X doesn't acknowledge, try Y after 15 mins, etc.
• I don't ever, ever want a pager to explode or be flooded. A problem should be noticed once and tracked. There should be no pager blizzards.
• Of course, I don't want this thing relying on my mail system for paging because, of course, my mail system could go down. An ability to dial out if the mail system is down would be nice.
• I want agents, hooks, interfaces, third-party add-ons, and every possible way of tying something into the monitoring system. I don't want dumb limitations like "you can only get an exit code from the OS and it acts on that" or something. For big monitoring, it's almost mandatory that some kind of API for agents is exposed.
• I want "I'm working on it, stop paging" blackouts. I want to be reminded to lift them.
• I want it to tie into my change-management system. If I open a ticket and say that server X is down for 2 hours on this date, I don't want to have to remember to black it out.
• I want reports. I don't care about silly little charts and graphs, but a history of everything that has every gone wrong with device Y would be nice.
• I want more info on my page-receiving device than just "HOST X IS DOWN". I want context so I can decide if I have to drop everything immediately.

Etcetera. These are some of the things that make sane large monitoring systems. I don't think any open source product has all of them, alas.

Zabbix is easy to maintain and flexible

[bigtrike]

Zabbix allows you to build some fairly powerful rulesets and chains of overrides using its web gui. It's not perfect, but it keeps improving and the attitude of the developers is friendly unlike some of the other projects.

Re:Zenoss

[Ranger Rick]

And this is why we (OpenNMS) don't play the per-node. It's not any harder to run OpenNMS when managing 1000 nodes than when managing 100, you only need to scale hardware appropriately. Per-node pricing is an artificial limitation.

We also don't play the "you get a special price behind closed doors" game, our support prices are public, fair, and the same for everyone -- and that's only if you need commerical support -- our prices are $0 if you don't need or want support.

If you do the math, it's $0 for the software, plus $14,995/year for support for any number of nodes, and the software is 100% open-source and fully capable of replacing or exceeding OpenView. ;)