Slashdot Mirror
What Would You Want In a Large-Scale Monitoring System?
Krneki writes "I've been developing monitoring solutions for the last five years. I have used Cacti, Nagios, WhatsUP, PRTG, OpManager, MOM, Perl-scripts solutions, ... Today I have changed employer and I have been asked to develop a new monitoring solution from scratch (5,000 devices). My objective is to deliver a solution that will cover both the network devices, servers and applications. The final product must be very easy to understand as it will be used also by help support to diagnose problems during the night. I need a powerful tool that will cover all I need and yet deliver a nice 2D map of the company IT infrastructure. I like Cacti, but usually I use it only for performance monitoring, since pooling can't be set to 5 or 10 sec interval for huge networks. I'm thinking about Nagios (but the 2D map is hard to understand), or maybe OpManager. What monitoring solution do you use and why?"
Publish them in DNS, and have the NSA monitor them for me!
"Speaking the Truth in times of universal deceit is a revolutionary act." -- George Orwell
Hyperic HQ may be worth checking out.
Don't assume that you can successfully diagnose the problem based on your understanding of the indicators. You don't know my institutional context. Instead, give me a decision support system that I can use by adding rules that key off the monitored indicators and inject some of our own expertise into the diagnostic process.
That's all you should need. For 5000 devices I don't know that any of the options you listed would be appropriate.
OpenNMS is much more than monitoring, but I think that you'll appreciate the other features as well.
http://www.opennms.org/
Enjoy.
What limitations exist in current solutions that justifying developing a new one from scratch ?
I built a smal program to updating all bash profiles with timestamps, compare changes every few ticks and save changes to a database where the users were given unique ID and it would associated a parent child relationship when users su/sudo to show heightened privileges. Very useful as sys admins are known to wipe their bash historys and this kept a centralized history with relationships.
This is my sig. There are many like it but this one is mine.
I am going through this right now and am using and have used all the above mentioned solution. We are leaning towards System Center Operation Manager. http://www.microsoft.com/systemcenter/operationsmanager/en/us/default.aspx If you had told me 6 months about that it would be the way to go, I would have said over my dead body, but it has come a very long way in terms of usability and ease of setup.
There's only one way! Splunk it!
You can also look into Zabbix. It's open source, and has Enterprise support available. I haven't used it yet, but as soon as I have a spare moment to breath I intend to test it out for use in my environment.
Nothing to see here
You can pry my GKrellM from my cold, dead hands!
Yeah, for 5000 devices, the displays start to take up quite a bit of screen space, but that's what video walls are for!
*cough*
I've played with quite a few in the past. For your application, I'd stick with Nagios... the new version has plenty of scalability features. It has a pretty steep learning curve and just about all the configuration is text-based, but I've always found it well worth the investment in time. I currently use it to monitor services on only a couple hundred devices, and there are plenty of plugins to make it more useful. It's not great for creating and visualizing large 2D and 3D maps, but it has all the necessary hooks for it and with a bit of scripting you should be able to generate more useful views and reports of your farm.
Corps seem to buy into the commercial HP Openview a lot, but no one I've talked to that uses it seems to like it.
On a few of my servers, I also like to run Munin... it tracks and displays a little bit more information than Nagios, such as graphs of sensors, uptime, UPS stats, etc. It's come in handy on several occasions when Nagios had simply shown me that "the server went down", but the information from Munin showed that "the server room temperature started climbing up to 90F starting at 2AM".
For real-time monitoring, I really like GKrellm, which has a server/client mode of operation. It wouldn't be practical to have it up all the time, but it would be sweet to set up the daemon and have a link from Nagios launch a gkrellm client to a remote server, where you can see the affects of anything you do in real-time (rather than waiting for Nagios to refresh in 5-10 minutes).
MRTG does it right...most of the others do it wrong
When rolling up a days worth of data (averaging), you loose the peak information on most monitoring systems
So your 380Mbps peak that you had an hour ago is fine on today's graph
But tomorrow, when you look at "yesterdays" graph...the peak is down to 100Mbps
and next week, when you look at "last weeks" graph...there's a little 50Mbps peak
Damnit... I want to keep information on my peaks for capacity planning!
Twitter client, facebook integration, google maps mashup.
And a pony.
Thanks
I was really impressed by Zenoss, which has all the slick features that cost the earth from vendors like HP for Openview. You get automatic discovery, CMDB inventory, availability monitoring, alerting, and performance graphs all in a web portal.
You get open source, commercial support, and a good community of users and plug-in developers. The best of both worlds IMHO.
http://www.spiceworks.com/
Not sure how far it scales but I have played with it on some small installations, very easy to manage.
I have used Cacti but never felt it was mature or robust enough for very large environments
SCOM, System Center Operations Manager we are deploying now for our enterprise, however I would be afraid to manage IT on my own as it is a large system on to it self, yet very powerfull.
EA David Gardner -"... but the consumers have proven that actually what they want is fun."
I use Nagios and some custom rolled scripts myself.
For some other options, Nagios has now been forked, so if that is "close" to what you want, you may want to contribute to Icinga.
Reconnoiter also looked pretty kewl, but they haven't released anything yet, but it looks like they are planning it to be very scalable.
That is what I would want! ^^
Any sufficiently advanced intelligence is indistinguishable from stupidity.
I've really been impressed with OpsView. Can't say how well it scales on huge networks (but there are options for having multiple servers). Its based on Nagios, but its a lot less of a pain to configure and has a pretty good web interface. The only thing I don't really like is its graphing functionality. I use Cacti for monitoring bandwidth/server load/etc. But for availability checking OpsView does a fantastic job. I'm using it to monitor maybe twenty devices, including Linux and Windows servers, and HP/Cisco network devices. I tried Zenoss as well, but it seemed awkward to work with. For instance, with Opsview/nagios it's easy to add a check to verify that a DNS server is correctly resolving a record in a particular zone. I remember it was going to be a pain to monitor some of the things I wanted to with Zenoss. Maybe I'm biased because I used plain old Nagios for a while before I tried OpsView and Zenoss.
Every time you post an article on Slashdot, I kill a server. Think of the servers!
Talk to these guys http://netfuel.com. Excellent client server monitoring, started out as a trading app monitoring tool and grew. Scales to thousands of nodes and has lots of options including several API's for integration.
I spent last year converting a shop from OpenView to Nagios. They were in the same neighborhood as you (~5000 devices).
If you do not like the Nagios UI, you could create something else. The native Nagios UI is CGI based and implemented in C. The documentation is good and the sources are well commented.
The hardest decision about Nagios is how to implement the monitoring. I went w/SNMP (polling, not traps) for the most part. Sorting out all the Nagios plugins is something of a chore and many of them seem incomplete and abandoned.
MRTG also integrates w/Nagios, which can be useful.
Good luck.
http://argus.tcp4me.com/
Jeremy Kister
http://jeremy.kister.net./
Cons:
Some basic functionality, maybe not at the development level your looking for yet.
Shurely Shome cash is available to actually PAY somebody to sort this out, rather than ask for free help on /.? Hic - burp.
I want lots of buttons and dials! And flashing lights!
+1 IDisagreeSoHeMustBeATrollOrAnAstroturferOrAShill
The mistake is trying to monitor thousands of devices on a 2-D map. I'll look pretty to the suits, but be useless for the users. Nothing but endless slow clicky clicky clicky.
Give them a text screen of whats currently down ... that'll work.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I've only used MOM but for what it's worth the diagramming capabilities are much improved with the new visio plugin. Previously you could export your diagrams from OpsMgr to visio, but with the new plugin the visio diagrams reflect live health state. You can also create whatever diagram you want in visio and then tie it to monitored objects living in OpsMgr (for example rack diagrams)
"The final product must be very easy to understand as it will be used also by help support to diagnose problems during the night".
If you can develop a monitoring solution that night time support personnel can understand to diagnose a problem quickly and properly, I am going to nominate you for a peace prize. BTW, give it lots of rapidly updating graphs and eye candy, you know, bling the sh1t out of it. Management types love that.
For the hassle of developing your own solution, you could just cough up the $20k for a statseeker box (hardware and software included in that price too) and then you can monitor up to 200,000 interfaces with ease. We poll 186,000 interfaces twice every 60 seconds and we have 2 years of data taking up only 17 gig of space and the boxes still is speedy. I've used nagios, cacti and the like for years and they are great for smaller deployment but statseeker smokes them from almost every aspect. I know they have upgraded options if you need more than 200K of interfaces but so far we haven't hit the limit.
Have you checked out Intellipool Network Monitor ? They got a new version coming out (version 4 I believe), it got maps with drill down and all that stuff, the distributed edition can distribute workload over several gateways also, so monitoring large scale networks is not an issue.
http://www.intellipool.se/forum/index.php?automodule=blog&blogid=1&showentry=170
Here's a similar thread from a while back that covers most of the options: http://linux.slashdot.org/article.pl?sid=07/03/05/1812247
I have looked at Cacti, Nagios, and a few others, but I think rolling your own is easy enough and gives you the best flexibility. You could also use Nagios, or others, for example, and simply pull the results into your own system.
I built and managed a software system for me previous employer called the SMS (Server Management System). It basically tracked 50 of our web servers, database servers, and Endeca (full text search) farms at data centers spread around the country. It was pretty simple.
The system did push and pull operations. First, the system was built in PHP.
In order to push commands to the servers I used PEAR SSH2 class for communication when it became stable. Another option (and what I did back in 2003) was to use exec and other command line functions in PHP in conjunction with a SETUID script (written in C) -- which gave the command line output from PHP "true" rootly powers. The problem was I had to enter a password for each server I wanted to connect to, and the PHP functions couldn't handle real time input/output, so I designed the system to work by creating an SSH2 key pair on my master monitoring server and put it's public key on each of our external servers for passwordless SSH.
The pull part of the system simply had a PHP script running on a cron per server, that would deliver information about the health of the server, its running processes, etc, to the main SMS server every 5 minutes. All load activity for all servers was logged as well to MySQL. The push operations were used to update those scripts, as well as restart Daemons on command, clear cache (such as after we did a database update), etc. It was a pretty robust system and really automated the functions of our company, to where we could perform a FULL Database Update to our 30 web servers simultaneously (using PHP and fork()), clear all cache, etc, in under an hour. We would the monitor the servers using the SMS's main screen which showed real time server stats (updated every 5 minutes, or you could "force" a push operation to get the status). If we needed to rollback the update, that was a simple mouse click away too.
I also had a hidden screen that let me run any series of commands as root on any number of servers. Everyone objected to it but I convinced my boss to let me put it in. All of our servers were a mouse click away from being "rm -rf *" 'ed. ROFL. Anyway, I hope my little story about my system helps you out, in either avoiding what I did (LOL) or by giving you ideas.
What would I want in a monitoring system? The first thing that pops into my head is "lots and lots of knobs." The kind where when you turn them you get a nice satisfying click. And blinking lights. Lots of switches. Things that go "ping" at regular intervals would be nice. Oh! And a nice big screen that says, "All systems nominal" all the time.
This doesn't seem to have already be suggested, but we use SolarWinds Orion. Its cheaper than many of the big systems, such as HP OpenView - and much simpler to use and operate.
The basic Orion package, which you can get for $2000 for up to 100 servers, will pull the usual CPU/RAM/Disk/Network statistics via SNMP. Built in is a mapping engine, that allows you to take a network map, and drop active elements onto it for live interfaces and device information. In a NOC environment, you can show this on a screen and it'll even sound an alarm when a system Alert fires through the website.
You can then bolt on additional modules, such as their Application Performance Monitor. It has ready to use templates for common business applications, Exchange, Apache, IIS etc. You can also create your own mixing, SNMP, WMI and User Experience monitors. User Experience monitors for example allow you to actively poll HTTP/FTP/DNS/SMTP/IMAP/POP etc, services to ensure they are not only UP but responding as they should to requests.
For scaling, you can tack on Additional Pollers to spread polling load across them. You can also use hot-standby pollers to resume the work of a failed poller.
Just my 2 cents, and not a corporate plug - just a very content user!
As one of the core devs and large user, I can tell you it scales well, develops easy and has a lot prefab. The system does everything you're asking for. Let me know if you need help or paid support.
Custom electronics and digital signage for your business: www.evcircuits.com
I'm a software developer and, sadly, my knowledge of hardware systems isn't always what it should be. When I write an application to run on a server and it starts to get slow, I want to know where the bottleneck is. Is my application CPU-bound? I/O-bound? Memory-bound? Do I need more memory? Faster storage? More cores or faster processor speed? Is it the network that's causing the problem? I can usually figure this out using various linux command-line programs like netstat and top and all that, but I would sure love a big fat GUI to make it more graphic. I found something like this once and couldn't remember what it was called. It required all kinds of diagnostic utilities be manually installed.
Ideally, you could view a machine and get some quick idea of where the bottlenecks lie. Maybe that's asking a bit much, but the closer you can get to a single control panel where I could see see all my machines in a list with a status indicator and then drill down machine-by-machine, the happier I would be. It would be even cooler if the machines could contact me when they experience times of overload so that I could get a feel for when the trying times are so I can watch them more closely. I'm imagining a daemon that runs on each server and an admin gui that can speak to that daemon somehow. It would also be nice to have hooks so that I can easily report performance profiling information to the GUI from within my application.
The Activity Monitor utility found on Macs is pretty close to what I'm imagining.
What I'd like to see is a good monitoring support for JMX-capable Java services.
It'd be nice to set up an alarm based on time spent in garbage collector in a JVM running our application, for example.
I think Nagios should provide a good start.. they've recently added a lot of scalability features. Though it has a high learning curve and all of its configuration is done in text, I've always found it worth the time and effort. I currently use it to monitor services on a couple hundred machines.
Munin is a bit simpler, but I like the graphs it provides which occasionally are more useful than the data Nagios provides. In some cases, Nagios might tell me that a server went down, but I'd look at Munin and see that the server room temperature spiked to 90F before then. Also it's neat to see the uptime graphs for the year.
While it might not be practical to use GKrellm all the time, I'd find it useful for real-time feedback. You might set something up where you can launch a gkrellm client to a server of interest while you're working on it. Then you can see the effects of things you do without waiting for Nagios to refresh in 5-10 minutes.
That is the solution that i have implemented for our little environment that consists of about 50-ish solaris (8 and 10) servers, 80-ish windows servers, about 500 linux servers, and 40-odd cisco switches. Hobbit handles all host monitoring: availability, services, and a bunch of custom scripts written for it to check various aspects of our HPC grid, plus the SMS sending through an old nokia connected to the comm port of a solaris box. Smokeping is there to check latency, and cacti primarily for network traffic volume, and a custom module for FlexLM licenses. Works like a charm
It requires a substantial investment in learning how to set it up and how to use it, but then so do the big commercial products like OpenView and Tivoli.
The Dude is what we have been toying with lately.
http://www.mikrotik.com/thedude.php
It's *not* open-source, but it IS inexpensive. When I worked at a NOC, we used it to monitor hundreds of routers, switches, mainframes, Tandem systems, UNIX boxes, etc. It takes SNMP traps and displays them graphically on a 2D map, and the 2D map is very nicely implemented. You can have your top level view made up only of groups of devices, so if a group goes red you double-click that group to view its members and see which device actually has the error. IIRC, you can nest groups, so it ends up being a fairly scalable solution when you talk about screen space.
I claim first use of "Error No. 0B" - or "No. 0B error." It'll be the new ID 10T!
I use Nagios, but on a smaller scale than what you describe. I love the system, but I would imagine it being difficult to maintain on a larger scale. Nagios itself is requires manual configuration unless you use a separate front-end like Centreon, which is also far from perfect..
A friend of mine has been toying with OpenNMS for the last few months, and he's pretty happy with it although he reports that it's still got some minor issues that need to be worked out. It's FCAPS compliant, and I get the impression that it might be the better option for handling a large installation. There's a new version scheduled for release soon, so we'll see what that brings to the table.
There's also recently been an announcement of a Nagios fork, scheduled for release sometime around October. I forget the site or project name but I'm sure a bit of Googling will locate their site for you.
This sounds like the perfect opportunity to harness the power of app partisans to fix the wikipedia article comparing monitoring software. See http://en.wikipedia.org/wiki/Comparison_of_network_monitoring_systems . Some good info there. And probably bad info. But certainly has a good list of applications. Also, if you like nagios (and he seems to me to be fair to a lot of packages, including ossim), you might check out some of David Josephsen's articles (or Nagios book), etc.. His site is http://www.skeptech.org/ . A decent design article is here -- Best Practices for Designing a Nagios Monitoring System -- http://www.informit.com/articles/printerfriendly.aspx?p=705685 .
I really don't like the "War Room" video wall concept. I suspect such walls are made to look cool rather than to monitor.
What you want in large-scale monitoring is:
Etcetera. These are some of the things that make sane large monitoring systems. I don't think any open source product has all of them, alas.
Advice: on VPS providers
Which revision?
i tried it for a couple of months, and rather like it, but it'd simply stop monitoring stuff, triggers wouldn't fire reliably etc.
Deleted
Focus on usability and rapid deployment rather than wide-ranging featuresets that sit on the shelf for a decade. Nearly all products in this space really, really suck.
Conformity is the jailer of freedom and enemy of growth. -JFK
Big fan of intermapper (www.dartware.com) ... It can use nagios plugins as well.
It's fairly cheap.. We monitor about 1250 devices at the moment with it... can be set all way down to 5 seconds.
Server and Client are both in Java... so more or less it runs on any platform.
They give out 30 day demo keys.
I use Cacti, with THold and weathermap plugins.
But then I'm biased.
-- I care not for your foolish signatures.
It's not free, but if you want something with maps that is very easy to set up, and for regular users to work with, take a look at it.
As long as it has more cow bell!!
Foglight from Quest Software covers out most of the requirements out of the box and is script friendly. Its all Java based thou it itself isn't an OpenSource project.
There is also a community around customization that might be worth checking out over at www.foglight.org.
Then I advise you to let all the connection checks do by a machine dedicated for this task.
The statistical checks on each of the 5000 machines itself 24 hours a day while pushing/pulling data at intervals to that dedicated machine.
I advise 2 or more measuring machines each on a separate network each doing the same task and synchronizing their data for redundancy.
You can always try INM (http://www.intellipool.se/).
It's quite feature rich and it's worth a look.
--- Reality doesn't care about your opinions, it happens anyway and if you are in the way you'll get squished.
Was a step above Nagios in terms of reliability (I didn't have to restart the server four times a day just to keep it running), and did much better at autodiscovery.
That fact that it is also NRPE compatible was a plus - I could use all the Nagios plugins and check scripts I'd written.
I was also planning on using it to launch a more aggressive webmin-style management solution - since OpenNMS built this great database of data about my devices and hosts, I could use it to do actual management - change data/settings.
Cons: It's a Java/Tomcat tool, as much as that is really a con. It's not like you need to run Jboss or Websphere to use it (though I suppose you could).
I just did a quick survey and evaluation of the open source monitoring-market for my company, and found a few shortcomings/frustrations in a few aspects where none of the evaluated system seems to get it 100% right.
Transparent Planned Design
Many solutions out there seems to have been developed in what can only be described as an "organic" process. I.E. a few scripts were used from start, were hooked up with some other scripts, were slammed into a web-interface, got some more features, then something central were ripped out and replaced to allow yet more features and so on and so forth. (Read: Nagios) While this is of course often the best way to get something working for a particular need, and on a tight budget, it makes adoption really hard unless you happen to have exactly the same need.
Event management
Does anyone know a solution that can both receive from syslog and decode traps with a given MIB, and then do some simple logic, like squashing repeats, displaying on a web-page with archival-options, and dispatch to mail/sms based on configurable rules? Except for ZenOSS (and ZenOSS have other problems), I haven't found a single sensible system that does this.
Modularity/Seamless Integration
Since much of the monitoring systems out there doesn't seem to have a clear design, it's often very hard to add missing features. I.E. project X missing an event manager, or is the builtin not satisfactory? No probs, I'll just, ehh, where does this wire come from? Is this really a socket? Did anyone really connect that? It's ok with blackbox-solutions, as long as they serve all my needs, and have clear interfaces to combine with other solutions that serves related needs, but sadly no solution evaluated does everything we need it to and we end up struggling with manual routines to compensate for it.
Complexity
There are a few really neat systems that does almost everything one can ask for. (Short of flying cars). Unfortunately, the ones we've tried have always turned out to be very complex, and also do a lot of things we didn't want. Since it's then often not very modular, it hard to get it stop doing the things we don't want, or change the things we need implemented slightly differently. Also the huge codebase that comes along with trying to scratch everyones itch seems to get it's share of bugs, and troubleshooting in large more or less opaque systems is not a fun task.
The Perfect Monitoring System
After evaluating all options we could find, we've come to the conclusion that none of the systems we've looked at or tested really fits our needs (Although ZenOSS came close, we encountered just too many bugs and oddities to keep investing time in it). Furthermore, we could not find a combination of systems that integrates well, and together fits our needs, which I personally see as a bigger problem.
What I would really want to see in the world of Open Source Monitoring, is an eco-system of monitoring apps with an overarching design/architecture. Design a framework where different entities and steps in the monitoring are clearly defined and interfaced with each other, but still allows for differing implementations, and integration with unforeseen needs. For example, at our shop, we continuously analyze roughly 700mbit of streaming video for availability and quality. Noone designing a monitoring system could probably forsee this as an appliance, but in The Perfect Monitoring System, it should be clear for the average-skilled hacker how to integrate it.
As you've discovered, the free systems will fall over and die once they're past a certain size. I've worked with Tivoli customers that have tens of thousands of servers, and for all of its problems, Tivoli is scalable.
They way they do it is, obviously, divide and conquer. There are specific ways that they do it. I'm mixing up the architecture and terminology on purpose, because the Tivoli terminology will confuse you.
* there are agents on every box that do the monitoring
* they report to a region
* those regions report to a top-level region
That doesn't mean that you can't have a poll engine somewhere, poking machines. What it means is that if you do have a poll engine, it manages a specific number of machines and reports results upwards if necessary.
Tivoli has a bunch of other stuff that makes things like this easier, like profile-based management and lightweight (relatively) endpoints. You can simulate that using a centralized source control system that everything pulls from - configuration of monitoring, etc comes from those config files, and every your agents pull their configs depending on criteria, like their hostname, ip, or by looking in some file for what they're supposed to get. This also becomes your shared filesystem of sorts, because you can pull monitoring binaries from them as well as config files.
Management of alerts is always a problem. Having worked on an EMS I can tell you that all the free ones suck, so it doesn't matter which one you pick. Spend some money and buy the BMC Event Manager.
Besides that, avoid UDP - it fails when you need it the most. And don't do management by exception - it's for lazy admins. Instead, do some kind of thresholding on your stuff, so you can tell before it fails. MBE gets you there 5 minutes before your users call. Real monitoring allows you to ignore the problem for weeks, or at least blame someone else for not acting when the systems finally do fail.
Same size company. Currently using Nagios, Cacti, Kbox, MOM, MARS, ADM.... god knows what else. Beta testing a product called Nimsoft and it's actually really nice.
Hello!
I used to work for an ISP.
For monitoring and polling more than 8000 devices, i used cricket (a set of cgi-perl script).
Just search for it in sourceforge, its a couple of powerfull perl scripts that helps you creating RRDs.
Then later you can use their own cgi-scripts to visualize them on the web......or use a third-party script.
The thing about cricket was that it polled more than 10k objects in very few time, and had some interesting threeshold alerts (mail, sms, web).
The best practice out there is to merge something between cricket and cacti/openNMS.
hope my experience helps you.
regards,
Nicolas Royo
Needed features in random order: ...and a system for annotating the above. Raw data is neat, annotated data is even better.
* Scalability - few k machines is minimum. This probably means smart, decentralized collection and aggregation of data.
* Flexible whitebox monitoring - for given class of devices, I should be able to configure how to fetch this device's data (http, smnp, ssh+command, rpc, you-name-it) and how to interpret it ("read the status page there, get this and that value").
* Flexible blackbox monitoring - for given class of devices, I should be able to configure a set of actions that should be performed on it (fetch a page, ssh into, ping) and how results of that action should be interpreted (ok/nok, time to complete, etc.).
* Easy way to tag (source/machine/network segment) and aggregate (max/min/mean/stddev/%ile/sum) of the monitoring data.
* Some language to easily calculate derivative values from the data above.
* Interface for defining graphs, using collected data.
*
* Alerting subsystem, which should allow for defining different destinations, together with escalation rules. And custom alerts - using the .
* (nice to have) HTTP server with a simple HTML templating, to allow for easy creation of arbitrary dashboards.
* (if you have the above) predefined templates for most of common things. Both detailed ("everything about device X") and general ("if the background of the page is green, you're fine! If it's not, here you'll find a concise list of what's broken").
* hooks/libraries to use collected data "outside" of the system
I realize that's a lot, but boy, such system would be very useful and flexible.
-- we're here you're not
If your monitoring something of that scale, you should probably look into a profession solution.
I use Zenoss (open source) and like it quite a bit. It takes time to customize for your setup, but unless you have a bland network, that is almost always the case. I will say this, it's much easier to setup the Nagios was a couple of years ago when I was using Nagios. Though I've heard there has been some improvement.
That seems to be one split between the various different monitoring systems out there. Either it's intended for the network guys and its only understanding of host/server metrics is what it can poll out of SNMP, or it's SA-focused but has few of the broad, large-scale network features that the network guys want.
Personally (as an SA), I've been very satisfied with Xymon (nee Hobbit, which was a fork/rebuild of Big Brother). Performance is great, even with 5000+ devices, it's got an open and simple-to-parse protocol, and an incredibly extensible architecture. As an SA, being able to script up a monitor and throw it into a data stream as plain text makes it very easy to develop new tests (or add simple monitoring/logging/rrdgraphing) out of pre-existing scripts. Don't limit yourself to what SNMP gives you if you're dealing with servers, services, and higher-level app testing. KISS: http://en.wikibooks.org/wiki/Hobbit_Design_Document
Whatever you do, pick something you can easily customize: Hack together three different monitoring systems to come up with a best-of-both worlds solution. Everyone's monitoring needs are different, after all.
Hire a Linux system administrator, systems engineer,
For COTS I'd go with CA (formerly Concord) eHealth. Their SNMP agent is light-weight, flexible and the product scales out very well. It's also reasonably straight forward to deploy and configure and quite expandable. If you want an open source alternative that will grow with you and/or offer support options down the road I'd give Zenoss a shot. Steer clear of HP, BMC, or IBM solutions due to complexity and/or price.
If you want a monitor that can display useful information about thousands of nodes on a single display try clumon. We use it for our 1000+ node clusters. The software was developed in-house but is available under the University of Illinois/NCSA Open Source License Copyright (noticeware). If you're just going to use this in-house, the license shouldn't be an issue.
You can see a sample clumon display of a working cluster at NCSA Linux Cluster Monitor.The clumon page for that cluster shows you each the job status of each individual node (if the node is colored, it has a job assigned), the load on the machine (the height of the line is proportional to the load, and red tips show loads over 1.0 per cpu) and the service status (green underline is ready, yellow/black stripes is offline, and red is unexpected offline/no comms). If you mouse-over a node, a status box pops up with more information on that specific node.
As this was designed for a cluster with the Torque resource manager, it won't be exactly what you need, but since you are willing to write a monitor from scratch, it might be a really useful starting point. Design-wise, this monitor allows the engineer or manager to see what's going on in general, with problem areas being immediately obvious, and without being overly cluttered.
The open source Performance Co-Pilot software runs on each node to collect information, which is polled by the central server. Back end is MySQL. The dynamic display is PHP.
Straightforward, useful and very configurable.
The Internet has no garbage collection
He's just asking what to use
Noone
NOOOOOONE!!!!
But in a non-AC mode I would say something that agreed with you and possibly even offer my services in the actual coding of the said system. Just send me a message at /dev/null if you start this project.
LANREV
Seriously. That is what you are describing. That is what I use for well over 5000 devices.
http://www.lanrev.com/
I know I'm late to the party, but I haven't seen anyone bring this one up yet: Real-world alarm/notification capability (pager, buzzer, a machine that goes bing, something like that)
My reasoning: I run a small IT business with various support contracts. I, and probably quite a few others, can't afford to pay someone to sit at a monitor and watch a screen (or a bunch of screens) whilst tied to a desk.
Most of the monitoring solutions (Nagios, others) are capable of off-site notification, but it's the "last yard" that's the problem--how to tell someone, even a non-techy, there's a problem so he can call in the cavalry. Despite Verizon's "largest 3G network" claim, a lot of my clients and workers in Silicon Holler don't have cell coverage...so SMS, pagers, etc. aren't all that reliable. But we do have office staff who could be around to listen for an alarm, and we have a solid internet connection...so calling for help via the network is viable, but not paying someone to be otherwise unproductive because they can't go anywhere else.
I even started developing my own ATMEGA based solution...still working on it, and I think it's completely doable. If I ever get it up and running, I'll publish the plans, code, and scripts/software under GPL and let someone else worry about the marketing.
Never confuse movement with action. --Hemingway
I use a combined nagios + cacti solution with over a 1000 devices, around 10,000 nagios checks, and 50,000 cacti data points. It's all tied into our host allocation system via some scripting and templating, so requires very very little day-to-day management, and it looks like it would scale to at least your specified size with some more hardware thrown at it.
The biggest issue I have with scaling is that my devices are in a half a dozen different colos. Network latency hits me harder than anything else. Geographically distributed polling's pretty easily solved with nagios, less easily with cacti, but if you search on the forums there are some people who've done just that. If all your devices are in the same colo, then it's even easier.
Go with what you know, and make it do what you need it to do.
http://labs.omniti.com/trac/reconnoiter
This project has so much potential and has been developed from the ground up. If you are going to build a system from scratch may I suggest getting involved in the development of this wonderful project.
MRTG,
I know, I know...It's old but still works great.
I have almost 6000 devices monitored, running on an older HP 6000r server, 6 way, 4 gigs ram and 3 nic cards.
-js
very non invasive, monitors just about anything and graphs via open libraries. opensource and pretty easy to get started.
Steve Maher freeunixtraining.com
OpenNMS can easily handle a network that big. It is easy to use once its up and running.
http://www.opennms.org
We are using http://www.groundworkopensource.com/ for our monitoring. It is working pretty well, and we can use existing Nagios scripts with it.
Zabbix allows you to build some fairly powerful rulesets and chains of overrides using its web gui. It's not perfect, but it keeps improving and the attitude of the developers is friendly unlike some of the other projects.
I implemented The Hobbit Monitor where I work. Actually, its called Xymon now because of a copyright complaint (who knew?!) but I digress...
We monitor basic to complex information of about 5000 machines and a handful of NAS devices. It is a server/client setup, and highly customizable: an evolution of the Big Brother monitor from days of yonder. The histories can go back indefinitely, and all the configuration is done by flat-files: helpful if you like to roll-your-own automatic configuration tool.
It is pretty basic out of the box, but the way it is implemented makes it very easy to track whatever you want and write your own tests: from simple bash and perl scripts, to c programs with api hooks into your applications.
We didn't go with Nagios because it initial testing showed it was very chatty and the interface was unintuitive. I happen to like the easy 'smiley face good, frowning face bad' for taking a quick glance at our infrastructure.
I use SolarWinds Orion NPM and the Application Plugin. I have the ability to monitor all aspects of our network, servers, VMware environment, the works.
It is by no means cheap ($5000 or so to start) but I found hageling with the sales rep usually gets that price reduced.
Yes, it is a Windows based system, utilizing SNMP and WMI but i've never once had a problem with it and their user forums and tech support are awesome.
Just my 2 cents
Ganglia for performance monitoring (it's for clusters after all and has been shown to handle 5000+ machines with very little overhead) and Nagios for host/service down alerts.
Zabbix.
http://www.zabbix.com/
If it leaves something to be desired, please tell us what.
-fb Everything not expressly forbidden is now mandatory.
If all you want is Up/Down and a few core metrics (CPU, Memory, etc) or have nothing but 'nix servers and network devices to monitor then any of the OSS solutions can fill the need. If you work in a Windows world with loads of Microsoft apps, then it seems to me that Operations Manager 2007 is the best solution. I doubt any of their competitors can provide so much support for MS products and I would rather not build it all myself.
To be fair, I wouldn't say the Zope database (ZODB) is not a "solid foundation". It's one of the best parts of the Zope stack and, in 3 years of dozens of clients using it in Zenoss, Plone, and other apps, I've never had it corrupt or lose any data. It's a proper DB--ACID, MVCC, and all that--and you can even lop transactions off the storage to go back in time. Don't expect it to be a relational DB with the ad hoc query tools typical thereof; it's an object DB, with the aim of persisting graphs of Python objects transparently.
Now, if you aren't familiar with it, the ZODB can indeed seem opaque, but, just like any DB, there are tools to read and modify it. At the highest level, just stick "manage" after your Zenoss URL, e.g. http://example.com/zport/dmd/manage . That'll get you into the web-based Zope Management Interface (colloquially, "the ZMI"), where you can poke around at any object that someone's bothered to write a UI for. Deeper than that, you can connect to ZEO (a server that brokers access to the ZODB over a socket) and mess with the object graph using normal Python. When you're done, "import transaction; transaction.commit()". (The Zenoss developers are probably trying to scare you away from such digging around in fear that you'll violate their objects' invariants and leave them a real mess to solve.)
Now, I don't say that Zope isn't scary; it has over 10 years of scary stored up in it. But the ZODB is a cuddly, loving part.
Cheers!
I've used Spiceworks for multiple smaller sites and it works well...
Cacti was a pain to configure for every client(tried it first)...IMO
"Just Smile and Nod." --Huck
The question is, are you looking at developing and implementing the policy or the software itself?
If you are in charge of developing and implementing the policy I would look at the Compuware offering http://www.compuware.com/solutions/it_service_management.asp
I don't know if you are looking for just a reporting dashboard or the tools to gather server/network/application information too. I could go into more detail as to how the solution works, but it would be a waste of time not knowing what is in place.
Anyways take a look...it's impressive what can be done with the Compuware Vantage offering...according the Gartner/Forrerster Compuware is 2+ years ahead of the rest of the ITSM market when it comes to aligning bussiness with IT.
I would have to agree with the other poster that suggested SolarWinds Orion network monitor. You can monitor network swithes, each port, servers, apps on the server, other devices with SNMP strings, things that don't support that.... you can import multiple maps etc. At our site in Orion we have a US Map, state map, then campus maps for a few sites. then building maps then to server room. Custom views and alters. My login to orion shows different stuff then our telecom person's login. and no i don't work for them http://www.solarwinds.com/
We use up.time from http://www.uptimesoftware.com/ We monitor all aspects of our enterprise using it, including: Network devices, OS (Windows, Linux, NetWare), Applications (LDAP, DNS, MS Exchange, Oracle, etc), Performance, and hardware monitoring. We have much more than 5000 elements being monitored. We had a huge number of separate systems monitoring each flavor of component, including MOM, Nagios, NetMon, and many others. We wanted one pane of class to see the whole Infrastructure and be able to show the "service" availability. Obviously, each specialist system can monitor their own key element in some ways better, eg MOM can monitor things within Exchange better, but for a single Monitoring system this one won our evaluation process. Check it out.
Give Intermapper a try: http://www.intermapper.com/
Works great, simple to use. We end up relying on it more then our current tools deployed (Cacti, OV, NNM). It's also very reasonably prices.
Bias alert, I'm the Zenoss Community Manager.
Zenoss was written with the intention of making it easy to monitor and manage tens of thousands of network devices remotely. By using templates and device classes, once you have a single machine monitored the way you like, you can apply that to thousands of other devices, making individual changes as necessary. Zenoss handles network hardware, servers (Linux, Unix and Windows), databases, applications and just about anything else you need to monitor. There's a network map and a Google map mashup for mapping. No need to start from scratch, there's already an Open Source (GPLv2) Python-based solution with a large community and installers for Linux and OSX and a VMware image to get started (plus source for everything else). Lots of documentation and frequent releases, with commercial support available. If you're coming from Nagios or Cactii, you can reuse any custom plugins you've developed.
I vote Zabbix. Here's why.
1) Free but offers paid support if you need it
2) Can use agents, snmp or simple checks like ping
3) Agents can be extended with your own scripts and such. If a check isn't built in you can add it. For example, I added a very simple script for checking of MySQL replication had stopped or failed.
4) Templates, makes it easy to add a metric and create a trigger based on that metric to any host attached to that template
5) Triggers can be configured to minimize false positives (multiple dropped packets before sending an alert.
6) You can graph item, group of items or an aggregate value of items in a host group
7) Create your own maps
8) Create custom screens that group simple or complex graphs or whatever else you want onto a single page
There are some things to know about Zabbix though. You need to put some thought into items to get accurate values. Is the value you are getting from a device in bits or bytes for example. You can use custom multipliers to convert values into what you want to see.
Honestly, Zabbix is incredibly flexible and this flexibility also gives it a steep learning curve but once you get hosts entered and the templates situated the way you want it becomes very easy to add new hosts down the line. The biggest tip I can give is to make sure you spend a lot of time thinking out how to setup your templates. Zabbix includes a number of them and you'll want to customize them. One thing I found that wasn't a good idea is to make a template and then attach it to a template. It's much easier to join a host to multiple templates.
http://www.zabbix.com/
A far as the actual question goes, I think a patchwork of tools that you understand well and have proven themselves reliable is often a better choice than the all-singing all-dancing approach. The patchwork does take more time to roll out and configure, but if the tools are simple and easily managed, it is probably the better choice for large environments. However, at the 5000 device level, I'm not sure if you're at that break-even point. I've only personally deployed nagios, cacti, and similar tools on the small scale.
.
But more usefully, I'd recommend a tool that is *not* network monitoring to go along with it. Monitoring is great for seeing what is going on at the moment within a domain of events, but once you find out about a problem, how do you then dig into it? I really recommend feeding all your monitoring data and *other* IT data as well into a system that lets you investigate all of them. I think Splunk is pretty good way to do this. It's a search engine into all the time-series data in your environment, so you can learn things like what *else* broke at that time, who was logged in, and so on all pretty quickly. It's commercial, but reasonably priced, and can be used free at some data levels.
.
Caveat, they pay my bills, so I may be biased.
-josh
Zenoss for general device monitoring. Hyperic HQ for app monitoring. Splunk for scanning log files.
Nagios with Centreon. Centreon is a decent front end to Nagios, with commercial support if required.
This is a typical solution architecture project where I would do the following exercise;
-Assess your environment's needs (other than 5k+ devices and 2D maps). Integration with other reporting tools, notification (you needs SMS or just email?), access control (by group, user, etc). Scalability, clustering (did you forecast growth? consolidation?, etc...
-Budget? enterprise supported solution or OSS (other factors to consider for both options).
-Analyze what's out there in terms of options; Must have, nice to have
-Break down your list of candidates
-Internal knowledge with candidates, learning curves (project - installation, operations team training)
-Other key elements here.
There's a matrix that you need to do here in order to prepare for your solution.
Most people will suggest what they're comfortable with or what they've used.
You need to analyze your needs and what's our there. This process will likely take you more than the time allocated for the actual installation and environment preparation.
Good luck!
We use "The Dude" http://www.mikrotik.com/thedude.php where I work and it works great. It is a windows program but can be run under wine. You can can have multiple monitoring servers so you should be able to scale out. We currently monitor over 200 on a shared server with no issue.
OK, guys, I had into this for a while ago and had to choose what to do. Here is a list what I've tried (means really-really tried and even looked at source code) and my short opinion as a result. Disclaimer: was my own personal research and practice, so I might sound different from others. Any suggestions are welcome. :-) Also I want (yeah, I am picky):
So! Here it is:
I am sure they are not any leaders in monitoring technology. Also I even doubt they are leaders in monitoring in general. However, this worked for me and I wish Nagios all the best.
Personally I recommend go with OpenNMS. Not going to say it is excellent: it also has its gotchas. For example, don't even think installing it on a slow machines with low memory and/or on LVM. Also I would love to see it with other databases working... It is written in Java and it wants enough space in resources. However, once you give it to it, you will see all the best of it. Integr
for basic metrics and health, ganglia is very good. we use it exclusively to monitor a similar scale backtesting (compute) cluster, the footprint it makes is negligable. we've even modded it to do more high frequency measurements where appropriate in our production environment, it's proven very flexible and easy to mod for those purposes. very scalable where appropriate, and detailed where that is appropriate, with some work. we inject a lot of other application level metrics into it for workhorse servers. having one view and metrics wrapped up together is a nice feature.
we use nagios for catagorized system metrics ("what's the load on the netapps and who's beating on them now?") and i rely on hobbit (or whatever they call it now) for ready-for-business metrics. those eventually come down to "this stuff is supposed to be running on this host today, is everything necessary for that to happen in a good state?". it's invaluable for start of day troubleshooting, mixing together a lot of different greenlight systems into a single snapshot with state transition history.
basically, i don't really think you need a single holistic system to monitor everything you need, a lot of these things excel in their specific domain and the whole interrelated system may require many views into its health to monitor it effectively. i don't advocate looking for a one-size-fits-all system.
Hey, I don't know how I can get in touch with you since that our emails are protected on slashdot. I think I left a comment on your blog, but it was the best guess I had.
I can write a 100 pages of BS or I can just jump out and say it.
You want a solution that is easy and powerful with information to help IT track down issues with the plus that it is 100% scalable with a zero cost just to try it.
I recommend a group of programs made by SolarWinds.
I am the SolarWinds admin for the company I work for. I have monitoring setup across 3 data centers in the mid west, soon to be 4. Right now I am monitoring around 4000 items which covers everything in the buildings, if is has an IP address I am monitoring the device.
Intermapper is a solid monitoring / map system, nice selection of probes, very good value too
It only solves your real time needs, use Zenoss for the rest - trap management, cap planning etc
To this day I still can't under stand why the whole network monitoring domain has so many gaps and so low quality. ....
Have you ever tried a Cisco product? utter shit
I digress.....
Now a days you have to consider a monitor tool or tools that can do the following: 1. Event Correlation (don't page me or turn my entire dashboard red if I lose 2 severs in a 20 server load balanced pool) 2. Application Mapping, dashboards, and portals 3. User experience monitoring mapped to hardware ( eliminate finger pointing and shorten problem identification) 4. Ability to publish reports ( reports tailored to the person's skill level. The higher up the food chain the less they will understand very technical graphs) 5. Historical comparisons (We now have to justify clearly why we need the upgrade or the latest and greatest) 6. On large scale monitoring solution you have to manage your database, all this data can pile up quickly What tool you use is not as important as what you do with the information to quickly resolve issues and provide data on the health of your infrastructure.
I just went through the same exact thing you did.
The shop I work is is mostly MOM and the new version SCOM. For monitoring windows stuff it seems to do well. However, when we threw things like VMWare, Linux, Solaris and pretty much any other OS at it it had major disadvantages. Sure there were 3rd party plugins for VMWare (really expensive), some for Linux (quest, and talk about dirty and very unstable)....thats when I came in.
I first tried Cacti because I had some experience using it. Cacti is nice if you want pretty graphs and hosts monitoring. Service monitoring on the other hand is a total pain and not good enough for a NOC (war room or whatever you want to call it). Yes, we tried Thold and it worked ok, but Thold assumes to many things and does not allow the Noc to acknolowedge anything.
Skip foward to 4 months ago after being told we need something in place. I then looked at the other free solutions out there. Nagios was what I wanted but man setting up nagios with 300 hosts and around 1000 service checks is a beating. I then looked for nagios frontends and tried just about every one out there. Centreon came through with flying colors. Once you get used to it and understand how it works its great. Here are some things I have done so far with Nagios/Centreon:
Alerts are sent out via email with comments attached to them so the NOC will know how to respond
With ACL setup, I now have the screens in the noc grouped
I have a graph for just about every service
Our managers login to nagios and run monthly reports showing service downtime statistics (sort of a SLA if you will)
Nagios is very powerful, if you don't see a plugin for what you want its very easy to hack one together. If you can write a shell script to report useful information, you can then use NRPE to send that information to nagios at the interval you want (with graphs).
I even went as far as converting Cacti scripts over to work with Nagios.
Here are some of the interesting things we monitor:
I/O statistics - I wrote a plugin that runs iostat 2 times for 30 seconds I then output that data to a txt file and average the numbers and have nagios grab the data for very accurate io statistics (tps, r/sec, kBread/sec, srvtime ect).
Oracle - This is where cacti lacked. I found a nagios script called check_oracle_health, we use this to check if oracle is alive and it can also run custom sql. With the custom sql I run whatever our dba wants to watch
Tomcat - We connect using check_jmx to gather all sorts of stats
Swap - we watch pagein pageout and alert after so many minutes here
We have one application that run on a Linux machine that sometimes spins. I wrote a script that keeps count of cpu time. If cpu time is X I then restart the service automatically then alert nagios that this happens. Nagios gets the message then shortly checks to see if the service came back online and alerts if it does not.
VMWare - I monitor and have graphs for memory on ESX servers and there Virtual Machines. I can also gather stats at a cluster level.
Many other custom shell scripts, perl scripts to do various things along with all the freebies that you can find on the net.
A lot of people discount nagios and I think the majority of those that do get either frustrated or truly don't understand the power of it.
Another great thing about our setup is that we now have 2 pollers. One machine runs the Mysql db, Centreon and Nagios. This machine does about 1/4th of the polling. We have another machine that does 3/4's of the polling and reports back to the other.
Also don't be afraid to mix things in NRPE checks to get results that you want. I found one script that was almost perfect for us, but it reported to much data. So what I did was run this scripts output through sed to parse out information I did not want. It works like a charm with no noticeable overhead. Again, if you are somewhat versed in shell scripting you can use that knowledge to do all sorts of stuff. Nagios is pretty much what you want it to be if you get past that learning curve.
If you haven't evaluated BixData http://bixdata.com/ yet? You're missing out. No nag/reg required, free use for less than 30 hosts.
Does not include kitchen sink. Only the next generation advanced monitoring system that can handle phyisical and virtual as well as the hypervisors! VMware friendly.
On their science page http://www.bixdata.com/science, they say Bix is Borg. And they're not kidding.
"BixData is profoundly different. It took science fiction to provide the metaphor. Bix is Borg - 'an inter-connected collective' (self-organizing p2p) that 'assimilates' new life forms (cross-platform virtual machine), functions with a single hive-mind (n-cube datastore) and adapts through self-learning (cybernetic feedback loop) - all in pursuit of perfection. Resistance is Futile."
Just... Wow.
-- Robi
I use it both at home and on my previous job at a big datacenter with several hundreds of monitored devices. Worked very well and the map is superb. I haven't seen anything comparable. Espacially the realtime traffic status of network links (witch fade to red when overloaded) is a grad diagnostic tool.
If you are in a heterogeneous environment, this is a non-starter.
Great job security and resume enhancement, though!
Is the ability to monitor windows perfmon stats, even on tools that have been ported over to windows (looking at you, cacti). Seriously, most of the more useful sql server stats are reported only through perfmon, and snmp won't capture that data. Sure, sql server tracks that internally, but if you have a lot of servers, then you need external monitoring. There's such a wealth of information available through perfmon, isn't there any OSS tool that will monitor that stuff?
Big Brother (or Sister) which uses push agents so you are not generating vast amount of SNMP polls and you get instant feedback on a stupid simple dashboard.
http://www.bb4.org/
Project Observer is super easy to set up for SNMP and can auto-discover Cisco gear (with CDP). A good, simple SNMP monitor but it has serious scaling limitations.
http://www.observernms.org/
Nagios for hard core up/down monitoring with good flap detection and Cacti for performance monitoring.
OCS Inventory for push software distribution and inventory control.
Or you could drop some serious cash and just get Unicenter TNG and go bald from ripping your hair out.
Seriously, though, try a bunch of things and see what actually works for your team.
[RIAA] says its concern is artists. That's true, in just the sense that a cattle rancher is concerned about its cattle.
Fast, scalable and integrates the business view in the tool, as well as a real GIS (you can use it or not), SLA management and BI and datamining views. Everything related to monitoring in made in C++ using the multiplatform near real-time framework ACE, so it's really fast.
The Web console is based on TomCat (J2EE) and you can manage, deploy, configure or update agents from there.
You can develop new events and new agents to monitor "whatever" you want: stock shares, temperatures, web transactions,... it's up to you. And Osmius is real Open: There's no open core, nor closed features, and you can access the documentation, the analysis info, datamodel staff, and (of course) the code.
We are now (I'm in the development team) doing the final tests and dealing with the latest bugs and also testing scalability and the behaviour under stress of receiving millions of events every day, and thousands per seconds. It's working properly.
The D day is July the 30th. Every comment and suggestion, and even hard criticism, is very welcome.
We want Osmius to be:
Let's see if we made it.
Well, I suppose it depends.
How many large scales are you planning on monitoring?
I think you have already gathered from the large amount of responses that the problem has been "over solved" - many options to choose from. We provide a rip-n-replace service for HP Openview users (banks and trading exchanges), which, including any coding required and 24/7 support comes out at about 20% of those annual costs in year 1 and below 10% in subsequent years, but there's no point in telling you what Open Source product we use - you need to do your homework so you arrive at an answer that you and your boss understand yourself.
You will probably find a number of answers to your criteria - TRY THEM. Give the ones that seem viable in terms of support, community, code quality and your own ability to make it work for your company a good try - most you can even do in parallel. Only after a live test can you decide what you're going with, because you will be investing time in tuning it for your own needs - this is not the time you want to waste. A good preparation is worth 80% of the work for monitoring, or you will spend time monitoring the monitoring system instead which is a waste of your time.
Good luck :-)
Insert
Thanks for the info.
Love many, trust a few, do harm to none.
Both Yacoob and Afabbro have some great lists above (especially Afabbros list!). A combination of these features would be an ultimate system.
:))
I work for a company called iQuate - I have been (sometimes literally) developing a monitoring system for about 7 years. We do many of the things mentioned in the 2 lists, but not all (I wish!). We have a product called iQRMS which integrates several functions, the largest of which is monitoring.
- It is agentless - it uses about 30 different protocols (including SNMP obviously!) to connect to remote machines, so it can be deployed very quickly and gives a pretty "true" picture of client connectivity (which sometimes an agent based approach will not).
- It is horizontally scalable (you can have many scanning services on many computers and they will load balance between them).
- It has failover built in - when 1 or more of the scanning services die, the others redistribute the load.
- It has intelligent aggregation of data, recording max, min and average values for any monitor over time - for up to 6 years - in such a way that it doesn't just eat disk and kill performance (that one took a while to crack...)
- It has pretty graphs and in-depts reports on events
- It supports complex (or simple!) escalation rules to control who gets told about what, when and how often when events happen
- It integrates with a helpdesk (it's own or others)
- It allows you to create templates of monitors using different protocols to get a wider picture of an issue
- It is easy to understand and designed with 24x7 operations in mind (hence all that failover/scalability)
- It doesn't cost the earth
It also doesn't do some of (1 of) the things Timothy mentions at the start of the post (gratz on the new job btw!) - specifically it doesn't create a 2D map of the environment, although there are some plans to implement that in future. It treats and represents devices in the network as groups of hosts - it doesn't display them in relation to physical layout...
Maybe it's worth having a look at it Tim, I can certainly vouch for the support being excellent (but like I say above - I'm biased
JK
Both Yacoob and Afabbro have some great lists above (especially Afabbros list!). A combination of these features would be an ultimate system.
:))
I work for a company called iQuate - I amd the CTO and have been (sometimes literally) developing a monitoring system for about 7 years. We do many of the things mentioned in the 2 lists, but not all (I wish!). We have a product called iQRMS which integrates several functions, the largest of which is monitoring.
- It is agentless - it uses about 30 different protocols (including SNMP obviously!) to connect to remote machines, so it can be deployed very quickly and gives a pretty "true" picture of client connectivity (which sometimes an agent based approach will not).
- It is horizontally scalable (you can have many scanning services on many computers and they will load balance between them).
- It has failover built in - when 1 or more of the scanning services die, the others redistribute the load.
- It has intelligent aggregation of data, recording max, min and average values for any monitor over time - for up to 6 years - in such a way that it doesn't just eat disk and kill performance (that one took a while to crack...)
- It has pretty graphs and in-depts reports on events
- It supports complex (or simple!) escalation rules to control who gets told about what, when and how often when events happen
- It integrates with a helpdesk (it's own or others)
- It allows you to create templates of monitors using different protocols to get a wider picture of an issue
- It is easy to understand and designed with 24x7 operations in mind (hence all that failover/scalability)
- It doesn't cost the earth
It also doesn't do some of (1 of) the things Timothy mentions at the start of the post (gratz on the new job btw!) - specifically it doesn't create a 2D map of the environment, although there are some plans to implement that in future. It treats and represents devices in the network as groups of hosts - it doesn't display them in relation to physical layout...
Maybe it's worth having a look at it Tim, I can certainly vouch for the support being excellent (but like I say above - I'm biased
JK
Do you know Pandora FMS?. Pandora Flexible Monitoring System is a general purpose monitoring tool. It was born in 2002 at the IT department of a international finance corporation. The ultimate goal of Pandora FMS is being an adaptable platform for any organization, able to collect events of any type, generate alarms through a metric system and to represent obtained events in graphs, reports or maps. Pandora FMS can detect a network interface down, a defacement in your website, a memory leak in one of your servers applications, a delay in your website when the customer pays, or the movement of any value of the NASDAQ new technology market. Pandora FMS can show you the state of your servers, systems, applications, communications, or the sale level of your commercial team. Pandora FMS is extremely modular and decentralized. The most important component, and where everything is stored is the Database (right now only MySQL is supported). Every single component of Pandora FMS can be replicated and work under a pure HA system (Active/Passive) or under a clusterized system (Active/Active with balanced load). Pandora FMS can gather information locally with agents software or hardware: - Pandora FMS has specific agent software that runs on any operating system, GNU/Linux, AIX, Solaris, HP-UX, BSD/IPSO, and Windows 2000, XP and 2003, gathering data and sending it to a Data server. - Pandora FMS has a specific hardware agent, being able to connect any sensor to this devices. Using it, it is possible to monitor temperatures, lightness, movement, smoke, ...
Pandora FMS can also gather information remotely, without installing software or hardware agents:
- With the Network Server Pandora FMS can monitor any kind of service or port via TCP query, any devices via SNMP, and any communication latency or state via ICMP.
- With the Plugin Server Pandora FMS can monitor any kind of system with complex code. It is compatible with Nagios Plugins.
- With the WMI Server Pandora FMS can monitor any Windows via WMI.
- With the Web Server Pandora FMS can monitor web applications via complex checks. There are two kind of webchecks: A check for response time and check for availability. Of course, webchecks are not just making a simple http request to say if works or not, webchecks can make fully complex web operations, like perform logins, choose a parameter from a menu, enter text into a form, expect a specific response in each step and make sure that all programmed steps are done correctly before saying âoeweb application response is OKâ.
- With the Prediction Server Pandora FMS can detect trends. It implements in an statistic way a data forecast based on past data (to almost 30 days in four temporary references).
- With the SNMP Console Pandora can monitor any device via SNMP traps.
You can visit pandorafms.com.
Bye
If you have knowledge in Nagios, do not reinvent the wheel and use it together with NagViz and Business Process View plugins. You will get nice maps and easy corelation from technology point of view to the Business one.
I was a long time Nagios user but the manual config changes and management of it was just getting too much, I've recently switched to running a clustered OpsView setup monitoring 2 geographically separated sites and around 1000 devices. It "just works". It is easy to configure and manage, the data warehousing/searching/reporting feature is great, graphing is excellent, dashboard and nagvis elements let you present data nicely and there's even a scheduled reporting tool to email the management a PDF full of pretty graphs every month. Because it's built on Nagios there's a plugin for monitoring just about anything you can think of and it's free! I don't work for OpsView, I'm just a fan!
Well, firstly, I'd like it to work. That is all.
You openly admit that you monitor thousands of people's PC's without their consent and probably without their knowledge? I would be ashamed. Collected any good blackmail material yet?
I piss off bigots.
Most monitoring systems only check each server once every 5 minutes, giving an average of 2.5 minutes between error and alert; with a customer, you'll be out of bed at 3 in the morning in a matter of seconds :-)
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
Don't know why. No doubt someone will be able to explain why it's essential.
I've found Nagios and NagVis a solid solution. NagVis is a plugin/addon for Nagios which allows you to create a Heads-Up display with status information on your own network diagram. It has an interactive map, which is 100% customizable. When kept in a browser window, it will play a sound during an event and flash the icon for a host indicating the problem.
PNP adds graphing of performance data to Nagios. It allows you to click through the nagios interface directly to the graphs for a given host or process. It will graph anything that has performance data output.
Finally, Cacti is a great solution for things which you may not roll into your Nagios insallation. We use it for monitoring network bandwidth utilization, mostly.
[ this a general comment for anyone interested in Zabbix ]
I'm currently working on a project in a Hospital. We chose Zabbix (1.6.4) and myself and a colleague have set it up. It's currently monitoring 169 hosts (mixture of Windows, Tru64 Unix, Linux and UPSes) with 7108 items monitored and graphed and 1,704 trigger alarms. And this is really small potatoes compared to what Zabbix can handle. We've not had any issues with the Zabbix server processes. It runs just fine.
What I really like the most is its graphing capabilities: the ability to zoom in on a section of a graph by just dragging and selecting the time interval you want. This also works with "Favourite Screens" containing multiple graphs, where selecting a time window on one graph automatically zooms the rest of the graphs on that screen at the same time.
There's an excellent FireFox plugin too that gives you a summary icon at the bottom of the browser, a single bar to display the most recent event, and clicking on the icon slides a tabbed pop-up frame into view that contains all the information you'd expect to see on the Dashboard.
Like any monitoring tool of this size though, getting all the data in for the systems you want monitor is time consuming. My advise would be to concentrate first on getting a few systems just how you want them, then work out whether it's faster for you to clone them and mod system specific information or export the XML for those systems, use those as templates for other systems/hosts of the same time and work with XML import.
Zabbix gets a big thumbs up from me.
These softwares are very powerfull in network monitoring. They can monitoring large networks, with netflow traffic analisys (TRAFip).
The SLAview system is used for monitoring and managing the performance of key elements, critical to the good functioning of telecommunication networks.
There is a very interesting 2D map, with weather map included.
Site: www.telcomanager.com
enjoy
Here, I develop one solution that integrate Nagios and Proprietary Solution -> JAVA EE (EJB and Message Server) -> Adobe Flex (Client Interface). The Nagios send information to JAVA (Server) and JAVA send realtime information to Flex Clients like a broadcast chat.
I work in the 3rd bigger bank from Brazil, and we have a lot work to do. ThatÂs Great!
Thanks,
_______________________________
DANIEL GOMES NEGRI
(Analista de Sistemas)
Consultor em Arquitetura RIA - Rich Internet Application
Certified ScrumMaster
ï daniel.negri@hotmail.com
ï (62) 9218-7315 / (62) 8135-8339
GECAN - GerÃncia de Canais de Atendimento
BANCOOB - Banco Cooperativo do Brasil S/A
We used OpManager in production for over a year. It has terrible Linux support. None of their built-in plugins worked properly for monitoring even basic parameters like disk space, free memory, CPU usage, etc. When we pointed this out to their support people, they said we should build our own plugins with SNMP OIDs. Um....no. Not for the amount of money we paid for that steaming POS. We finally kicked OpManager to the curb about a month ago, and have our entire environment, Windows and Linux servers being monitored with Nagios. Nagios scales well, we are currently watching several hundred hosts and about 3500 services.
OpenNMS is also a good tool, its ability to map servers back to switch ports is extremely handy.
Unfortunately (or not), Windoze based. My experience with it started out unstable and feature poor at version 4, but it kept the relatively inexpensive (core, support, and add-on) price tags, and features kept getting better, and stability continues to improve at version 7.1. Remote windows and java consoles, remote pollers, SNMPv3, easy custom MIB compiles, functional dependencies, device grouping, custom alarms, restricted console views, packaged third party paging and email, custom tool integration, easy maps, acceptable (to me) TCP service monitoring and third party script support. Reporting is also integrated, or use the up-featured SQL add-on. I'm using it for just shy of a couple thousand devices on a single modest server. It's been able to accommodate every NMS feature I need, and a great many wants. My only real gripes are: console authentication still doesn't have a RADIUS, LDAP, or AD hook, and I'd like a Linux port for the backend. Other than that, it's shamefully simple to get new staff up and running, and it requires very little care and feeding. Good luck with your search.
HypericHQ would be good
PandoraFMS is another option.
ZenOSS and Zabbix are popular too
The biggest problems we've seen from a monitoring perspective is that most systems really do have a hard time scaling to large levels and being usable. [A common trick (and one we employ) is to have a multi-tier monitoring system in place, where one monitoring stack monitors the monitoring stack that is actually watching the service/hosts.]
Once one gets past that hurdle, the tricky part is dealing with the "it is OK if X% of my machines are down". Most monitoring systems that I've dealt with are based around the view that they are monitoring a single host/single service and not a collection of hosts where it is OK if chunks disappear. For those types of problems, one still ends up writing a lot of custom smarts it seems.
I work for a division of a very large corp and we just finished a rolling out ORACLE GRID control, it blew away all other tools we tested. It has agents for everything SqlServer, Cisco, NetApp, EMC, all *nix... everything. Highly recommend it
The gun is good - Zardoz
The solution is real simple. If you can program in anything then Hobbit/Xymon with Devmon is your only choice.
Create your own Weather Map for 2D, you never need a full 2D map of 5000 hosts... Less is more.
1. Free
2. Fully customizable
3. Easy administration
4. Offers clients for all the major OS (And quite a few minor ones)
5. Large support base (Users with high technical level)
6. Nice author (Replies to comments and considers all ideas)
7. You can write a test for anything you can think of and easily add it into hobbit
8. Offers client/server montoring, remote monitoring, script monitoring, snmp monitoring(devmon) or scripts
The possibilities with Hobbit are endless
Personally I use Hobbit to monitor over 2400 devices, including Cisco hardware, AIX, Windows Servers, VMware Clusters, Exchange, Sharepoint etc.etc.etc.etc.
I've never encountered a system I could not monitor with Hobbit (Or scripts that send their results into hobbit).
For the price, I am a big fan of SolarWinds Network Performance Monitor (NPM)and ipMonitor. Together, these give me the ability to track and monitor as many devices as I want. They both have network discovery, and can monitor network devices, servers, workstations, and applications. I have not tried the Application Monitor add-on for NPM as a replacement to ipMonitor, but it looks like it would work very well. I have written several c# scripts to augment ipMonitor for the custom applications I need to monitor. The only downside I can see for you is that it these are Windows based productions, and NPM requires Microsoft SQL 2005. I love the map capabilities in NPM and the graphs it makes. I am able to alert on any OID I collect data on, which is a plus. Also, NPM has thousands of MIBs already installed, which makes finding an OID much easier. Best of all, NPM supports OID tables, which makes my monitoring very dynamic. I have, for example, created an alert on disk partitions getting more then 95% full. I do not have to worry about making an monitor for each possible partition, or even worry about how many partitions are on a server. I just monitor the OID table. As long as I have set SNMP properly (of course) I see all the partitions I care about. ipMonitor I use mostly for application monitoring. In this respect, it is very nice, since I can execute custom scripts. For your scenario, I would seriously look into NPM. This is a very easy product to learn, very powerful, and can be fairly cheap to implement.
The answer to the tool lies in what you are attempting to monitor and how the data is to be used to address the problem(s) that you are trying to solve....
I dont see any requirements here - probably you have them, but not listed...
I dont see any expectations of what is to be delivered - you probably have them but not listed either....
I would suggest that having the requirements and expectations will be help you create the solution - and make the best tool rise to the top....
I write that sort of stuff in awk. Hire me.
Have you looked at Kaseya?
www.kaseya.com
In the orchestra, you need different instruments, musicians and a Maestro who knows the whole partition and can render it. Nagios is not playing like Cacti, MOM, Quest, etc...Even if they play in the same sandbox....I mean here that in your environment, you may already have some monitoring solutions specific to each System Management disciplines: Database, Servers, Network devices,....They have all their own user, admin & configuration consoles and may be also their own agents. You need an integration of all these Monitoring solutions ( at least the most important, valuable and easy ones). If you drop all the monitoring solutions and go for a single one from scratch, it will sound like a ring tone in the hall of an airport at 10:00AM...You need a chief, a manager...a Maestro...an Event Manager (centralized or distributed if needed). The role of this Event Manager is to get the most valuable monitoring information from these monitoring tools and to consolidate all these alerts using a single syntax & semantics ....Consolidation: enrichment, correlation, filtering, reaction... of the alerts reported by the monitoring tools to an Event Management Solution sounds like repeating again and again with the orchestra for the D day of the concert. This Event Management Solution will be the Maestro. All the musician (sysadmins) will recognize their partition and their instrument (their monitoring tool) when the music will play => Investigate an Enterprise Event Management Solution OVER your current monitoring instruments. Look for the most flexible (easy to integrate..easy scripting), the one able to speak through multiple and SIMPLE protocols. The one that will serve you on a gold plate the root cause of all your troubles when the poultry will make some noise. There are some of them on the market (Open world included).
Music Maestro !
We have similar goals with our project Clearsite.sourceforge.net. We've learned our lessons and think we can begin taking on the likes of SolarWinds, OSSIM, ZenOss, SpiceWorks etc... We made the mistake of being to geared toward one vendor(cisco) but no longer. We're making the software work for us, were not working with the software. Crating a Snort interface that highlights the portion of the packet that trips the content rule, being able to note FP's, highlight the portion that's a FP in the packet, and it's added to the rule once you click submit. Some user-agent rule goes off, but it's your own app, highlight the user-agent your app uses, click submit and content:!"user-agent: xyz"; gets added to a display filter and or the actual sig itself. A snort rule is triggered for Bittorrent being used, a cron job connects via wmi, snmp or ssh to a host, runs a netstat -abn effectively and figures out the process and location of the executable that triggered the rule, or the lack of being able to get such a result back might further point to a FP or a machine not under your control. If no contact, check the mac address db to see if it's one of yours, if not, snmp set fa0/22 disable. Proactive. Naturally there are more checks and balances in there, but that's where were heading with just the snort portion. Again making the software work for us. As always we'll use our very popular ajax search for everything we can. http://clearsite.blogspot.com/search?updated-min=2007-01-01T00%3A00%3A00-08%3A00&updated-max=2008-01-01T00%3A00%3A00-08%3A00&max-results=3 -rich (google: xinn.org contact)
You bring up an excellent point: ZODB doesn't do any referential or data-type integrity checking; it's pretty much just a dumb (though rather concurrent and durable) graph store. Thus, ZODB-using apps have to take care of data integrity themselves or else interpose another layer (which you'd want to do in a "shared" situation like you mention).
I guess that's the tradeoff ZODB makes: really fast and agile development (no schemas to maintain, etc.) in exchange for no particular constraint enforcement. In practice, the latter is mitigated (and lots of painful debugging saved) through use of constraint-enforcement frameworks like Archetypes, but that still makes me queasy in a multi-app situation, as you'd have to make sure everybody uses the framework.
Personally, I'm both a ZODB and a Postgres wonk. What I'd love to see is the best of both worlds: a language-agnostic graph DB with internal constraint enforcement and, as my pony, a declarative ad hoc querying language. :-)
We are using a combination of Cacti and mon to monitor about 200 devices, both network gear and PC servers. Cacti is used to graph performance data(bandwidth, cpu, mem, temp) and maps for the visually inclined, while mon is used to do the actual service monitoring and alerting.
I won't comment on Cacti, since it has been mentioned here already, though iI will say that you CAN change the default behavior of "sample averaging" by increasing the size of the RRD database. There are discussions on the Cacti forum/wiki that cover this topic.
Mon on the other hand, I didn't see mentioned at all, so here's my blurb on that. The core of mon is a scheduler written in perl, which handles running monitor tests(also perl or any script/program that can exit with a 1/0) and then alerting(also perl, or other languages, and can do more than just sending mail or paging) when necessary, based on the configuration for that service. Like most open source projects, it is extremely flexible, if you have the initial time investment to set up your tests and dependencies correctly, but once this is done, the tests/alerts can be reused, or further modified. There are quite a few monitor tests and alert scripts already included, along with some handy tools for interaction through a web browser(via moncgi), generating dependency trees, generating reports, and more. Theres also a perl module, Mon::Client, that provides an API for interacting with the mon scheduler. The downside, besides configuring it with a text file(m4 can be helpful here), is there hasn't been any activity since 2007(according to the CVS repo on sourceforge).
Probably not the solution for an extremely large number of hosts, though resource-wise, it could handle it, but maybe someone else might be able to benefit from it. If you need very specific tests(number of BGP routes, verifying NH on routes, customer redundancy) and smart alert logic, it's worth looking at.
I want DRDs.....
I would suggest you give GroundWork a go. It an amalgamation of all the best open source monitoring tools previously mentioned in these comments such as Nagios and Cacti but they are fully integrated into one interface and reduces complexity.
GroundWork Open Source uniquely combines the most mature and successful open source projects available today into a single package. These amalgamated projects have been downloaded over the last decade for more than 4 million times and have a strong codebase and a strong community behind them.
Combining these projects into a single package that is commercially supported gives you a simplified deployment experience, a single console for managing and monitoring, and a comprehensive view of your IT operations efficiency.
Other GroundWork Monitor advantages include open APIs and an open event manager so the information collected by GroundWork Monitor can be biâ"directionally shared with your other ITSM systems such as asset tracker, ticketing system or a CMDB.
But the best part about GroundWork Monitor is the low cost for the enterprise IT management system and its fast return on investment and value.
GroundWork amalgamates and supports established, mature projects including:
* Nagios® â" for event handling and notification
* SNMP and SNMPâ"TT â" for network management protocol
* RRDtool â" for underlying data collection and management
* BIRT â" for adâ"hoc reporting
* Ganglia â" for grid and cluster monitoring
* Cacti â" for graphing and trending
Each of these projects and more are included with each GroundWork Monitor edition tier. GroundWork Monitor has three editions: Enterprise, Professional and Community Edition. All editions are based off of the same codebase so trade-up is easy.
It's based on Debian, you are able to run scripts as an action, can page, email, has a Postgresql backend (Very easy to backup). It can do SNMP v1-3, Syslog, Traps, Portmonitoring graphs, monitor CIFS/NFS volumes, Linux/Windows services, etc
http://www.netmon.ca
I recently went through the process of evaluating new monitoring software for my company as well and which product you go with really depends on what your needs are. I have also worked with may of the products that you noted and they all have both their strengths. If you aren't familiar with it OpenNMS really is a great product, I have used it for years, have attended the training offered by the OpenNMS Group, and also had the commercial support for it which was fantastic. Tarus and the rest of the guys with the OpenNMS group are all fantastic to work with and the community support is awesome as well. If you are used to working with OpenSource products and are familiar with RRD (which you likely are from using Cacti) then OpenNMS is absolutely worth a try. If you have some budget to work with and you want a more commercial solution that is MS Windows based I would suggest you take a look at SolarWinds Orion. Orion has a great monitoring solution for a good price. SolarWinds also offers several modules for Orion for things such as high level application monitoring, VoIP monitoring, network device configuration management, IP address space management, etc. Like I said, it all really comes down to your needs and where your personal comforts are.
Totally separate the data collection from the user interface. Keep both of those totally separate from the system that selects and delivers the alerts. Make sure the system as a whole won't make the problem worse (e.g. if you lose a major piece of infrastructure, will it send you 300 alerts?)
Orchestra and the Maestro is one the of the best analogies I've heard to describe the role of the Event Management system. RapidInsight aspires to be the Maestro over all other management to consolidate IT management information from different tools used to monitor systems, network and applications but also fault, performance, config/change, tickets, etc.
have a look at MonALISA
http://monalisa.cern.ch