Slashdot Mirror
Network Monitoring and Alerting?
SpamMonkey asks: "At work I am trying to implement a central monitoring and alerting service. We have in excess of 250 Windows servers, approx 15 AIX servers and another 30 Linux servers (mainly SLES/Suse). My investigation into systems that will allow us to monitor critical areas on each of these systems has so far led me to a clustered Linux server running Nagios with passive and active checks. What I'm curious about though is how Slashdot readers are carrying out their own jobs and how they can comfortably sit back, without having to repeatedly check that various systems are still operational and how to cut down their own response times when something goes wrong."
Why do you need a cluster?
Used at my Fortune 500 company to monitor thousands of Win/Linux/Solaris. http://www.bb4.org/
No sitting back.
Security/monitoring is a process not a product. When you finish the checklist you start over.
Also, i would recommend trying to cut back the windows server's somehow, maybe mention the right words when the licenses expire or its that time of the year to upgrade to a "free" solution?
It takes a man to suffer ignorance and smile
Be yourself no matter what they say
All in all I'm monitoring about 200 different processes across our network as well as running MRTG on the same box. Never felt once I needed to cluster.
"War doesn't determine who's right, just who's left"
Steven Wright
is still someone calling you up at 3 am screaming at you that the network is down and yelling at you to get your ass in there asap. Works every time!
Monstar L
You blame the new hire. If there is more than one new hire you blame the one that spends his waking life playing everquest or .
As for preventitive mesures, you put the new hire in charge of the task and sit back and relax.
1st rule of managment is misdirection.
"A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - DNA
We are in the middle of a large scale Linux project .. replacing 900 SCO Unix servers with Linux. We are wondering the same thing .. what monitoring tools can be put in to watch these servers?
... the IBM equivalant (A Tivoli product I think?). No need to reinvent the wheel.
... how do you balance your need to know everything about every box all the time (well, the business and management want to know at least) vs the tiny amount of data you can push without interfering with mission critical applications running over that same link?
Our big hitch comes from the fact that we only have a satellite connection to each of the remote sites, so we can't do real time monitoring, so things like HP Openview NNM are out of the question - they use too much bandwidth.
Our solution (And the reason I'm working late right now) is to build a custom suite of tools that does batch reporting every night by polling logs and custom programs, then sends it back in a handy xml file. We take that file, dump it into a large informix database, and then we can do whatever we want to create reports.
It's a little more work than just installing a package, but we're getting EXACTLY what we want out of the product. It works with our very unique communications and configuration, and it's modular so I can add whatever monitoring/checks I want by writing a new ksh script. All the output is standardized and all the parsing is done at the office by a very clever xml parser one of the db guys wrote.
I think for whaty ou're looking for, theres things like Big Brother, MRTG, HP Openvie umm
But I'd love some feedback for people who are working in a bandwidth sparse shop like me
We emerge from our mother's womb an unformatted diskette; our culture formats us. - Douglas Coupland
So you're concerned about letting things go on auto-pilot and missing an alarm . . .
:). Either way, nagios has ASSLOADS of event notification options.
Why not slap a modem into the head nagios box and have it page you when things fail. Don't worry about having to wear a beeper - you can page most cellphones via your carrier's SMS gateway (still dial-capable).
Too much hassle? How about AIM? YIM? Jabber? Email?
If you're TRULY the teeth jittering, chain smoking NOC type, buy some x10 crap and build a physical network alarm interface like I did
Here's a couple of the monitoring solutions:
Opennms
Mon
Big Brother
For system information polling I'd go with:
Cacti hands down this is the best polling system out there and it's simple to setup and run.
Check out Mon and Mon.cgi
Depending on what you are doing, but for that size of deployment, give Indicative a call/email.
I do testing for them, and we really do have a very cool system going. We monitor the systems, the network, and the applications. We have pre-defined tests for almost anything you could think of from simple http tests to fancier cisco router tests, to perfmon integration. We monitor Weblogic and Websphere, and soon are going to be able to monitor other app servers as well.
So ya, give us a call and inquire about how we can set you up. This is by far the easiest product to use in the monitoring field.
I use IPSentry to monitor our servers. It's cheap, easy to use, and can check just about anything you'd want to. You can even write your own monitoring and alerting plugins if that's your thing.
At work, we use a program called JFFNMS (Just For Fun Network Monitoring System). It's come a long way from it's inception, and has to ability to run on *nix and Windows.
We tried Cacti and just didn't like it. I've looked at Nagios, but not in detail.
Find JFFNMS at www.jffnms.org
Alcohol & calculus don't mix. Never drink & derive.
Profiler Rx
.sigs are for post^Hers.
Rule #1: Do not add to the problem.
This is an entire category of Operations Management and can encompass everything. Don't take it lightly and don't be afraid to start small. The first thing you need to do is categorize what you want to monitor into individual sections and work on the easiest stuff first. By the time you work up to the tough stuff, you'll have an idea of whats available, what your capabilities are and hopefully the easy stuff can be quickly rewritten/integrated into a netter solution. Don't miss the critical stuff in a morass of junk alerts. Sample consideration (everything that moves in the data center):
Hardware:
---------
1) Server
2) Storage
3) Network
4) Power
5) Environment
6) Security
Software:
---------
1) OS
2) Applications
3) Security
Events:
-------
1) Failures
2) Alerts
3) Misfires
4) Security
Triggers:
---------
1) Notification/False positives
2) Action plans/Event handling
3) Documentation, Documentation, Doumentation
4) Reporting aka analysis and cya
5) Security
Once you're done building it, start over. The last tier is the most visible e.g. delegating a raid rebuild page to the opcenter flunky without proper documentation is a Career Limiting Move (CLM), building the best monitoring system is a fucking waste unless you pay attention to it. The most apt cliches for monitor normalisation are all military: Warrooms, Bridges, Weapons Hot, Communications channels etc. View everything as SNAFU and work from there.
Rule #1: Do not add to the problem.
It's set up to do network topology and fault isolation based on the "root" location you're at. You can use multiple monitoring hosts and set the systems to monitor each other and if one of them fails the other can take over..
It may suck for you, since the author is a fool, but it may work perfect for your application too, i'll likely never know :)
I haven't heard anyone mention NetIQ's Appmanager.
http://www.netiq.com/products/am/default.asp
I think Microsoft sold a watered down version of it as MOM which seems to never have taken off.
Anyway, we looked at all the monitoring packages for keeping track of over a hundred Windows servers and it was the best for us due to flexibility, mid-cost, and capabilities to be centralized but monitor remote sites locally.
You can even write your own jobs to do what you want with almost no limits.
It has evolved and now monitors Unix based systems as well.
Hope this helps.
Kenneth
Get a cheap Mac and put Whistle Blower on it. There's a version for OS X, but you can still get a version that runs on OS 9 (I don't know if the older version is missing any features present in the OS X version).
You can configure it to do almost anything to alert you if a server goes down or becomes non-responsive.
There's quite an impressive list of clients using this software.
Check Zabbix if you're looking for a solution which is free, supports all platforms, and easy to deploy. Look at screenshots.
My company uses it for several months already in a mixed Windows/Unix environment (~180 servers) with great success. We use nearly all features (notifications, graphs, network maps, SLA monitoring, cool screens) Zabbix provides. Very useful stuff indeed. We tried Nagios before, but found it complex and hard to maintain. Besides performance of Nagios was disappointing (not enough tuning?).
Rog
argus is not bad, is pretty simple to set up, and scales reasonably well.
it's also pretty flexible so you can plug it into just about any paging system and monitor just about any service you can imagine.
it handles heirarchies so you dont get 300 pages at once, etc. and it has a simple, fast, clean web interface which isn't bloated with gigabytes of shiny widgets and is even perfectly usable via lynx.
it has a few rough edges but the overall ease of use and simplicity make up for it.
At my workplace, we're using HP-OVO. We have 300+ HP-UX, 200+ Windows and 50-odd Linux servers. It is highly customizable and allows you to set up what all to monitor, the time slots within which to ignore them, etc etc etc. You basically have to install SPIs (Smart Plug-Ins) for each of the parts you wanna monitor (OS, DB, hardware, custom applications...)
And yes, it can also be configured to alert you via email/SMS.
-- rxmx --
Love all, Trust few, Follow one.
Our big hitch comes from the fact that we only have a satellite connection to each of the remote sites, so we can't do real time monitoring, so things like HP Openview NNM are out of the question - they use too much bandwidth.
You might be interested in Intermapper for its Remote component. You can run a monitoring system at each location yet administer them centrally. Your remote datastream will basically be the set of events that's interesting to the Human In Charge.
It's commercial software, so you have to weigh the cost of the software vs. getting it right yourself.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Look at Klir Technologies- www.klir.com Klir has developed an enterprise wide tool and dashboard technology that increases the visibility into the performance of all infrastructure (data, wireless, VPN, servers, applications, routers, etc) without the use of agents, hardware, or software. They also tie performance to cost so executives and directors can quantify ROI on existing infrastructure, and get a better understanding of how infrastructure is being used in a business context.
I think you've highlighted an important notion when you talk about creating a centralized alerting system that will alert you to problems but will also help cut down your response time when there is a problem.
The key here is to move beyond just instrumentation, and add a layer of service mapping and correlation. The challenge is to map the data and events you will get from Nagios (or other monitoring tool/agent) to the services that the systems are provisioning and your service objectives (performance/availability). The goal is to correlate and add context to the alarms so that you can be reliably alerted on real problems, prioritize your response, and efficiently perform root cause analysis. Otherwise you are likely to find yourself chasing redundant problem indications and false alarms.
A number of the larger systems management vendors (HP, BMC, etc.) have begun selling this approach. I'm not a big fan of most of their solutions though because they often require multiple, poorly integrated products and a lot of consulting dollars to implement. I recommend checking out Managed Objects (http://www.managedobjects.com). The product allows you to easily bring in data from multiple systems and map it to custom defined service maps and service objectives. It also allows you to integrate with change management and trouble ticketing systems or asset databases so you can create trouble tickets automatically or automatically ignore alarms for servers that are down for scheduled maintenance, etc.
I so far haven't come across anything open source that follows the netcool event processing model. For those of you who never had to use netcool (i'ts big $$$, btw, but very popular among large telcos/isps), all it is is a relational database based event processor that heavily relies on stored procedures and triggers to perform "deduplication" and correlation and is highly customizeable. I don't see anything that their (proprietary) database engine does that couldn't be done with PostgreSQL.
I have not tried this one but, I saw it a few months ago and thought that it looked interesting. It isn't as powerful as OpenNMS or Nagios but it does have some advantages of its own.
This one is Don O'Neill's own combination of several different small packages. Basically, it provides a very nice front end to fairly extensive MRTG monitoring. But, it does give you an idea of what you can do and it certainly looks customizable.
There's this product named ManageEngine OpManager which can do it for you. Try it out ...
http://www.opmanager.com
Big brother
http://www.bb4.org/
and big sister
http://bigsister.graeff.com/
Big Sister does for you:
* monitor networked systems
* provide a simple view on the current network status
* notify you when your systems are becoming critical
* generate a history of status changes
* log and display a variety of system performance data
I worked for a company thats been using big brother for years. great but the config syntax sucks. Big sister is easier i believe but im not sure.
Sorry, have to get in a plug for my employer.
e tview/
:)
n terprise-console/
http://www-306.ibm.com/software/tivoli/products/n
You can buy it on its own, or for even more coolness just buy our Tivoli Enterprise Console and NetView is included free, with an unlimited license (for NV, not TEC)
http://www-306.ibm.com/software/tivoli/products/e