Slashdot Mirror
Network Monitoring and Alerting?
SpamMonkey asks: "At work I am trying to implement a central monitoring and alerting service. We have in excess of 250 Windows servers, approx 15 AIX servers and another 30 Linux servers (mainly SLES/Suse). My investigation into systems that will allow us to monitor critical areas on each of these systems has so far led me to a clustered Linux server running Nagios with passive and active checks. What I'm curious about though is how Slashdot readers are carrying out their own jobs and how they can comfortably sit back, without having to repeatedly check that various systems are still operational and how to cut down their own response times when something goes wrong."
Used at my Fortune 500 company to monitor thousands of Win/Linux/Solaris. http://www.bb4.org/
We are in the middle of a large scale Linux project .. replacing 900 SCO Unix servers with Linux. We are wondering the same thing .. what monitoring tools can be put in to watch these servers?
... the IBM equivalant (A Tivoli product I think?). No need to reinvent the wheel.
... how do you balance your need to know everything about every box all the time (well, the business and management want to know at least) vs the tiny amount of data you can push without interfering with mission critical applications running over that same link?
Our big hitch comes from the fact that we only have a satellite connection to each of the remote sites, so we can't do real time monitoring, so things like HP Openview NNM are out of the question - they use too much bandwidth.
Our solution (And the reason I'm working late right now) is to build a custom suite of tools that does batch reporting every night by polling logs and custom programs, then sends it back in a handy xml file. We take that file, dump it into a large informix database, and then we can do whatever we want to create reports.
It's a little more work than just installing a package, but we're getting EXACTLY what we want out of the product. It works with our very unique communications and configuration, and it's modular so I can add whatever monitoring/checks I want by writing a new ksh script. All the output is standardized and all the parsing is done at the office by a very clever xml parser one of the db guys wrote.
I think for whaty ou're looking for, theres things like Big Brother, MRTG, HP Openvie umm
But I'd love some feedback for people who are working in a bandwidth sparse shop like me
We emerge from our mother's womb an unformatted diskette; our culture formats us. - Douglas Coupland
So you're concerned about letting things go on auto-pilot and missing an alarm . . .
:). Either way, nagios has ASSLOADS of event notification options.
Why not slap a modem into the head nagios box and have it page you when things fail. Don't worry about having to wear a beeper - you can page most cellphones via your carrier's SMS gateway (still dial-capable).
Too much hassle? How about AIM? YIM? Jabber? Email?
If you're TRULY the teeth jittering, chain smoking NOC type, buy some x10 crap and build a physical network alarm interface like I did
Here's a couple of the monitoring solutions:
Opennms
Mon
Big Brother
For system information polling I'd go with:
Cacti hands down this is the best polling system out there and it's simple to setup and run.
Check out Mon and Mon.cgi
Rule #1: Do not add to the problem.
This is an entire category of Operations Management and can encompass everything. Don't take it lightly and don't be afraid to start small. The first thing you need to do is categorize what you want to monitor into individual sections and work on the easiest stuff first. By the time you work up to the tough stuff, you'll have an idea of whats available, what your capabilities are and hopefully the easy stuff can be quickly rewritten/integrated into a netter solution. Don't miss the critical stuff in a morass of junk alerts. Sample consideration (everything that moves in the data center):
Hardware:
---------
1) Server
2) Storage
3) Network
4) Power
5) Environment
6) Security
Software:
---------
1) OS
2) Applications
3) Security
Events:
-------
1) Failures
2) Alerts
3) Misfires
4) Security
Triggers:
---------
1) Notification/False positives
2) Action plans/Event handling
3) Documentation, Documentation, Doumentation
4) Reporting aka analysis and cya
5) Security
Once you're done building it, start over. The last tier is the most visible e.g. delegating a raid rebuild page to the opcenter flunky without proper documentation is a Career Limiting Move (CLM), building the best monitoring system is a fucking waste unless you pay attention to it. The most apt cliches for monitor normalisation are all military: Warrooms, Bridges, Weapons Hot, Communications channels etc. View everything as SNAFU and work from there.
Rule #1: Do not add to the problem.