Slashdot Mirror
Monitoring Your Unix Boxen?
"I know a few people who 'tail -f' the main log files, or who run 'top' every so-often. These require constant monitoring though, and you could miss essential error messages if you step away for too long. Are there any projects that do this successfully? I've seen a couple out there that started to do this, but appear to be abandoned.
Ideally, I would like some type of all-in-one, that possibly generates a daily (email/web) report of network statistics, user logins, and (web)server traffic/hits, as well as anything 'suspicious' that might be happening, perhaps what apps have been taking most of the processor time, or if any of the daemons have been busier than they normally would be. I know there probably isn't one single app out there that does all of this, so what's the best configuration , for keeping tabs on multiple machines, something I can skim for a minute or two each day, to make sure things are the way they should be? I want to know what works best, and just as importantly, what *doesn't* work (I do realize that relying on a single solution would be bad here too, so if you have more than one suggestion, that would be appreciated)."
I cron tripwire on an old BSD box I have running and it works well enough. Linxen:
Tripwire.org
FAQ
sourceforge page
I watched C-beams glitter in the dark near the Tannhauser gate.
I've user Big Brother for many years and it is very configurable. You can monitor anything from cpu usage, memory, disk space, available services, to random things like the weather and server room temp.
All that being said, I found it to be flukey in its behavoir. Sometimes it would report that everything was not responding and it had to be punted before I would get the all clear. The other negative is the license. The program consists of nothing more than shell/perl scripts so it's obviously open, but it has some strange clauses about Non-Commercial use.
Overall, I'd recommend trying something else, because BB was unreliable in my use, but YMMV.
maybe it is now ....'
times change and you sound like grandpa saying 'Back when I was young
Any network monitoring applet docked to your environment will do for real-time stuff, but for historical logs you should consider keeping MRTG logs as well. MRTG works with *everything* and the log file format it uses doesn't grow over time (magic!)
I use logcheck (available as a Debian package). I run it only one one machine and I have all the other machines send their syslogs to that machine.
-- Don't Tase me, bro!
'top' apparently is the best tool for monitoring boxen. :)
http://www.remix.net/
The extensions for BB are at http://www.deadcat.net/
I also like tripwire. Checksums of files on the system to know if important files have been changed. last time I used TripWire it has email alerts. The paid for version has an enterprise monitor.
LogWatch is another. Generates email.
Go through your linux and bsd daily, hourly and weekly scripts to see all the tools they run by default. These can be moved to most Unixs. Since most of these are shell and perl rpograms, some might be adaptable under windows using activeXPerl or Cygwin.
The hardest part is fine tuning the emails and alerts to those things you really care about.
MTRG and agreat snmp tool and tied in with BigBrother.
I've has to set these up for security purposes at one site. For monitoring a server fam at another site. A compile farm for doing builds at my current job.
Actually, boxen IS a word. I quote: [B]boxen[/B] \Box"en\ (b[o^]ks"'n), a. Made of boxwood; pertaining to, or resembling, the box (Buxus). [R.] The faded hue of sapless boxen leaves. --Dryden. [I]Source: Webster's Revised Unabridged Dictionary, © 1996, 1998 MICRA, Inc.[/I]
Nagios rocks my socks. Does everything most commercial apps do, and it's free. Rock solid too.
...but it's being eaten...by some...Linux or something...
I always thought syslogd could do this over SNMP... that is transport all your logs to your workstation :-)
or something
I'm not sure if I'm bullshitting or not
I'm likely misinformed
Buttsex.
I'm running Nagios. It was SAINT, and before that it was known as SATAN. I've also used big sister before. That's a pretty good big brother clone. Nagios will do what your after though. Just remember that whatever you build will probably take awhile. Creating the config files takes forever.
/* oops I accidentally made a comment, sorry */
I use Orca (but then I'm its author :) ) to monitor Solaris and Linux boxes. I used it at Yahoo!/GeoCities to monitor 200 boxes and it was easy to see when systems were doing odd stuff.
Sample Solaris and Linux plots. The Solaris version shows a whole ton of web server stats.
logcheck will mail you about unusual stuff that appears in log files.
monit will monitor running damons and can restart them if they crash, use too much CPU/RAM, etc, mailing about anything interesting.
tripwire or lire are nice for monitoring filesystem integrity, but these tools aren't easy to use. The database they use must not be located in a safe place, which can make them impractical.
I think the best thing would be doing all logging to a safe computer that only runs the logging daemon, so that you can be sure you're not missing anything.
I know this is slightly offtopic, even though he used it in the topic of this article. but seriously, the use of the word boxen is stupid, we dont say "a box factory makes boxen", so dont use the term for computers.
Whenever someone uses that word to me i turn around and stop listening to them, it really makes me question there inteligence both in the IT field and in general inteligence.
Logwatch is a pretty decent system. I comes with Red Hat (and probably other distributions as well) and mails you a summary of the system log. The main thing I use it for is to keep track of what IPs are connecting to which services how many times.
Have you looked at http://www.adminux.com It does security monitoring, error monitoring, performance monitoring. Cross platform support. It does cost... I used it to monitor 50 HP-UX boxes, 30 AIX boxes, some Suns, and Linux systms.
I rolled my own, mostly in Ruby (and ran it in parallel with the previous solution for several months). The main reason? I wanted to know about the things I wanted to know about, and not have to dig the information out of a lot of other cruft. So I do a lot of filtering to supress details that fall within what I define as "normal" for my setup, and only report the exceptions.
The main benifit of this turned out to be that I learned a lot about a configuration that I thought I knew inside and out. Yes, it was more work than dropping in a ready made package, but in retrospect it was well worth it.
-- MarkusQ
I wrote an app called LogMon that allows the user to sorta have multiple 'tail -f' sessions in one terminal (does a 'split-screen' effect). Also does syntax coloring in a user configurable file...
"Ignorance more frequently begets confidence than does knowledge"
- Charles Darwin
what is the easiest way to keep tabs on all of the activity?
Well, Office Depot has an excellent selection of tabs, I prefer the plain clear ones, but they also have packs of the colored ones................
When I fist ran across the problem of monitoring servers, I downloaded every one I could find, got free trials of all the commercial ones. I ended up with netsaint, not because it was better, but because it did exactly what I wanted it to do and nothing more. I wrote a couple of little modules for some switches and I was in business. I needed monitoring yesterday, so I didn't wan't anything too complex, I needed something with a GUI for others, hey it worked....
owing to the fact almost no product will fit everyone's needs
here are aspects where you can compare what you will find
aspects of monitoring:
-availability
-uptime(subtly different from availability)
-performance
-security
-capacity
-log or otherwise event-based monitoring
nature of tools:
-web based
-daemon with web based front end
-daemon without web based front end
-other
language tool is written in, license and source
-closed source, nuff said, available in licensed per cpu, licensed per target/service, etc...
-open source, but with paid-for license that includes support(shameless plug... I do support for this kinda thing)
-open source, roll your own support
-perl
-php
-java
-python
-c/c++
integration with other products
-by snmp traps
-by snmp agent extensibility(smux/agentx/proxysnmp,etc...)
-by proprietary methods
-by sharing a RDBMS with another monitoring tool(usually used for things like remedy ARS)
measure of performance/capacity/throughput/usage
-by the exec family of functions
-by the language of choice's own internal library conventions
-by snmp
-by proprietary methods to a Manager of Manager or NMS system
-by ciscoflow/other hardware vendor's protocol
-by parsing logs
-by exec-over-ssh-connexion
examples that don't fit neatly into any category that comes to mind is monitoring of backups(were they performed, how much, which files were skipped, etc, location in jukebox of which tape for which file...
Hope this helps you even draw the lines towards evaluating the product that meets YOUR needs
Palantir kan be found at www.netsonde.com. It's a system not entirely unlike Nagios, written mostly in Perl. Works with all the unix-like OSs I can think of in addition to Windows.
I had good experience with the following tools: cacti
It's based on RRD the successor of MRTG (not much developed anymore, but still a good tool). Thanks Tobi btw.
OpenNMS is a really powerful realtime monitoring tool
Nagios also...
Don't forget snort for your IDS needs and add acidlab for good visualization of snort's results.
I've been extremely impressed with Cacti for statistic monitoring. It can be found at: http://www.raxnet.net/products/cacti/ It's quite easy to set up, and for larger sites, it has an excellent user privilege system.
InterMapper is what I said the last time someone asked this question on Slashdot.
Unfortunately, as you might guess by the wide varity of recommendations here, you need a combination of packages to create a complete solution. There isn't one program that will satisfy all requirements (monitoring, notification, performance stats, reporting, trending, etc.).
As an example, we use the following:
Nagios
Notifications and real-time monitoring.
Logcheck
Daily syslog reports.
cfengine
Configuration and limited problem correction.
SAR
Performance data. Well, it was free with the OS. Unfortunately, we don't have a good solution to collate and analyse this data.
WebSphere Resource Analyser
Actually, we just fire this up for a laugh and to impress management. Ghod forbid we ever took anything it said seriously. Avoid proprietary crack!
There are other tools we run on an adhoc basis, like nmap. I think snort is an excellent security tool, but our network isn't set up to support it yet. If you bite the bullet, you can probably achieve a lot by installing NET-SNMP everywhere, with a decent SNMP monitoring package.
Remember this: syslog is your friend. If you have a lot of locally-developed scripts that run regularly, you can make a big gain by inserting commands to log important events at a single, reserved syslog priority level, which is directed to a monitored file on a central loghost. E.g.:
logger -p local1.info -t $0 "ran OK"
logger -p local1.warning -t $0 "failed"
etc.
Be prepared to spend several weeks or months configuring and tuning this stuff to give you what you want. If you want something integrated or complex, be prepared to write the code for gluing it all together.
Ade_
/
Big Bubbles (no troubles) - what sucks, who sucks and you suck
It's skinnable, configurable and supports plugins. I've seen it working on Solaris and Linux, YMMV. It's here (with screenshots).
I installed Zabbix on some boxes recently with good results. It monitors the health of your boxes as well as the health of numerous programs running on those boxes and it will email you whenever certain conditions (which you define) change. It focuses more on making sure all programs are running properly and tracking system resources, so it may not be as security oriented as it sounds like you want, but it isn't too hard to add monitoring of new things, so you could probably add triggers for what you consider suspicious without a whole lot of trouble.
-----
Free P2P Backup, Windows & Linux
I spent an hour trying to configure Nagios recently before finally giving up. As a result of its great flexibility and tremendous feature set, it's a horrible bitch to configure. Think Sendmail before m4, and you've got a good idea.
I'll just check back on their site every few months. When they've got m4 for Nagios, we'll talk.
-Waldo Jaquith
... somewhat related. The best situation is where the machines stay stable and don't give trouble in the first place. So avoid bleeding-edge products, use the best-engineered hardware you can afford, and run the machines well within their resource and performance limits. Works for Unix-type machines, and it worked for VMS-controlled boxen back when I was still working with them. Admittedly, it's the 'gold-plated' approach, but if the main thing you need to worry about is whether the box is dead or alive then monitoring becomes much easier.
I would wholeheartedly reccomend SiteScope from Freshwater Software. I was introduced to this software at my last job. It does a great job of monitoring multiple sites on a host, as well as monitoring the host itself (i.e. CPU utilization, memory utilization, etc.)
The best part about SiteScope is that it does not require any sort of client on the servers that it monitors. It uses SSH/telnet/rlogin/etc to make a connection and use normal system utilities to parse out the data that it needs. You can even monitor NT boxes with SiteScope, but you have to install SiteScope on an NT box in order to remotely monitor other NT boxes.
It can be pricey, but it is proven. You can also extend it to do whatever types of monitors you can think of via custom scripts that you write. They do offer a free demo verson from their website if you want to try it out.
-- Charles A. Plater