Monitoring Your Unix Boxen?

Griim asks: "I've been using Linux for years and loving it, and have also worked a bit on a few Sun stations and BSD boxes as well. My question is this: what is the easiest way to keep tabs on all of the activity?"

"I know a few people who 'tail -f' the main log files, or who run 'top' every so-often. These require constant monitoring though, and you could miss essential error messages if you step away for too long. Are there any projects that do this successfully? I've seen a couple out there that started to do this, but appear to be abandoned.

Ideally, I would like some type of all-in-one, that possibly generates a daily (email/web) report of network statistics, user logins, and (web)server traffic/hits, as well as anything 'suspicious' that might be happening, perhaps what apps have been taking most of the processor time, or if any of the daemons have been busier than they normally would be. I know there probably isn't one single app out there that does all of this, so what's the best configuration , for keeping tabs on multiple machines, something I can skim for a minute or two each day, to make sure things are the way they should be? I want to know what works best, and just as importantly, what *doesn't* work (I do realize that relying on a single solution would be bad here too, so if you have more than one suggestion, that would be appreciated)."

Comments

Tripwire

[daeley]

I cron tripwire on an old BSD box I have running and it works well enough. Linxen:

Tripwire.org

FAQ

sourceforge page

Big Brother

[MJArrison]

I've user Big Brother for many years and it is very configurable. You can monitor anything from cpu usage, memory, disk space, available services, to random things like the weather and server room temp.

All that being said, I found it to be flukey in its behavoir. Sometimes it would report that everything was not responding and it had to be punted before I would get the all clear. The other negative is the license. The program consists of nothing more than shell/perl scripts so it's obviously open, but it has some strange clauses about Non-Commercial use.

Overall, I'd recommend trying something else, because BB was unreliable in my use, but YMMV.

Re:Big Brother

[Zocalo]

If you look at, or already use, Big Brother then *please* make sure you read the article on it in issue #60 of Phrack as well. Owing to the way the software is implemented, the thing can be a goldmine of information for hackers and it is *essential* that your BB box is properly secured.

That said, it does appear to be a capable, fully-featured package and I'd guess that as long as you take the proper precautions you should be OK. I can't comment on the stability though; the security concerns I had were enough to cause me to move along to the next product on my list.

Keep an eye on your network traffic

[forged]

Any network monitoring applet docked to your environment will do for real-time stuff, but for historical logs you should consider keeping MRTG logs as well. MRTG works with *everything* and the log file format it uses doesn't grow over time (magic!)

logcheck

[Col. Klink (retired)]

I use logcheck (available as a Debian package). I run it only one one machine and I have all the other machines send their syslogs to that machine.

To quote a recent job candidate I interviewed..

[Nathan Ramella]

'top' apparently is the best tool for monitoring boxen. :)

He's watching you....

[mpechner]

Take a look at big brother. http://bb4.com. Big brother is cross platform and has many hooks. It will monitor all unix and win machines. I do suggest using a UNIX machine as the server. BB has both email and pager support.

The extensions for BB are at http://www.deadcat.net/

I also like tripwire. Checksums of files on the system to know if important files have been changed. last time I used TripWire it has email alerts. The paid for version has an enterprise monitor.

LogWatch is another. Generates email.

Go through your linux and bsd daily, hourly and weekly scripts to see all the tools they run by default. These can be moved to most Unixs. Since most of these are shell and perl rpograms, some might be adaptable under windows using activeXPerl or Cygwin.

The hardest part is fine tuning the emails and alerts to those things you really care about.

MTRG and agreat snmp tool and tied in with BigBrother.

I've has to set these up for security purposes at one site. For monitoring a server fam at another site. A compile farm for doing builds at my current job.

It's all about Nagios...

[Dimwit]

Nagios rocks my socks. Does everything most commercial apps do, and it's free. Rock solid too.

Re:It's all about Nagios...

[Deagol]

Nagios is pretty sweet -- we use it at our shop. It's handy to be notified as soon as a key server goes down.

One thing I like to do personally is randomly pick a startup script (that's actually used in a particular server's configuration), and bury a single line in it that emails me "hostname has rebooted!" as the subject whenever it reboots. That way I know if a machine is ever rebooted with or (more importantly) without my knowledge.

Nagios

[nocomment]

I'm running Nagios. It was SAINT, and before that it was known as SATAN. I've also used big sister before. That's a pretty good big brother clone. Nagios will do what your after though. Just remember that whatever you build will probably take awhile. Creating the config files takes forever.

Lots of stuff

[vadim_t]

logcheck will mail you about unusual stuff that appears in log files.

monit will monitor running damons and can restart them if they crash, use too much CPU/RAM, etc, mailing about anything interesting.

tripwire or lire are nice for monitoring filesystem integrity, but these tools aren't easy to use. The database they use must not be located in a safe place, which can make them impractical.

I think the best thing would be doing all logging to a safe computer that only runs the logging daemon, so that you can be sure you're not missing anything.

Adminux

[jkidd]

Have you looked at http://www.adminux.com It does security monitoring, error monitoring, performance monitoring. Cross platform support. It does cost... I used it to monitor 50 HP-UX boxes, 30 AIX boxes, some Suns, and Linux systms.

monitoring, no one size fits all

[perlchild]

owing to the fact almost no product will fit everyone's needs

here are aspects where you can compare what you will find

aspects of monitoring:
-availability
-uptime(subtly different from availability)
-performance
-security
-capacity
-log or otherwise event-based monitoring

nature of tools:
-web based
-daemon with web based front end
-daemon without web based front end
-other

language tool is written in, license and source
-closed source, nuff said, available in licensed per cpu, licensed per target/service, etc...
-open source, but with paid-for license that includes support(shameless plug... I do support for this kinda thing)
-open source, roll your own support
-perl
-php
-java
-python
-c/c++

integration with other products
-by snmp traps
-by snmp agent extensibility(smux/agentx/proxysnmp,etc...)
-by proprietary methods
-by sharing a RDBMS with another monitoring tool(usually used for things like remedy ARS)

measure of performance/capacity/throughput/usage
-by the exec family of functions
-by the language of choice's own internal library conventions
-by snmp
-by proprietary methods to a Manager of Manager or NMS system
-by ciscoflow/other hardware vendor's protocol
-by parsing logs
-by exec-over-ssh-connexion

examples that don't fit neatly into any category that comes to mind is monitoring of backups(were they performed, how much, which files were skipped, etc, location in jukebox of which tape for which file...

Hope this helps you even draw the lines towards evaluating the product that meets YOUR needs