Slashdot Mirror
How Should an Application's Logs Work?
emmjayell writes "You've been there, loaded up a new application (think server-based app like Apache or Samba ...), it's working okay for a few days or a few months, then the intermittent problems start. Usually it's the CEO or someone else of relative importance that is the first victim. You can't readily duplicate the problem, so you go to find out where the application put's it's logs - maybe it's in var/log/messages - maybe in it's own directory - sometimes it's right there and available in some administrative GUI. So what makes you happiest when diagnosing the problem? Do you want tools to access it? UI or command line? Do you want it formatted to use tools like cut and sed? Do you have any examples of an app that does a great job with system logging and diag logging? Background: My team is working on an application that is gearing up for a first release. We have a logging framework in place already (we are using Apache: logging.apache.org/) -- so that covers how we are logging, but not what we should log and how it should be laid out for optimal use."
Comment removed based on user account deletion
Well, I'd like to abandon syslog in favour of logging to an SQL database for everything (a central sysloggable database would be nice, a standard API and what have you). Text logs are nice, but given the choice, I'd rather put everything into MySQL. It's searchable, it's archivable, it's a lot faster to process than plaintext. A hundred megs of logs takes forever to process with perl, but if it's in the database, you can make a lot of queries live.
..." --database logs' and such.
SELECT SUM(transfersize)/1024 FROM logs.apache WHERE vhost = "files.darien.ca" AND date > "2005-05-01 00:00:00" AND date "2005-06-01 00:00:00"
Now I have the amount of bandwidth (in kilobytes) that was served from my files site in the month of May. Doing this with raw logs is absurdly slow in comparison, and only gets moreso as time goes on. If you want to archive them in compressed format, you can do 'mysqldump --where="date
So my recommendation for logging: support mysql!
I really don't much care where logs are kept or what particular format they are in. However, it's important that the man page tells me where the logs are, and clearly documents the format of the log files. What do flags mean? What do particular messages mean?
Also, formatting the logs in such a way that they can be quickly searched with grep or parsed by a simple script is most helpful. One of my favorite loggers does this:
This lets me see everything in chronological order, but I can quickly parse the log. Splitting on ':' will yeild the first two feilds consistently, and the first four chars are *always* the type of log message. So doing something like:
Lets me immediately see all the errors for a given day. The key to good logging is, IMO, making sure that the logs can be parsed effectively.
We may not imagine how our lives could be more frustrating and complex—but Congress can. – Cullen Hightower