How Should an Application's Logs Work?

emmjayell writes "You've been there, loaded up a new application (think server-based app like Apache or Samba ...), it's working okay for a few days or a few months, then the intermittent problems start. Usually it's the CEO or someone else of relative importance that is the first victim. You can't readily duplicate the problem, so you go to find out where the application put's it's logs - maybe it's in var/log/messages - maybe in it's own directory - sometimes it's right there and available in some administrative GUI. So what makes you happiest when diagnosing the problem? Do you want tools to access it? UI or command line? Do you want it formatted to use tools like cut and sed? Do you have any examples of an app that does a great job with system logging and diag logging? Background: My team is working on an application that is gearing up for a first release. We have a logging framework in place already (we are using Apache: logging.apache.org/) -- so that covers how we are logging, but not what we should log and how it should be laid out for optimal use."

Comments

The logs are in the log directory

[sydb]

as I once said to a colleague. /var/log

If you have simple logging needs, log via syslog and leave the details to the site.

For more complex needs, especially if you have several logs, /var/log/appname/* is good.

Obviously, the logs should be a text file. You ask if special tools should be provided. For text files we already have grep, sed, awk, perl.

The exception is if you are providing some kind of administrative GUI, say a web app. Logs that relate to specific functionality should be near the controls for that functionality. By using a GUI you are saying "I don't want to get my hands dirty" which, for time-pressed admins, is a perfectly legitimate approach for apps with complicated configuration architectures (Sendmail, WebShere 5). So the GUI should take away the complexity of having to know where the logs are. It should always be possible, though, to get at the text of the logs and run standard tools against them.

MHO.

A good reference

[delirium of disorder]

Not log specific, but this describes the Unix-Way(tm) of doing file formats.

http://www.faqs.org/docs/artu/ch05s02.html

Coerce Consistency

[BoyBlunder]

I work for a log analysis firm, and the bane of our lives are logs where the information is presented inconsistently from one message to the next. So in some cases a message might have, say, an IP Address as the first word, and in another message its somewhere else in the line with as IP address:port, etc. It's a right royal PITA to write code to extract the IP address in this example if you have to find out every potential message that the app will ever issue in order to automate analysis.

So, in order to make storage, analysis and reporting easy, your framework should attempt to coerce a consistent approach to the data logged - even the plaintext "human readable" data if you can. If you can do the same with metadata about the event (e.g. ID fields, links to online KBs etc), so much the better.

BB