How Should an Application's Logs Work?

emmjayell writes "You've been there, loaded up a new application (think server-based app like Apache or Samba ...), it's working okay for a few days or a few months, then the intermittent problems start. Usually it's the CEO or someone else of relative importance that is the first victim. You can't readily duplicate the problem, so you go to find out where the application put's it's logs - maybe it's in var/log/messages - maybe in it's own directory - sometimes it's right there and available in some administrative GUI. So what makes you happiest when diagnosing the problem? Do you want tools to access it? UI or command line? Do you want it formatted to use tools like cut and sed? Do you have any examples of an app that does a great job with system logging and diag logging? Background: My team is working on an application that is gearing up for a first release. We have a logging framework in place already (we are using Apache: logging.apache.org/) -- so that covers how we are logging, but not what we should log and how it should be laid out for optimal use."

Comments

The logs are in the log directory

[sydb]

as I once said to a colleague. /var/log

If you have simple logging needs, log via syslog and leave the details to the site.

For more complex needs, especially if you have several logs, /var/log/appname/* is good.

Obviously, the logs should be a text file. You ask if special tools should be provided. For text files we already have grep, sed, awk, perl.

The exception is if you are providing some kind of administrative GUI, say a web app. Logs that relate to specific functionality should be near the controls for that functionality. By using a GUI you are saying "I don't want to get my hands dirty" which, for time-pressed admins, is a perfectly legitimate approach for apps with complicated configuration architectures (Sendmail, WebShere 5). So the GUI should take away the complexity of having to know where the logs are. It should always be possible, though, to get at the text of the logs and run standard tools against them.

MHO.

log4j

[Anonymous Coward]

is easily the best logging package.

Configurable log levels, and you can define your own appenders.

Eg, I typically configure an email appender for severe/fatal errors, so they come straight to my inbox. Often I know of problems before the users do as a result of this.

Also, something described in one the Pragmatic Series of books is an RSS appender - just point your RSS reader at the channel, and wait for any errors to occur.

A good reference

[delirium of disorder]

Not log specific, but this describes the Unix-Way(tm) of doing file formats.

http://www.faqs.org/docs/artu/ch05s02.html

Re:syslog!

[EnronHaliburton2004]

Apache and many others decided to roll their own.

That's partially because Apache runs on Windows, which doesn't use syslog by default. Syslog also runs differently on different Unixes, and since not all Unixes are open-source, you can't always fix it.

Coerce Consistency

[BoyBlunder]

I work for a log analysis firm, and the bane of our lives are logs where the information is presented inconsistently from one message to the next. So in some cases a message might have, say, an IP Address as the first word, and in another message its somewhere else in the line with as IP address:port, etc. It's a right royal PITA to write code to extract the IP address in this example if you have to find out every potential message that the app will ever issue in order to automate analysis.

So, in order to make storage, analysis and reporting easy, your framework should attempt to coerce a consistent approach to the data logged - even the plaintext "human readable" data if you can. If you can do the same with metadata about the event (e.g. ID fields, links to online KBs etc), so much the better.

BB

FYI

[mmkkbb]

MS requires that non-MS software sets bit 29 on all custom error codes, giving absurd decimal interpetations of these codes.

Re:Document and be grep-friendly

[Anonymous Coward]

That's actually a pretty bad log format

#1 wastes space with colons. that means you technically have the split on ": ", not " ".

#2 doesn't put the date first. That means if you're merging logs from multiple sources, you can't just sort them.

#3 uses some bizarre ridiculous date format that the author probably made up himself. Where does "May" come from? Is it dependent on the locale? Why does it have "05.May"?? What is CDT? There is more than one time zone that has abbreviation CDT. And if you're talking Central Daylight, well, sometimes that will be *CST* which makes parsing and conversion to UTC just that much more difficult.

#4 The severities can't be matched by Perl and Ruby's "\w". One of them has an exclamation mark.

So, if you're a programming, don't use that log format please! Do this:

2005-05-06 02:04:06 error The printer is on fire!
2005-05-06 02:05:12 notice The printer is no longer on fire.

Where the dates are in the local time zone.

Better yet, just print to stderr WITHOUT dates, and let an external logging package (like multilog) add them.

Re:It's a log

[Anonymous Coward]

less is the tool for log viewing, it will even show binary gob without freaking your terminal.

Re:loggin and microsoft

[Kalgash]

http://www.eventid.net/

Enjoy.

Windows 2000

[Chemisor]

You know what logs I really like? The Windows 2000 system logs. All the services write in them, so there is never any hunting for files. They can have a lot of information packed into them, if the application takes some time to do it. They rotate automatically. They have severity icons so you know which ones are the errors and which are not. And they are all in a nice GUI list, so you don't need a command line PhD to view them. User friendly, indeed!