How Should an Application's Logs Work?

emmjayell writes "You've been there, loaded up a new application (think server-based app like Apache or Samba ...), it's working okay for a few days or a few months, then the intermittent problems start. Usually it's the CEO or someone else of relative importance that is the first victim. You can't readily duplicate the problem, so you go to find out where the application put's it's logs - maybe it's in var/log/messages - maybe in it's own directory - sometimes it's right there and available in some administrative GUI. So what makes you happiest when diagnosing the problem? Do you want tools to access it? UI or command line? Do you want it formatted to use tools like cut and sed? Do you have any examples of an app that does a great job with system logging and diag logging? Background: My team is working on an application that is gearing up for a first release. We have a logging framework in place already (we are using Apache: logging.apache.org/) -- so that covers how we are logging, but not what we should log and how it should be laid out for optimal use."

Comments

loggin and microsoft

[Velex]

Any kid of log is fine with me as long as it's there and it gives me some kind of insight into what's going wrong, e.g. "can't open this file," "that file's corrupt," "null pointer." Of course, text files are nice, because you can actually search through them.

Sadly, most applications for M$ operating systems usually just leave things like, "Error #543892157893421 occured." When you go to look up what error 84901257893423 is, no one in the world seems to have had it. Tech support proceeds to blame your hardware vendor, who blames your software vendor, ad nauseum. Seems like most applications for m$ operating systems just pull error numbers out of their asses.

When Where Who What

[NoSuchGuy]

Maybe you want to know

When from Where did What by Whom

When = ISO 8601 Timestamp (from)
Where = IP Adress / Name of computer...
Who = which Login /registered user did
What requested file foo/bar?a=213b=dfg

syslog!

[Anonymous Coward]

The distressing part is that there's an answer: the syslog daemon, a quite capable application. The problem is that some prominent apps don't use it at all. This is in part due to the limitations of syslog, but the solution is not to reinvent the wheel, it's to modify the wheel so that it does as you wish. syslog doesn't have the facilities you want? Change it so it does. It doesn't deal with things like splitting logs for virtual (web) hosts? Change is so it does.

I say this as a sysadmin for a large number (several thousands) of servers. So many problems could be solved by Apache et al using, and modifying where necessary, the existing solutions. But they don't, they roll their own, and so we see problems.

I realize this is a sorta OT rant, but it's causing major problems for me at work and so I think it's justified. Stuff like rotatelog has functionality to tell Apache "dump your logs, I'm rotating the file!" (which by the way doesn't always work) which could be easily rolled into a single notification of the syslog daemon. But it isn't. For whatever perverse reason, Apache and many others decided to roll their own.

It pisses me the fuck off, having to deal with it. The greatest shortcoming by far of OSS is this insistency on reimplemtning proven, robust, existing solutions in favor of a trivial fix. This is a particularly egregious example, one the OSS world would be served well by acknowledging.

The logs are in the log directory

[sydb]

as I once said to a colleague. /var/log

If you have simple logging needs, log via syslog and leave the details to the site.

For more complex needs, especially if you have several logs, /var/log/appname/* is good.

Obviously, the logs should be a text file. You ask if special tools should be provided. For text files we already have grep, sed, awk, perl.

The exception is if you are providing some kind of administrative GUI, say a web app. Logs that relate to specific functionality should be near the controls for that functionality. By using a GUI you are saying "I don't want to get my hands dirty" which, for time-pressed admins, is a perfectly legitimate approach for apps with complicated configuration architectures (Sendmail, WebShere 5). So the GUI should take away the complexity of having to know where the logs are. It should always be possible, though, to get at the text of the logs and run standard tools against them.

MHO.

A good reference

[delirium of disorder]

Not log specific, but this describes the Unix-Way(tm) of doing file formats.

http://www.faqs.org/docs/artu/ch05s02.html

Document and be grep-friendly

[Proteus]

I really don't much care where logs are kept or what particular format they are in. However, it's important that the man page tells me where the logs are, and clearly documents the format of the log files. What do flags mean? What do particular messages mean?

Also, formatting the logs in such a way that they can be quickly searched with grep or parsed by a simple script is most helpful. One of my favorite loggers does this:

MESG: 2005-05.May-09@09.02.54CDT: Started run
WARN: 2005-05.May-09@09.03.17CDT: Couldn't find file 'control.rc', creating
ERR!: 2005-05.May-09@09.03.18CDT: Unable to create 'control.rc', terminating
MESG: 2005-05.May-09@09.02.54CDT: Completed run. 1 error, 1 warning.

This lets me see everything in chronological order, but I can quickly parse the log. Splitting on ':' will yeild the first two feilds consistently, and the first four chars are *always* the type of log message. So doing something like:

$ egrep '^ERR!: 2005-05.May-09' report.log

Lets me immediately see all the errors for a given day. The key to good logging is, IMO, making sure that the logs can be parsed effectively.

Coerce Consistency

[BoyBlunder]

I work for a log analysis firm, and the bane of our lives are logs where the information is presented inconsistently from one message to the next. So in some cases a message might have, say, an IP Address as the first word, and in another message its somewhere else in the line with as IP address:port, etc. It's a right royal PITA to write code to extract the IP address in this example if you have to find out every potential message that the app will ever issue in order to automate analysis.

So, in order to make storage, analysis and reporting easy, your framework should attempt to coerce a consistent approach to the data logged - even the plaintext "human readable" data if you can. If you can do the same with metadata about the event (e.g. ID fields, links to online KBs etc), so much the better.

BB

Errors should be clear

[__david__]

I agree with what others have said--I don't care about the format too much as long as it's text and has a timestamp in front.

What really matters to me is that errors get logged nicely. "Error number #57575" or even "permission denied" doesn't help at all. It needs to be specific "permission denied while opening file /blah/blah" is infintely better since it lets you actually fix the error without looking something up on google. Speaking of that, if you *do* have some kind of strange error you are reporting and you know it's not going to make sense, at least make it long and unique so I *can* look it up on google. Terseness can be a good quality in a program, but it is *not* a good quality in an error log.

Just remember, the people installing your software and using probably don't the first thing about the way it works on the inside. You have to explain to them what went wrong and give them clues on how to fix it in a short error line.

Oh, it's also nice to have a link back to the original source line that caused the problem. In C, use __FILE__, __LINE__, and __func__ (if you have it) so that if I'm really stumped I can download your source, quickly find the error and start working backwards to find the cause. However, this could get confusing if there are a number of different versions of your code floating around, and it's not as important as the other things I mentioned.

-David