EFF's Logfinder

clonebarkins writes "EFF has just released a new software tool called "logfinder" to help server admins find (and delete) unnecessary log files on their boxen. "By finding unwanted log files, logfinder informs system administrators when their servers are collecting personal data and gives them the opportunity to turn logging off if it isn't gathering information necessary for administering the system.""

Comments

Redhat

[RalphLeon]

The last time I checked out redhat (about version 8 I rekon) they inluded this nice little utility called "logviewer". And, I though, wow a text viewer how novel, Linux doesnt have many text viewers.

So not only is this a text viewer, but it also finds all those logs hidden in /var/log/*, it must be hard to find anything in /var/log/* ...

interesting...

[Spider[DAC]]

Actually, it uses lsof and a few other niceties to locate open files that change over time, then scans them for presence of time/date stamps, mailaddress or other "log" activity.

So, no, its not just "locate log" that somone suggested, nor is it "find /var/log" either, but a bit more complex.

As for the comment about competent site-admin. This is a bit more than that too, its also about users and active software, peoples IRC logs, various ftp clients that clobber up and log passwords along with everything else in their config dir. And so on and so forth.

Re:interesting...

[scrotch]

Right. From their README:

"... We have created a program called logfinder as a sample means of locating files that might be logs on an existing system. logfinder uses regular expressions to find local files with "log-like" contents; you can customize those expressions if necessary to meet your needs. logfinder requires Python 2 or greater and finds logs in text files on a POSIX-like system. (It might also find some log-like data in binary files if the binary files represent that data in textual form.)

logfinder can, if the lsof program is installed and when run with
appropriate privileges, detect open files systemwide that grow larger over time. It can also search for text that may indicate logging activity within a given directory hierarchy, or systemwide. As we suggest above, a program like logfinder can find some, but not all, kinds of logging activity. For example, logfinder will generally not identify logs in binary (non-text) formats or logs kept inside
databases. Therefore, using a program like logfinder is usually a supplement to, not a replacement for, answering questions like those given above.

logfinder should be run as root. If logfinder is invoked without any arguments, it will examine open files systemwide to see whether they grow larger, and then indicate whether files that appear to be growing contain log-like text. (This requires lsof to be installed, and lsof's ability to report open files accurately may depend on your operating system. So far, we've had success with Linux and MacOS X, and some difficulty with FreeBSD and OpenBSD.)

If logfinder is given one or more directory names as arguments, it will search for log-like text in files in those directories. ..."

I haven't run it (and likely won't), but I'm curious whether it would ever flag stuff in /var/mail. It would be a lot of fun to have clueless admins cronning this and deleting mail every night.

Re:I appreciate the effort but...

[ObsessiveMathsFreak]

Most MCSE trained, NT sysadms don't really have a complete understanding of their servers and how they work. Most are just part time admins, doubling up as postmasters, network support and helldesk frontliners. A great many Windows server administrators are simply in fact, the company management accountant, who may never have recieved any computer training whatsoever! Many will not know where to begin looking for files without googling for the answer. This issue stems from the poor quality of the MCSE courses about, rather than from organisational difficulties with the NT servers themselves.

Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over /etc /var /usr /root /usr/X11 and even (I kid you not) /bin. The rather haphazard way different programs save their files about *nix systems can be a headache sometimes. It would be nice if someone would standardise the process. However, such a thing has been tried with disasterous results, i.e. the windows registry, so I guess I should be careful what I wish for!

In short, competant *nix admins will know most of the many location where their important daemons are storing logfiles. NT admins on the other hand, many not even know what daemons are running on the machine anymore, let alone where they store their log files!

P.S.
Hey wait! This is a python app. I guess NT admins will just have to keep on googling.

My experience with logfinder

[carpe_noctem]

Running openbsd 3.6 on x86:

tresor:src$ tar xfvz logfinder-0.1.tar.gz
logfinder-0.1
logfinder-0.1 /logfinder.py
logfinder-0.1/README
logfinder-0.1 /COPYING
tresor:src$ cd logfinder-0.1
tresor:logfinder-0.1$ sudo ./logfinder.py
Scanning for open files systemwide...
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'

(repeats several dozen more times...)

[Errno 2] No such file or directory: '(/dev/wd1a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/raid0a)'
[Errno 2] No such file or directory: '(/dev/raid0a)'
[Errno 2] No such file or directory: '(/dev/wd1a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd1a)'
Scanned sizes of 30 files.
Waiting for log activity; please allow time to elapse.
Press Enter to continue.
[Errno 2] No such file or directory: '(/dev/wd1a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/raid0a)'
tresor:logfinder-0.1$ cd ..
tresor:src$ rm -rf logfinder*

At least the EFF's lawyers are better than their programmers. ;)

Re:Can't subpeona what doesn't exist?

[jafiwam]

Destroying evidence is indeed illegal. However, before you are aware of, or have "reasonable belief" of a lawsuit or criminal investigation logs are not evidence yet and may be deleted freely.

I do exactly that with logs for my company. Once a month I clean out everything we don't need, including "email logs" and other stupid shit MS piles up in various places in the operating system. If/When the lawyers/cops come knocking, I can point to the policy and scheduled reminder and say "sorry, dont have that".

Logs are not the only place stuff resides and piles up, but it's one easy fix and keeps my servers and machines clear of unnecessary disk-space robbing files.

Re:WTF is Boxen?

[TheRaven64]

From the Jargon File:

boxen /bok'sn/ pl.n. [very common; by analogy with VAXen] Fanciful plural of box often encountered in the phrase `Unix boxen', used to {Unix">describe commodity {Unix hardware. The connotation is that any two Unix boxen are interchangeable.

And yes, you are seriously behind the the times. The oldest copy of the Jargon File I have is from the early '90s and that contains the word boxen.

It can be quite useful, since boxen are always computers, while boxes can be the packaging the computers came in.

Could be moderately useful

[WarmBoota]

This tool could be moderately useful, especially in an environment where the administrator can't be expected to know all of the ins and outs of third-party add-ons.

I was once assigned to a dotcom that used a third-party component to allow for credit card transactions. What the admin didn't realize was the default configuration left the component in debug mode, placing all user-submitted credit card data in plain text files on the web server

We only found the log file accidentally while performing an unrelated search for files modified in the last 'n' days. The admin relied on the developers to configure the third-party component and the developers were relying on another set of consultants who didn't know or didn't care about the log files.

Re:I appreciate the effort but...

[EnronHaliburton2004]

Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over /etc /var /usr /root /usr/X11 and even (I kid you not) /bin. The rather haphazard way different programs save their files about *nix systems can be a headache sometimes. It would be nice if someone would standardise the process.

I don't think you understand *nix logging, or you've been working with poorly-designed systems.

Locations for log files has been pretty well standardized by Posix and the LSB. Logs generally go in /var/log (or /var/adm on older systems), or in $APPLICATION_ROOT/log. A sysadmin might write a log to /var or /root, but those are temporary logs.

Logfiles which end up in /etc, /bin, /usr or /usr/X11 is the result of poor or very old configuration.

Now, compare this to a Windows 2003 Server running Exchange 2003, where the log files in c:\windows c:\Windows\system c:\Windows\system\Logfiles c:\Windows\system\security
C:\Program Files\Exchsrvr\ C:\Program Files\Exchsrvr\MDBDATA C:\Program Files\Exchsrvr\mtdata . Many of the logfiles are not viewable with a text viewer. Some of the log files really aren't "Log files", but are "Transaction Logs", which is a different thing in my book.

Some of this makes sense, some of this does not. But I'm not a windows admin, and I didn't design this network here, so maybe this is the result of a poor configuration.