EFF's Logfinder

clonebarkins writes "EFF has just released a new software tool called "logfinder" to help server admins find (and delete) unnecessary log files on their boxen. "By finding unwanted log files, logfinder informs system administrators when their servers are collecting personal data and gives them the opportunity to turn logging off if it isn't gathering information necessary for administering the system.""

Comments

I just made one, too

[Anonymous Coward]

locate log

Is a new tool really necessary?

[Lord Kano]

A competent admin will know that his/her boxen are collecting personal data. An ethical admin will get rid of any unneeded data.

LK

Can't subpeona what doesn't exist?

[PornMaster]

Is the point of this that law enforcement can't subpoena records that don't exist?

Re:Can't subpeona what doesn't exist?

[xC0000005]

I think so, but really it's just another step in an arms race. How long until we see court orders to collect this sort of information? Or forbidding the use of log destruction/filtering tools?

Re:Can't subpeona what doesn't exist?

[sporktoast]

If an admin is just using this tool to destroy potentially incriminating logs, then they are using it poorly. Like trying to pound a screw in with a hammer.

The use this has for an admin is to survey (or for the less experienced admin, to discover) what logs the system is currently, so that the admin can decide as a policy which logs should be active or not, and with what level of detail. The itch this tool scratches is that many systems as a default keep more logs than perhaps are necessary. A good admin will shut off whatever is deemed unnecessary, based on multiple criteria (security, system load, user/company privacy).

Forbidding the use of log destruction tools (rm?) is moot. Destroying evidence is illegal. Now, laws (or court orders) mandating a level of logging are a completely different matter.

Re:Can't subpeona what doesn't exist?

[jafiwam]

Destroying evidence is indeed illegal. However, before you are aware of, or have "reasonable belief" of a lawsuit or criminal investigation logs are not evidence yet and may be deleted freely.

I do exactly that with logs for my company. Once a month I clean out everything we don't need, including "email logs" and other stupid shit MS piles up in various places in the operating system. If/When the lawyers/cops come knocking, I can point to the policy and scheduled reminder and say "sorry, dont have that".

Logs are not the only place stuff resides and piles up, but it's one easy fix and keeps my servers and machines clear of unnecessary disk-space robbing files.

I appreciate the effort but...

[garcia]

I would seriously hope that:

a) the sysadmins are competent enough to handle this themselves. I would think that a sysadmin would know how to use some sort of local file search.

b) the EFF understands that it's not always up to the sysadmins to determine the amount of time to keep logs that might be used against an individual.

Re:I appreciate the effort but...

[ObsessiveMathsFreak]

Most MCSE trained, NT sysadms don't really have a complete understanding of their servers and how they work. Most are just part time admins, doubling up as postmasters, network support and helldesk frontliners. A great many Windows server administrators are simply in fact, the company management accountant, who may never have recieved any computer training whatsoever! Many will not know where to begin looking for files without googling for the answer. This issue stems from the poor quality of the MCSE courses about, rather than from organisational difficulties with the NT servers themselves.

Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over /etc /var /usr /root /usr/X11 and even (I kid you not) /bin. The rather haphazard way different programs save their files about *nix systems can be a headache sometimes. It would be nice if someone would standardise the process. However, such a thing has been tried with disasterous results, i.e. the windows registry, so I guess I should be careful what I wish for!

In short, competant *nix admins will know most of the many location where their important daemons are storing logfiles. NT admins on the other hand, many not even know what daemons are running on the machine anymore, let alone where they store their log files!

P.S.
Hey wait! This is a python app. I guess NT admins will just have to keep on googling.

Re:I appreciate the effort but...

[EnronHaliburton2004]

Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over /etc /var /usr /root /usr/X11 and even (I kid you not) /bin. The rather haphazard way different programs save their files about *nix systems can be a headache sometimes. It would be nice if someone would standardise the process.

I don't think you understand *nix logging, or you've been working with poorly-designed systems.

Locations for log files has been pretty well standardized by Posix and the LSB. Logs generally go in /var/log (or /var/adm on older systems), or in $APPLICATION_ROOT/log. A sysadmin might write a log to /var or /root, but those are temporary logs.

Logfiles which end up in /etc, /bin, /usr or /usr/X11 is the result of poor or very old configuration.

Now, compare this to a Windows 2003 Server running Exchange 2003, where the log files in c:\windows c:\Windows\system c:\Windows\system\Logfiles c:\Windows\system\security
C:\Program Files\Exchsrvr\ C:\Program Files\Exchsrvr\MDBDATA C:\Program Files\Exchsrvr\mtdata . Many of the logfiles are not viewable with a text viewer. Some of the log files really aren't "Log files", but are "Transaction Logs", which is a different thing in my book.

Some of this makes sense, some of this does not. But I'm not a windows admin, and I didn't design this network here, so maybe this is the result of a poor configuration.

Oh, yeah

[Otter]

God forbid professional sysadmins should be expected to understand how their services are configured and what files are being written. If I were a user on one of their systems, sendmail log files would be the least of my concern.

Re:Oh, yeah

[stephenbooth]

In an ideal world every system would be administered by a well trained and experienced system admin, or a trainee admin being mentored by one, who had plenty of time to investigate and maintain the machine. In practice most system admins are people in other roles (developers, DBAs, desktop support or even receptionists) who have been handed the task of managing half a dozen white box Wintel servers (with maybe a SCO or Linux box or even an aging Sun box in the mix) and probably a Netware server doing file and print, most were built and installed by someone one of the manager's knows or have been inherited third hand from another company. If they're lucky they get a training course where they'll learn a few of the GUI screens, more likely they'll be given a few dozen pages of handwritten notes (aka 'the manual') and told to go to the nearest Waterstones/Borders/Whatever and buy a book if they need more.

That was pretty much my first job. I had trained as a C programmer; then I found myself managing 70 desktops running various versions of Windows, a dozen or so White Box Intel based servers running Windows NT 3.51 and 4.0, a SCO OpenServer box, an Alpha running VMS, a 3 member VAX cluster running VMS and an RS6000 running AIX. All with no usable documentation or training. A little later they added in DBAing the Oracle databases and managing the network (a variety of devices from 3Com, Cisco and Bay), at the time I only knew a bit of SQL and wasn't really sure of the difference between a router and a switch. After spending a lot of money on books then a lot of time reading them (I didn't have web access at the time, when I did I started reading websites as well) I eventually learned what I needed to know.

This script is a separate issue. Inpractice I don't expect those sorts of admins to run it, they probably wouldn't know what to do with the information if they did. Where I think it would be useful is for the professional admin who suddenly inherits a bunch of machines (maybe they've moved companies or their company has merged with another). Put this script on them and run it for a few days then see what it turns up. No matter how wonderful and professional you are unless you built and installed a machine yourself and can guarantee that no-one else has ever had the root/admin password to a box you can't be 100% sure that there's not some process running somewhere that is quietly logging something somewhere. No-one who manages a non-trivial number of machines has time to check every machine to make sure that there are no new or unexpected services that have snuck in (and remember it's not something you could do once and then not again, you'd have to keep on doing it). That's why you need scripts that look for anything that could point to unexpected activity. Not just looking for anything that looks like a log on a box but also ports that shouldn't be open (I've lost count of the number of times I've found a box with port 25 open when I know I've disabled SMTP, only to find that someone has re-enabled it without telling me) or unexpected activity on a switch or firewall port. Not only do we have too many machines to manage but also users who delete files they shouldn't which then must be restored from backup, managers who constantly demand reports on system availabity stats and projects that we have to keep an eye on to make sure they don't run wild and break every standard we have.

Stephen

"Boxen"

[m_member]

Can I have a tool to locate and delete people who use the word 'boxen'? GPL preferably.

is this stupid?

[digitalgimpus]

I'm sorry, but this might just be the dumbest move yet they have made...

lots are crucial for many reasons:

1. Hacking attacks (how else do you track them, and prevent them)?
2. Abuse problems (spammers, credit card fraud)
3. aggregate statistics (what percentage of my customers are based in Europe?)

I can't see why someone would shoot themselves in the foot and use this.

Like log files are really intrusive anyway.

Just as an example...

[PartialInfinity]

This is just EFF's way of reminding sysadmins to be vigilent about their log files, it's not meant to replace good administration habits.

Interesting Motive

[peterdaly]

My first thought was the main purpose of this would be to identify and eliminate "wasted" disk space. There are a bunch of logs that, without management, really just end up being wasted bits on your disk. Generally, that may be a useful utility, at least to me.

I was suprised to see the EFF seems to have a totally different motivation. It seems their real motivation is that the government can't demand logs that don't exists, or more specifically you can't get in trouble for not providing what you don't actually have.

Not sure what I think of that...

Re:Excellent

[ab384]

Just two observations: (the second of which is actually relevant to this parent)

1. It took me around 3 minutes to find out that this thread applies to POSIX-like systems only (ie. won't work on this winXP). The fact should really have been mentionned in the summary. I only say this because recently, some summaries seem to have been "hastily" written.

2. I am myself wary of huge, hidden log files that either winXP itself or other programs create. As the only user and sysadmin on this system and keen to minimise disk wastage, I would want to prune all logs regularly. Trouble is, they aren't all *.log files. So, how do I find them, short of going through every single program and investigating any logs it might or might not create?

interesting...

[Spider[DAC]]

Actually, it uses lsof and a few other niceties to locate open files that change over time, then scans them for presence of time/date stamps, mailaddress or other "log" activity.

So, no, its not just "locate log" that somone suggested, nor is it "find /var/log" either, but a bit more complex.

As for the comment about competent site-admin. This is a bit more than that too, its also about users and active software, peoples IRC logs, various ftp clients that clobber up and log passwords along with everything else in their config dir. And so on and so forth.

Re:Thanks EFF!

[innerweb]

As is always, that which helps to protect the innocent can be used to protect the *evil*. The problem is the innocent do not know what is being done, and the *evil* are studying and learning to use and abuse. Nothing new there.

InnerWeb

Re:neat

[e2d2]

Any tool could probably be used for evil. For instance I have a calendar on my wall. If I took it down and rolled it up, I could probably beat you half way to death with enough strong blows.