EFF's Logfinder

clonebarkins writes "EFF has just released a new software tool called "logfinder" to help server admins find (and delete) unnecessary log files on their boxen. "By finding unwanted log files, logfinder informs system administrators when their servers are collecting personal data and gives them the opportunity to turn logging off if it isn't gathering information necessary for administering the system.""

Comments

I just made one, too

[Anonymous Coward]

locate log

Is a new tool really necessary?

[Lord Kano]

A competent admin will know that his/her boxen are collecting personal data. An ethical admin will get rid of any unneeded data.

LK

Can't subpeona what doesn't exist?

[PornMaster]

Is the point of this that law enforcement can't subpoena records that don't exist?

Re:Can't subpeona what doesn't exist?

[xC0000005]

I think so, but really it's just another step in an arms race. How long until we see court orders to collect this sort of information? Or forbidding the use of log destruction/filtering tools?

Re:Can't subpeona what doesn't exist?

[sporktoast]

If an admin is just using this tool to destroy potentially incriminating logs, then they are using it poorly. Like trying to pound a screw in with a hammer.

The use this has for an admin is to survey (or for the less experienced admin, to discover) what logs the system is currently, so that the admin can decide as a policy which logs should be active or not, and with what level of detail. The itch this tool scratches is that many systems as a default keep more logs than perhaps are necessary. A good admin will shut off whatever is deemed unnecessary, based on multiple criteria (security, system load, user/company privacy).

Forbidding the use of log destruction tools (rm?) is moot. Destroying evidence is illegal. Now, laws (or court orders) mandating a level of logging are a completely different matter.

Re:Can't subpeona what doesn't exist?

[jafiwam]

Destroying evidence is indeed illegal. However, before you are aware of, or have "reasonable belief" of a lawsuit or criminal investigation logs are not evidence yet and may be deleted freely.

I do exactly that with logs for my company. Once a month I clean out everything we don't need, including "email logs" and other stupid shit MS piles up in various places in the operating system. If/When the lawyers/cops come knocking, I can point to the policy and scheduled reminder and say "sorry, dont have that".

Logs are not the only place stuff resides and piles up, but it's one easy fix and keeps my servers and machines clear of unnecessary disk-space robbing files.

Re:Can't subpeona what doesn't exist?

[Threni]

> Once a month I clean out everything we don't need, including "email
> logs" and other stupid shit MS piles up in various places in the operating
> system. If/When the lawyers/cops come knocking, I can point to the policy
> and scheduled reminder and say "sorry, dont have that".

"...but if you'd have come yesterday you could have had 30 days worth".

I think I prefer the policy apparantly in place at www.cryptome.org, which is to delete all your logs every 24 hours.

I appreciate the effort but...

[garcia]

I would seriously hope that:

a) the sysadmins are competent enough to handle this themselves. I would think that a sysadmin would know how to use some sort of local file search.

b) the EFF understands that it's not always up to the sysadmins to determine the amount of time to keep logs that might be used against an individual.

Re:I appreciate the effort but...

[ObsessiveMathsFreak]

Most MCSE trained, NT sysadms don't really have a complete understanding of their servers and how they work. Most are just part time admins, doubling up as postmasters, network support and helldesk frontliners. A great many Windows server administrators are simply in fact, the company management accountant, who may never have recieved any computer training whatsoever! Many will not know where to begin looking for files without googling for the answer. This issue stems from the poor quality of the MCSE courses about, rather than from organisational difficulties with the NT servers themselves.

Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over /etc /var /usr /root /usr/X11 and even (I kid you not) /bin. The rather haphazard way different programs save their files about *nix systems can be a headache sometimes. It would be nice if someone would standardise the process. However, such a thing has been tried with disasterous results, i.e. the windows registry, so I guess I should be careful what I wish for!

In short, competant *nix admins will know most of the many location where their important daemons are storing logfiles. NT admins on the other hand, many not even know what daemons are running on the machine anymore, let alone where they store their log files!

P.S.
Hey wait! This is a python app. I guess NT admins will just have to keep on googling.

Re:I appreciate the effort but...

[Otter]

Unix admins versus MCSE's aside -- do you want your admin (on any platform) deleting files without understanding why they're there, just because some script from the EFF pointed them out to him?

Re:I appreciate the effort but...

[EnronHaliburton2004]

Admittedly NT logfiles are slightly more organised than *nix logfiles. Most will at least be under c:\Windows\system rather than spread over /etc /var /usr /root /usr/X11 and even (I kid you not) /bin. The rather haphazard way different programs save their files about *nix systems can be a headache sometimes. It would be nice if someone would standardise the process.

I don't think you understand *nix logging, or you've been working with poorly-designed systems.

Locations for log files has been pretty well standardized by Posix and the LSB. Logs generally go in /var/log (or /var/adm on older systems), or in $APPLICATION_ROOT/log. A sysadmin might write a log to /var or /root, but those are temporary logs.

Logfiles which end up in /etc, /bin, /usr or /usr/X11 is the result of poor or very old configuration.

Now, compare this to a Windows 2003 Server running Exchange 2003, where the log files in c:\windows c:\Windows\system c:\Windows\system\Logfiles c:\Windows\system\security
C:\Program Files\Exchsrvr\ C:\Program Files\Exchsrvr\MDBDATA C:\Program Files\Exchsrvr\mtdata . Many of the logfiles are not viewable with a text viewer. Some of the log files really aren't "Log files", but are "Transaction Logs", which is a different thing in my book.

Some of this makes sense, some of this does not. But I'm not a windows admin, and I didn't design this network here, so maybe this is the result of a poor configuration.

Re:I appreciate the effort but...

[Darth_Burrito]

the sysadmins are competent enough to handle this themselves.

I didn't bother to read a description of the tool, but there's nothing to say that a competent admin might not want something like this if it eased his burden. Also, there's the matter of incompetent admins. Many of us wear multiple hats. I do development, support, and administration on linux and windows for a small office, mostly by myself. Suffice to say, nobody can be perfect at everything. I'm always looking for tools that help me automate common tasks and, generally speaking, manage more with less.

Re:Thanks EFF!

[Anonymous Coward]

Pair it with TOR and hacking becomes untraceable! 2005 is going to be a great year.

Oh, yeah

[Otter]

God forbid professional sysadmins should be expected to understand how their services are configured and what files are being written. If I were a user on one of their systems, sendmail log files would be the least of my concern.

Re:Oh, yeah

[stephenbooth]

In an ideal world every system would be administered by a well trained and experienced system admin, or a trainee admin being mentored by one, who had plenty of time to investigate and maintain the machine. In practice most system admins are people in other roles (developers, DBAs, desktop support or even receptionists) who have been handed the task of managing half a dozen white box Wintel servers (with maybe a SCO or Linux box or even an aging Sun box in the mix) and probably a Netware server doing file and print, most were built and installed by someone one of the manager's knows or have been inherited third hand from another company. If they're lucky they get a training course where they'll learn a few of the GUI screens, more likely they'll be given a few dozen pages of handwritten notes (aka 'the manual') and told to go to the nearest Waterstones/Borders/Whatever and buy a book if they need more.

That was pretty much my first job. I had trained as a C programmer; then I found myself managing 70 desktops running various versions of Windows, a dozen or so White Box Intel based servers running Windows NT 3.51 and 4.0, a SCO OpenServer box, an Alpha running VMS, a 3 member VAX cluster running VMS and an RS6000 running AIX. All with no usable documentation or training. A little later they added in DBAing the Oracle databases and managing the network (a variety of devices from 3Com, Cisco and Bay), at the time I only knew a bit of SQL and wasn't really sure of the difference between a router and a switch. After spending a lot of money on books then a lot of time reading them (I didn't have web access at the time, when I did I started reading websites as well) I eventually learned what I needed to know.

This script is a separate issue. Inpractice I don't expect those sorts of admins to run it, they probably wouldn't know what to do with the information if they did. Where I think it would be useful is for the professional admin who suddenly inherits a bunch of machines (maybe they've moved companies or their company has merged with another). Put this script on them and run it for a few days then see what it turns up. No matter how wonderful and professional you are unless you built and installed a machine yourself and can guarantee that no-one else has ever had the root/admin password to a box you can't be 100% sure that there's not some process running somewhere that is quietly logging something somewhere. No-one who manages a non-trivial number of machines has time to check every machine to make sure that there are no new or unexpected services that have snuck in (and remember it's not something you could do once and then not again, you'd have to keep on doing it). That's why you need scripts that look for anything that could point to unexpected activity. Not just looking for anything that looks like a log on a box but also ports that shouldn't be open (I've lost count of the number of times I've found a box with port 25 open when I know I've disabled SMTP, only to find that someone has re-enabled it without telling me) or unexpected activity on a switch or firewall port. Not only do we have too many machines to manage but also users who delete files they shouldn't which then must be restored from backup, managers who constantly demand reports on system availabity stats and projects that we have to keep an eye on to make sure they don't run wild and break every standard we have.

Stephen

"Boxen"

[m_member]

Can I have a tool to locate and delete people who use the word 'boxen'? GPL preferably.

WTF is Boxen?

[Evil W1zard]

Am I behind the times in cool admin speak or was this simply boxes mispelled? In any case I could definitely see this being used for both good and bad.

Re:WTF is Boxen?

[MojoSF]

I also think of it as a nod to the old days when a room full of DEC VAX computers would be referred to as vaxen.

Re:WTF is Boxen?

[TheRaven64]

From the Jargon File:

boxen /bok'sn/ pl.n. [very common; by analogy with VAXen] Fanciful plural of box often encountered in the phrase `Unix boxen', used to {Unix">describe commodity {Unix hardware. The connotation is that any two Unix boxen are interchangeable.

And yes, you are seriously behind the the times. The oldest copy of the Jargon File I have is from the early '90s and that contains the word boxen.

It can be quite useful, since boxen are always computers, while boxes can be the packaging the computers came in.

Re:WTF is Boxen?

[Mr. Slippery]

Just trying to figure out why admins call them "boxen," not supporting the strange practice.

It's a running joke. See also this.

is this stupid?

[digitalgimpus]

I'm sorry, but this might just be the dumbest move yet they have made...

lots are crucial for many reasons:

1. Hacking attacks (how else do you track them, and prevent them)?
2. Abuse problems (spammers, credit card fraud)
3. aggregate statistics (what percentage of my customers are based in Europe?)

I can't see why someone would shoot themselves in the foot and use this.

Like log files are really intrusive anyway.

Re:is this stupid?

[Dasein]

Don't use it. Seriously. If you are comfortable with the level of logging you have, okay. There are others, who may be in the positions of not wanting log files that identify users because of the expense associated with discovery or complying with a subpeona.

Note, this is why large companies have email retention policies -- because having to do discovery or comply with a subpeona on email records going back years is expensive. So doing this type of thing isn't anything new or sinister.

Re:is this stupid?

[ducomputergeek]

I have to agree. I don't do much system administration work outside of our data-storage network these days, but even then I want to see what employees are putting on the 6TB system. Why? We've found porn before, against company policy, warezed games, against company policy, personal files, etc. before. For most random logs, they are archived every month and backup to optical media every quarter. But the storage system is supposed to be used to store completed video projects for the past year. Nothing else. The front office and bean counters have their own servers just for accounting information etc.

Back when I was doing more web-based server admistration logs were my friends. We could tell people trying to find an open mail port, hacking attempts, DDOS attempts, etc..

The EFF is becomming more like the ACLU in many regaurds. Had a very good reason for formation, but at some point went off the deepend. Privacy on the net is non-existant. Never was there, but somewhere along the way someone thought that the Internet was going to be this great tool of anaminity and I want to know who thought up that bloody idea. Because it was and always will be flase.

Redhat

[RalphLeon]

The last time I checked out redhat (about version 8 I rekon) they inluded this nice little utility called "logviewer". And, I though, wow a text viewer how novel, Linux doesnt have many text viewers.

So not only is this a text viewer, but it also finds all those logs hidden in /var/log/*, it must be hard to find anything in /var/log/* ...

Just as an example...

[PartialInfinity]

This is just EFF's way of reminding sysadmins to be vigilent about their log files, it's not meant to replace good administration habits.

Interesting Motive

[peterdaly]

My first thought was the main purpose of this would be to identify and eliminate "wasted" disk space. There are a bunch of logs that, without management, really just end up being wasted bits on your disk. Generally, that may be a useful utility, at least to me.

I was suprised to see the EFF seems to have a totally different motivation. It seems their real motivation is that the government can't demand logs that don't exists, or more specifically you can't get in trouble for not providing what you don't actually have.

Not sure what I think of that...

Re:Excellent

[ab384]

Just two observations: (the second of which is actually relevant to this parent)

1. It took me around 3 minutes to find out that this thread applies to POSIX-like systems only (ie. won't work on this winXP). The fact should really have been mentionned in the summary. I only say this because recently, some summaries seem to have been "hastily" written.

2. I am myself wary of huge, hidden log files that either winXP itself or other programs create. As the only user and sysadmin on this system and keen to minimise disk wastage, I would want to prune all logs regularly. Trouble is, they aren't all *.log files. So, how do I find them, short of going through every single program and investigating any logs it might or might not create?

interesting...

[Spider[DAC]]

Actually, it uses lsof and a few other niceties to locate open files that change over time, then scans them for presence of time/date stamps, mailaddress or other "log" activity.

So, no, its not just "locate log" that somone suggested, nor is it "find /var/log" either, but a bit more complex.

As for the comment about competent site-admin. This is a bit more than that too, its also about users and active software, peoples IRC logs, various ftp clients that clobber up and log passwords along with everything else in their config dir. And so on and so forth.

Re:interesting...

[scrotch]

Right. From their README:

"... We have created a program called logfinder as a sample means of locating files that might be logs on an existing system. logfinder uses regular expressions to find local files with "log-like" contents; you can customize those expressions if necessary to meet your needs. logfinder requires Python 2 or greater and finds logs in text files on a POSIX-like system. (It might also find some log-like data in binary files if the binary files represent that data in textual form.)

logfinder can, if the lsof program is installed and when run with
appropriate privileges, detect open files systemwide that grow larger over time. It can also search for text that may indicate logging activity within a given directory hierarchy, or systemwide. As we suggest above, a program like logfinder can find some, but not all, kinds of logging activity. For example, logfinder will generally not identify logs in binary (non-text) formats or logs kept inside
databases. Therefore, using a program like logfinder is usually a supplement to, not a replacement for, answering questions like those given above.

logfinder should be run as root. If logfinder is invoked without any arguments, it will examine open files systemwide to see whether they grow larger, and then indicate whether files that appear to be growing contain log-like text. (This requires lsof to be installed, and lsof's ability to report open files accurately may depend on your operating system. So far, we've had success with Linux and MacOS X, and some difficulty with FreeBSD and OpenBSD.)

If logfinder is given one or more directory names as arguments, it will search for log-like text in files in those directories. ..."

I haven't run it (and likely won't), but I'm curious whether it would ever flag stuff in /var/mail. It would be a lot of fun to have clueless admins cronning this and deleting mail every night.

Re:Thanks EFF!

[innerweb]

As is always, that which helps to protect the innocent can be used to protect the *evil*. The problem is the innocent do not know what is being done, and the *evil* are studying and learning to use and abuse. Nothing new there.

InnerWeb

Good work, but...

[IBeatUpNerds]

You could be treading in some dangerous territory. Let's say, for instance, as a sysadmin, you know one of your users has been accessing some machine they should not access for whatever reason (immoral, illegal, etc...). Well, you run this tool and uncover evidence to support that theory, then discuss with JaneUser and, out of the goodness of your heart, decide to remove the logs in question. All is well.

Two months later, "they" subpoena your logs to find no trace of evidence. Suspecting log-alteration, they subpoena the upstream providers logs and find correlating evidence that is mysteriously missing from your logs. So, JaneUser ends up getting in trouble, and the kind-hearted sysadmin gets slapped with evidence tampering.

I think, if you're going to carry out any activity that needs covering up, then you need to be more in tune with the circumstances rather than dealing with these sort of things after the fact. Or you could just avoid illegal activity all together...

Re:Good work, but...

[Secrity]

I am not a lawyer, the following is a general discussion, your mileage and your laws may vary. It is possible that some jurisdictions may have laws that require the retention of data, I know of no such requirement in the US. Did I mention that I am not a lawyer?

There is a difference between evidence tampering (illegal) and system administration (legal). If you remove data because it may be incriminating, you are tampering with evidence. It would also be illegal to delete data after you receive a subpoena or other legal demand. If you don't collect data or you have a policy to remove data after a certain period of time, you are administering a system. Another valid system administration policy is to remove log data when you fill a certain amount of disk space. You could also have a policy that says that you do not back-up certain logs. If you maintain logs or other data, a documented data retention policy is a Very Good Idea.

Log Retention Policy

[dRn-1]

I can see how useful logfinder could be/is. And how it along with relevant administration, can eleviate the possibility that your systems are harbouring evidence of criminal activity... Could they seize any systems they deem necessary? I certainly wouldn't want any of my systems seized because I don't have a log retention policy, and hence when they ask do you have logs for such and such @ such and such a date, a reply of I'm not sure wouldn't go down too well! In short and IMHO having a log retention policy is a good idea... think I might recommend one myself.

Re:neat

[e2d2]

Any tool could probably be used for evil. For instance I have a calendar on my wall. If I took it down and rolled it up, I could probably beat you half way to death with enough strong blows.

My experience with logfinder

[carpe_noctem]

Running openbsd 3.6 on x86:

tresor:src$ tar xfvz logfinder-0.1.tar.gz
logfinder-0.1
logfinder-0.1 /logfinder.py
logfinder-0.1/README
logfinder-0.1 /COPYING
tresor:src$ cd logfinder-0.1
tresor:logfinder-0.1$ sudo ./logfinder.py
Scanning for open files systemwide...
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'

(repeats several dozen more times...)

[Errno 2] No such file or directory: '(/dev/wd1a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/raid0a)'
[Errno 2] No such file or directory: '(/dev/raid0a)'
[Errno 2] No such file or directory: '(/dev/wd1a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/wd1a)'
Scanned sizes of 30 files.
Waiting for log activity; please allow time to elapse.
Press Enter to continue.
[Errno 2] No such file or directory: '(/dev/wd1a)'
[Errno 2] No such file or directory: '(/dev/wd0a)'
[Errno 2] No such file or directory: '(/dev/raid0a)'
tresor:logfinder-0.1$ cd ..
tresor:src$ rm -rf logfinder*

At least the EFF's lawyers are better than their programmers. ;)

Re:Only if you don't do backups.

[tchuladdiass]

That's why you use a tape backup tool that has an "Enron/Anderson" mode. Before writing any file to tape, encrypt it with a random key and store that key in you file locator database. Then when it is time to expire a particular file version, all you have to do is delete the index record. No need to wipe the tape. This is useful if you have data on a tape that expires at different times.
Also, for security, the random keys should then be passed through a public key encryption prior to being written to the database.

Re:Thanks EFF!

[flyingsquid]

As is always, that which helps to protect the innocent can be used to protect the *evil*."

Yeah, sure. You guys are so paranoid, next you're going to be telling me that the flesh-reanimation technology I've been working on can be used for evil too.

Could be moderately useful

[WarmBoota]

This tool could be moderately useful, especially in an environment where the administrator can't be expected to know all of the ins and outs of third-party add-ons.

I was once assigned to a dotcom that used a third-party component to allow for credit card transactions. What the admin didn't realize was the default configuration left the component in debug mode, placing all user-submitted credit card data in plain text files on the web server

We only found the log file accidentally while performing an unrelated search for files modified in the last 'n' days. The admin relied on the developers to configure the third-party component and the developers were relying on another set of consultants who didn't know or didn't care about the log files.

Re:Thanks EFF!

[theguyfromsaturn]

Yeah, sure. You guys are so paranoid, next you're going to be telling me that the flesh-reanimation technology I've been working on can be used for evil too.

Sure can. My wife just used your invention to bring back to life her dead mother. That is nothing but pure evil, believe you me.

anybody else have this lock up their system?

[rcpitt]

OK - I downloaded it, untar'd, and ran it as root on my Toshiba laptop (RH-9 2.4.20-31.9 kernel, 1G RAM) and the machine locked up. I had switched to another window and was looking at a log file while the program "thought" in the original window. I also had a number of remote ssh sessions open. The machine had been up for about 30 minutes today already.

This system is rock solid, in use for hours/day with the exact same mix of programs running constantly (evolution, mozilla, ssh/rxvt windows to external systems, etc.)

comments?

Plurals...

[Otto]

If somebody could find me a device that would hunt down annyoing people who complain about improper pluralization among tech nerds, then I'd definitely pay for that.

"Boxen" is fine. If the plural of ox is oxen, then pluralizing box as boxen seems perfectly acceptable to me. It also helps to understand that somebody is talking about a bunch of computers as opposed to a bunch of cardboard boxes. :-)

But I swear that the next person who tells me (in person) that virii is not correct is getting a punch in the face. I mean it.

Re:Just what I needed....

[0x000000]

Slashdot won't do you no good for new vulnerabilities. They are always a few days late with the news, and most system admins will have patched by then!