Bayesian Tail

flok writes "We all know anti-spam-software using Bayesian filtering. The results with these are amazingly good. So that made me thinking: why not create a tool which monitors logfiles and determines using a Bayesian filter what events to display and what not? That's why I created btail. Btail is just that: it monitors a logfile and filters it with a Bayesian filter. The results are above my own expectations!"

Comments

examples

[rogueuk]

Do you have any examples of what type of stuff it learns to filter and what it learns to show? The btail site is kind of lacking of what it outputs versus what it filters

Site getting sluggish already

[Kiaser Zohsay]

Blockquote from the readme.txt:

Step 1. compile & install

make install

Step 2. configure btail

Default configuration file:
db_bad = .btail_db_bad
db_good = .btail_db_good
db_conf = .btail_db_conf
logfile = /var/adm/messages

db_... are the database files which are filled by blearn. They are
used as reference when btail calculates if an event is bad or good.
logfile is the logfile which you want to monitor. As you see, one
needs a seperate configurationfile AND databases(!) for each file
to monitor.

Step 3. learn logging

blearn -g good_logging
blearn -b bad_logging

good_logging should contain events which are considered ok.
bad_logging should contain logging of events you want to see, e.g.
disk errors, invalid loggings, etc.

Step 3. use btail

btail

This will read the logfile defined in btail.conf and emit events
which are considered not-ok by the bayesian filter.

--- folkert@vanheusden.com

Still very preliminary at this point, but shows promise. Now, to build and try it out!

Re:Cool idea but may be dangerous

[dougmc]

This is a cool idea but I wouldn't want to use it on to filter logs on important systems... every line may be crucial.

Perhaps, but doesn't the same apply to your email? Every email may be crucial as well -- but if you miss a crucial email because it was buried in spam, isn't the effect the same as if it was caught by an overzealous spam filter?

Re:Cool idea but may be dangerous

[cpuffer_hammer]

Why not use it to colorize, Or to rebuild the logs in HTML.

If this were Trek...

[AndroidCat]

01:37 Overheat in plasma injector #1.
01:56 Plasma injector #1 offline, switching to #2 backup.
02:23 Overheat in plasma injector #2.
02:44 Failure to shutdown plasma injector #2.
02:58 Overheat in reactor core.
03:20 Containment weakening.
03:25 Containment weakening.
03:30 Containment weakening.
03:35 Five minutes to containment failure.
03:40 FIVE SECONDS TO WARP CORE BREACH!!!

Better be careful to train the filter about those warnings that don't happen very often, but when they do, you really want to know about them.

Re:Cool idea but may be dangerous

[flok]

If you need something that colorizes and/or does regular expression filtering, merging with other (log-)files, multiple windows, etc. etc. then maybe multitail might come in handy.

Initially I wanted to integrate btail into multitail, but multitail is bloated enough already :-)

Re:What I would like to see

[tonkdude]

I currently use CRM114 and on the mailing list, some one (Evan Prodromou) has created a program that does just this using the CRM114 language. It is called "Monkeyplexer" based on the idea that you could train a monkey to sort your mail box into folders.

If you pop over to the CRM114 site and search the general list archives for monkeyplexer to find the discussions about it.

Here is the last version announcement that I could find in my mailbox:

monkeyplexer is a tool for automatically sorting incoming email messages into appropriate folders. A new version of monkeyplexer, 0.7, is now available. http://bad.dynu.ca/~evan/monkeyplexer/monkeyplexer -0.7.tar.gz

This version includes the following changes:
You can specify which mailboxes to use, instead of which mailboxes to exclude. This can save some typing and some time at runtime, at the expense of dynamically updating the list. You can tell the monkeytrainer to only train messages that were received in the last few weeks, days, hours, minutes -- whatever. The monkeyplexer remembers which messages have been trained for which folders. If you train a message for a different folder, the monkeyplexer will automatically forget the first folder before training for the new one. Thanks to everyone who has installed monkeyplexer already. I hope this new version helps some people out. I find it easier and more accurate.

~ESP

Bayesian is good for almost everything

[Ki Master George]

Bayesian filtering could be used for lots of things outside of spam. One example could possibly be Wikis, determining spam from ham modifications (well, yes, it is spam here). I've had some other ideas that involve Bayesian, but they've escaped me for the moment.

Well, no it doesn't ...

[Chromodromic]

All due respect, you're being a bit hard on the guy. He's not doing badly here.

The [brackets] used in the usage message are standard in the Unix world for specifying an optional or default argument. Just look at any man page. So that, actually, is pretty straightforward. The name of the default config file would likely also be spelled out in the man page, which I would expect, so that's not confusing.

As for changing the if construct into a switch, well, I'm trusting the accuracy of your excerpt, but I didn't find his code to be very difficult to read, to be honest, and certainly not a candidate for DailyWTF, which typically contains laughably horrible code.

As far as other code may go, the guy states that this is in a nascent stage, so jumping on his source files seems like a bit of an easy shot :|

Re:This code belongs on

[Hard_Code]

That aside, your code would be easier to read (slashcode's broken formatting nonwithstanding) if you used a switch construct.

Speak for yourself. Given that the switch cases are all mutually exclusive, and disregarding the default case, there are only 2 paths, switch is more obfuscatory than clarifying in my opinion.

Reinvent the Wheel Much?

[runswithd6s]

(Stage Left) Enters the Controllable Regex Mutilator, crm114, with a noticable strut. He's been there, done that.

CRM114 is a system to examine incoming e-mail, system log streams, data files or other data streams, and to sort, filter, or alter the incoming files or data streams according to the user's wildest desires. Criteria for categorization of data can be by satisfaction of regexes, by sparse binary polynomial matching with a Bayesian Chain Rule evaluator, a Hidden Markov Model, or by other means. Accuracy of the SBPH/BCR classifier has been seen in excess of 99 per cent, for 1/4 megabyte of learning text. In other words, CRM114 learns, and it learns fast .

Re:Sure...

[I_Love_Pocky!]

Why would you run this on an MS system? The critical errors are so common that btail would discard them with the rest of the log file.

Here's how to make this a lot more useful

[Julian Morrison]

Step 1: Allow the option to automatically discover and load canned training packages, eg: a directory under /etc. Make it automatically pick the right training file to use when called with a logfile (so eg: btail httpd.conf knows to look for the training for httpd.conf files).

Step 2: Include btail with major distros

Step 3: Any package for an app that generates logs can come with a ready-made canned training package, which gets dropped into the /etc directory.

That way, you could apt-get a package, start btail-ing its logfiles immediately without the need to tediously train the filter first. Training would still be possible, to personalise the filter.

Bayesian AIM bot

[duncangough]

I love Bayes stuff - and there's a very nice Python module written by divmod.

I was playing around with AIML to cobble together a basic chat bot when I realised that I could use a Bayesian parser to radically cut down the amount of AIML that I needed to write. AIML is an XML style of chat bot repsonses, it's clever in that it's highly recursive but the downside is that you need to create a rule for every eventuality.

By adding in a bit of Bayesian guessing before the AIML parser got it hands on the conversation, I'm able to keep the AIML files very focused and give the chat bot a bit more sparkle - you don't have to train him about everything. After a while he realised that 'yo', 'hi' and 'hello' are all the same thing, so he just guesses that you're saying hello and pulls out the correct response from the AIML file (rather than creating an AIML rule to deal with all the variations on 'hello').

If you're interested I'd strongly recommend installing GrokitBot. You can get the source and a bit more explanation at my site, Suttree.com

Playaholics : Free Online Games