Bayesian Tail

Cool idea but may be dangerous by PhilippeT · 2004-12-29 05:51 · Score: 2, Insightful

This is a cool idea but I wouldn't want to use it on to filter logs on important systems... every line may be crucial.

Anyhow credits on a decent idea

--
A psychopath can't tell the difference between right and wrong. A sociopath knows the difference - he just doesn't care.

Re:Cool idea but may be dangerous by dougmc · 2004-12-29 06:08 · Score: 4, Insightful

This is a cool idea but I wouldn't want to use it on to filter logs on important systems... every line may be crucial.
Perhaps, but doesn't the same apply to your email? Every email may be crucial as well -- but if you miss a crucial email because it was buried in spam, isn't the effect the same as if it was caught by an overzealous spam filter?
Re:Cool idea but may be dangerous by bzebarth · 2004-12-29 06:18 · Score: 1, Insightful

I guess it depends on the kind of log file and what you are looking for. If you are talking about a database log file for instance, all errors lines may start with ERR or something and the log file may contain entries for every login and logout that you really don't care about. If you are only interested in certain types of entries it certainly seems useful.
Re:Cool idea but may be dangerous by cpuffer_hammer · 2004-12-29 06:18 · Score: 4, Insightful

Why not use it to colorize, Or to rebuild the logs in HTML.
Re:Cool idea but may be dangerous by flok · 2004-12-29 06:36 · Score: 3, Informative

If you need something that colorizes and/or does regular expression filtering, merging with other (log-)files, multiple windows, etc. etc. then maybe multitail might come in handy.

Initially I wanted to integrate btail into multitail, but multitail is bloated enough already :-)

--

www.vanheusden.com - home of Multitail, HTTPing, CoffeeSaint, EntropyBroker, rsstail, bsod, listener, nagcon, nagi
Re:Cool idea but may be dangerous by AndroidCat · 2004-12-29 06:53 · Score: 0, Offtopic

Or add sound effects. All movie and TV computers have sound effects! :) (Even if you don't need SFX for ZoneAlarm, the zip on my page has a fair collection of wav files.)

--
One line blog. I hear that they're called Twitters now.
Re:Cool idea but may be dangerous by Anonymous Coward · 2004-12-29 07:00 · Score: 0

thanks for the wavs, eh
Re:Cool idea but may be dangerous by OrangeSpyderMan · 2004-12-29 21:12 · Score: 1

You mean like, errr, 'grep'? I think the whole point of this tool is for situations where 'grep' can't easily filter out the rest. In my ideal world developers always tag log entries in such a way that getting the entries you want out is as easy as grepping the file ;-)

--
Try NetBSD... safe,straightforward,useful.
Re:Cool idea but may be dangerous by GlassHeart · 2004-12-30 09:51 · Score: 2, Insightful

The far more important difference is that we cannot control the generation of incoming email, which is why we are reduced to filtering as intelligently as possible.
Server logs are not the same at all. The administrator has some control over the logs that get generated, and the programmer has full control. There isn't supposed to be the equivalent of email spam at all, because useless messages should just be filtered or redirected at the source. Leaving everything at "verbose" and relying on filtering just doesn't seem like the right approach to the problem.
It is a cute idea, though, and probably applicable to some specific cases (no source code, etc).
Re:Cool idea but may be dangerous by lars_stefan_axelsson · 2004-12-30 23:48 · Score: 2, Informative

Why not use it to colorize, Or to rebuild the logs in HTML.

I published a paper, with GPL source code (you need Python etc) a few months back using visualisation (colorisation) to lend the user insight into the operation of a Bayesian classifier.

It actually works pretty well, and the idea could be applied to other uses of the Naive Bayesian classifier.

--
Stefan Axelsson

examples by rogueuk · 2004-12-29 05:52 · Score: 3, Interesting

Do you have any examples of what type of stuff it learns to filter and what it learns to show? The btail site is kind of lacking of what it outputs versus what it filters

Site getting sluggish already by Kiaser+Zohsay · 2004-12-29 05:56 · Score: 4, Informative

Blockquote from the readme.txt:

Step 1. compile & install

make install

Step 2. configure btail

Default configuration file:
db_bad = .btail_db_bad
db_good = .btail_db_good
db_conf = .btail_db_conf
logfile = /var/adm/messages

db_... are the database files which are filled by blearn. They are
used as reference when btail calculates if an event is bad or good.
logfile is the logfile which you want to monitor. As you see, one
needs a seperate configurationfile AND databases(!) for each file
to monitor.

Step 3. learn logging

blearn -g good_logging
blearn -b bad_logging

good_logging should contain events which are considered ok.
bad_logging should contain logging of events you want to see, e.g.
disk errors, invalid loggings, etc.

Step 3. use btail

btail

This will read the logfile defined in btail.conf and emit events
which are considered not-ok by the bayesian filter.

--- folkert@vanheusden.com

Still very preliminary at this point, but shows promise. Now, to build and try it out!

--
I am not your blowing wind, I am the lightning.

Re:Site getting sluggish already by Lars+T. · 2004-12-29 10:01 · Score: 1

Thanks. Now I know what to do with the source I can't get from the site.

--
Lars T.
To the guy who modded me down from perfect to terrible Karma - Apple haters still suck

Well, whereæs the story ? by noselasd · 2004-12-29 06:09 · Score: 1, Interesting

Did that story end a bit quick ?
Heck, I wanna know what the results are goddamnit. What made the thing so great.

Re:Well, whereæs the story ? by Anonymous Coward · 2004-12-29 08:20 · Score: 1, Funny

The missing bit was that his expectations had been that the code would burst into violet flame, emitting strange new quantum rays that would turn his bones into mercury.

When it merely turned a couple of dogs inside out, he knew the time had come to offer it as a Slashdot story.

my comment: by anonieuweling · 2004-12-29 06:13 · Score: 1

Go Folkert, go! ;-]

What I would like to see by bhima · 2004-12-29 06:19 · Score: 1, Interesting

The environment I work in is highly E-mail centric and I work on many projects. I would like to see some sort of Bayesian filtering employed to sort all of the e-mails I get into folders based on projects.

--
Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.

Re:What I would like to see by tonkdude · 2004-12-29 06:41 · Score: 4, Informative

I currently use CRM114 and on the mailing list, some one (Evan Prodromou) has created a program that does just this using the CRM114 language. It is called "Monkeyplexer" based on the idea that you could train a monkey to sort your mail box into folders.

If you pop over to the CRM114 site and search the general list archives for monkeyplexer to find the discussions about it.

Here is the last version announcement that I could find in my mailbox:

monkeyplexer is a tool for automatically sorting incoming email messages into appropriate folders. A new version of monkeyplexer, 0.7, is now available. http://bad.dynu.ca/~evan/monkeyplexer/monkeyplexer -0.7.tar.gz

This version includes the following changes:
You can specify which mailboxes to use, instead of which mailboxes to exclude. This can save some typing and some time at runtime, at the expense of dynamically updating the list. You can tell the monkeytrainer to only train messages that were received in the last few weeks, days, hours, minutes -- whatever. The monkeyplexer remembers which messages have been trained for which folders. If you train a message for a different folder, the monkeyplexer will automatically forget the first folder before training for the new one. Thanks to everyone who has installed monkeyplexer already. I hope this new version helps some people out. I find it easier and more accurate.

~ESP
Re:What I would like to see by rmohr02 · 2004-12-29 07:07 · Score: 2, Informative

POPFile is exactly what you're looking for.
Re:What I would like to see by Anonymous Coward · 2004-12-29 08:11 · Score: 0

Try a project management site like Basecamp (www.basecamphq.com).

Email is dead, and for project management it SUCKS.
Re:What I would like to see by Anonymous Coward · 2004-12-29 10:19 · Score: 0

While I haven't done this you can probably use http://bogofilter.org to do this by using different wordlists for each of your projects.

So have a ~/.bogofilter-spam for spam filtering, ~/.bogofilter-gui for your GUI project and ~/.bogofilter-database for your database one.

You'll then treat email scoring as a spam against the -gui list to be a GUI related email.

If this were Trek... by AndroidCat · 2004-12-29 06:24 · Score: 5, Insightful

01:37 Overheat in plasma injector #1.
01:56 Plasma injector #1 offline, switching to #2 backup.
02:23 Overheat in plasma injector #2.
02:44 Failure to shutdown plasma injector #2.
02:58 Overheat in reactor core.
03:20 Containment weakening.
03:25 Containment weakening.
03:30 Containment weakening.
03:35 Five minutes to containment failure.
03:40 FIVE SECONDS TO WARP CORE BREACH!!!

Better be careful to train the filter about those warnings that don't happen very often, but when they do, you really want to know about them.

--
One line blog. I hear that they're called Twitters now.

Re:If this were Trek... by aoteoroa · 2004-12-29 07:21 · Score: 2, Interesting

True. But if the Star Trek error log resembled real life then it might look more like:
01:37 [error] Overheat in plasma injector #1.
01:37 [warning] Cargo bay door 2 is open.
01:38 [warning] Cargo bay door 2 is open.
01:38 [warning] Oxegen sensor on deck 2 not responding.
01:39 [warning] Cargo bay door 2 is open.
01:40 [warning] Cargo bay door 2 is open.
01:41 [warning] Oxegen sensor on deck 2 not responding.
01:56 [error] Plasma injector #1 offline, switching to #2 backup.

In other words real interesting errors in the logs can get hidden by a bunch of trivial log entries.

I use tail all the time when developing php applications. PHP logs errors to the apache log file so I type:
tail -f /var/log/apache/mysite.com-error.log
To track changes to the apache logs as I test the php pages.

But the truth of the matter is that I am only interested in php errors, and not broken links, and missing images. So if I can train btail to pay attention to php errors like:
[Wed Dec 29 10:58:04 2004] [error] PHP Fatal error: Call to undefined function: badFunction() in /home/aoteoroa/www/pages/info.php on line 1
and ignore file not found errors like:
[Wed Dec 29 11:16:22 2004] [error] [client 192.168.0.2] File does not exist: /home/aoteoroa/www/pages/info-over.gif
it would make my job just a little bit easier.
Re:If this were Trek... by metalhed77 · 2004-12-29 07:58 · Score: 1

If I were you I'd just write a special script to do that, probably far less hassle and more accurate.

--
Photos.
Re:If this were Trek... by sahtanax · 2004-12-29 08:06 · Score: 1

tail -f /foo/bar/mysite.com-error.log | grep -i php

... will show only the php errors. :-)
Re:If this were Trek... by ThisNukes4u · 2004-12-29 08:07 · Score: 1

As others have said, it would be much easier to pipe that through grep or a perl script first instead.

--
thisnukes4u.net
Re:If this were Trek... by AndroidCat · 2004-12-29 08:10 · Score: 1

Oh sure, I can definitely see where it would be useful for highlighting the important stuff in a sea of logs, and easier for crafting a general solution rather than a pile of rules and regexs.
For my firewall sound effects program, I basically tail the ZoneAlarm logs, and play a selectable sound effect depending on the port/type. It's cute and even useful for detecting patterns (if you don't mind the noise), but I'm thinking about if Bayesian filtering could be applied to a real security report. It might be tricky since importance depends on patterns and trends more than single events.
BTW, One thing they really need on Trek is a cron'ed scan of who's where. It'd be nice if the computer would mention "Lt Spooge has disappeared from the ship" before someone has to ask where he is.

--
One line blog. I hear that they're called Twitters now.
Re:If this were Trek... by RAMMS+EIN · 2004-12-29 08:45 · Score: 1

My guess is that you are probably exactly interested in the uncommon log messages.

That said, I don't buy logfile filtering until I see it works. Sometimes you are interested in messages of one kind, sometimes in messages of another kind. I still think that fixed pattern matching can do the job better. Of course, that's what many people feel about spam filtering.

--
Please correct me if I got my facts wrong.
Re:If this were Trek... by Wolfbone · 2004-12-29 10:03 · Score: 1

As has been said: it's easiest just to write ad hoc filters for this sort of thing. This is what I've been using since my wmail stopped working properly:

tail -F /var/log/messages | awk '/from=/,"\033[1;32m&\033[0m",$7); system("aplay sounds/newmail.wav&")}; {sub(/.*/,"\033[37m&",$1); sub(/.*/,"\033[33m&\033[0m",$3); print}'

I use similar rules for alerts about SSH break-in attempts, mail relay probes and machine check exceptions. I know there are all sorts of sophisticated log analyzer and colorizer tools around but I've never needed them.
Re:If this were Trek... by Wolfbone · 2004-12-29 10:10 · Score: 1

Sorry - that didn't come out right because of the angle brackets in it:
tail -F /var/log/messages | awk '/from=</ {sub(/<.*>/,"\033[1;32m&\033[0m",$7); system("aplay sounds/newmail.wav&")}; {sub(/.*/,"\033[37m&",$1); sub(/.*/,"\033[33m&\033[0m",$3); print}'
Re:If this were Trek... by Anonymous Coward · 2004-12-29 10:29 · Score: 0

Actually, I strongly recommend logging to a Postgresql server. That way you can do a nice SELECT on the logs. Use Samhain on each box if you don't trust the network protocol of syslog (and you shouldn't).
Re:If this were Trek... by Anonymous Coward · 2004-12-29 12:43 · Score: 0

Ummm...

> tail -f /var/log/apache/mysite.com-error.log | grep -i php
Re:If this were Trek... by ibbey · 2004-12-29 14:36 · Score: 1

That said, I don't buy logfile filtering until I see it works. Sometimes you are interested in messages of one kind, sometimes in messages of another kind. I still think that fixed pattern matching can do the job better. Of course, that's what many people feel about spam filtering.

I think you're misinterpreting what the tool is meant for. Often, when you are looking at the logs, you are looking for something in particular. In those cases, as you suggest, grep is probably the best tool for the job. But, as far as I can tell, btail is not meant for that. Instead, btail is intended as a tool to allow periodic monitoring of the system. It filters out all of the routine stuff, but passes through anything important or out of the ordinary. For this purpose, grep is pretty much useless. Like in most parts of life, choose the best tool for the job. Sometimes it will be grep, but for its intended purpose, btail will make things much easier.
Re:If this were Trek... by Nevyn · 2005-01-02 17:44 · Score: 1

I do this kind of thing all the time, allow me to share...

tail -f foo.log | grep "PHP Fatal error"

--
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
Re:If this were Trek... by Anonymous Coward · 2005-01-04 07:09 · Score: 0

I strongly recommend logging to a Postgresql server. That way you can do a nice SELECT on the logs.
Because only Postgresql has SELECT.

Bayesian is good for almost everything by Ki+Master+George · 2004-12-29 06:59 · Score: 4, Interesting

Bayesian filtering could be used for lots of things outside of spam. One example could possibly be Wikis, determining spam from ham modifications (well, yes, it is spam here). I've had some other ideas that involve Bayesian, but they've escaped me for the moment.

--
Before you walk a mile in someone's shoes, you should insult them so you know how they are and what they're doing.

Re:Bayesian is good for almost everything by dasunt · 2004-12-29 20:33 · Score: 2, Interesting
Bayesian filtering could be used for lots of things outside of spam. One example could possibly be Wikis, determining spam from ham modifications (well, yes, it is spam here). I've had some other ideas that involve Bayesian, but they've escaped me for the moment.
- Email sorting filters: imagine a baynesian setup that can decide if a new mail should be sorted into "work", "friends", "ebay", "amazon", "project", etc.
- Interest filters: Run slashdot stories and comments through your own trained baynesian sorting system and filter out the stories you probably don't want to see. Do the same for news.google.com, cnn, or usenet.
- Music sorter: Can Baynesian filters be taught to understand music (pitch, amplitude, etc?) If so, can they sort on it? If I see a song playing in xmms, can I use my nifty baynesian_sort plugin to play more songs that sound like that for the rest of the day? Consider tying it in to the 'next' button -- if I don't play a song completely, I probably don't want to hear songs like that for the next few days.
- IM secretary: Add a 'secretary' feature to your IM client. When you enable it, it will show you only messages that it thinks you want to see.
There are a ton of possibilities available.
Re:Bayesian is good for almost everything by Phil+John · 2005-01-03 01:05 · Score: 1

I always thought a bayesian (adult) web-site content filter would be a good idea, a la netnanny but without a canned list of "bad" url's.

In fact, I may just go and make a ff extension that does just that, hmmm, mebbe call it "NNSFW"?

--
I am NaN

Re:This code belongs on by rmohr02 · 2004-12-29 07:14 · Score: 2, Insightful

Give him a break--it is the first release, and I doubt he's had much feedback yet.

Re:This code belongs on by flok · 2004-12-29 07:18 · Score: 1

Pfff.
Did you see the version number?
There are tons of things that need to be redone.
For example the commandline parameters. No, not with a switch statement (or even my if-construction) but with getopt() etc.

--

www.vanheusden.com - home of Multitail, HTTPing, CoffeeSaint, EntropyBroker, rsstail, bsod, listener, nagcon, nagi

Well, no it doesn't ... by Chromodromic · 2004-12-29 07:20 · Score: 4, Insightful

All due respect, you're being a bit hard on the guy. He's not doing badly here.

The [brackets] used in the usage message are standard in the Unix world for specifying an optional or default argument. Just look at any man page. So that, actually, is pretty straightforward. The name of the default config file would likely also be spelled out in the man page, which I would expect, so that's not confusing.

As for changing the if construct into a switch, well, I'm trusting the accuracy of your excerpt, but I didn't find his code to be very difficult to read, to be honest, and certainly not a candidate for DailyWTF, which typically contains laughably horrible code.

As far as other code may go, the guy states that this is in a nascent stage, so jumping on his source files seems like a bit of an easy shot :|

--
Chr0m0Dr0m!C

I concur by nietsch · 2004-12-29 07:41 · Score: 1

Go Folkert! Your site is still standing, so lets wait and see what happens when your story hits the frontpage ;-)

--
This space is intentionally staring blankly at you

Re:This code belongs on by Anonymous Coward · 2004-12-29 08:23 · Score: 0

Actually a better design would be to not use ANY default config. What's the point? Reduce code bloat and useless choices. If the admin wants a "default" config, let him make a shell script that exec's the program.

if (argc == 2) conf_file = argv[1]; else die("USAGE: btail <configfile>");

In fact I bet the program could be rewritten so it doesn't need a config file at all. Think how much time could be saved if people didn't waste time writing config file parsers.

Re:This code belongs on by Hard_Code · 2004-12-29 08:32 · Score: 4, Insightful

That aside, your code would be easier to read (slashcode's broken formatting nonwithstanding) if you used a switch construct.

Speak for yourself. Given that the switch cases are all mutually exclusive, and disregarding the default case, there are only 2 paths, switch is more obfuscatory than clarifying in my opinion.

--

It's 10 PM. Do you know if you're un-American?

Reinvent the Wheel Much? by runswithd6s · 2004-12-29 09:28 · Score: 4, Informative

(Stage Left) Enters the Controllable Regex Mutilator, crm114, with a noticable strut. He's been there, done that.

CRM114 is a system to examine incoming e-mail, system log streams, data files or other data streams, and to sort, filter, or alter the incoming files or data streams according to the user's wildest desires. Criteria for categorization of data can be by satisfaction of regexes, by sparse binary polynomial matching with a Bayesian Chain Rule evaluator, a Hidden Markov Model, or by other means. Accuracy of the SBPH/BCR classifier has been seen in excess of 99 per cent, for 1/4 megabyte of learning text. In other words, CRM114 learns, and it learns fast .

--
assert(expired(knowledge)); /* core dump */

Why learning with supervision? by MoobY · 2004-12-29 10:39 · Score: 2, Interesting

I thought this app was learning everything was in the log, and then only showed the new out-of-the-ordinary log entries that didn't quite fit in with the rest. This would allow to filter out freak events from the log and show them to the user. How different would such an app be from the proposed btail? And how confident would you be about such an unsupervised log analyzer?

--
--- Sigmentation Fault - Comments Dumped

Bayesian is good for almost everything-Dessert. by Anonymous Coward · 2004-12-29 12:22 · Score: 1, Interesting

"I've had some other ideas that involve Bayesian, but they've escaped me for the moment."

Recovering the Slashdot lost since 2000, by eliminating most (-1) material e.g.GNAA,FP,etc. Eliminating the human biasis in the moderation system (Since client-side moderation is out). Tagging interesting material (A Baysian agent).

Re:Sure... by I_Love_Pocky! · 2004-12-29 12:27 · Score: 3, Funny

Why would you run this on an MS system? The critical errors are so common that btail would discard them with the rest of the log file.

new pr0n! by robdeadtech · 2004-12-29 14:22 · Score: 1

do I get a discount if I already have a subscription to Black Tail?

--
Heil Sig! -Rob

Hey! by Jeremiah+Cornelius · 2004-12-29 17:34 · Score: 2, Funny

Now even geeks can get a little tail!

--
"Flyin' in just a sweet place,
Never been known to fail..."

This is great by Matt+Clare · 2004-12-29 17:36 · Score: 0

Why is everony giving this guy a hard time. Just wait until Apple wants him to work on the Panther server Admin tool. Think about all those Grandmas using a new G5 in some new complex in Florida. A tool like this might prevent a huge DDoS! It might also be good for people usin Linspire, but I doubt Linspire is in use on that many high speed networks.

Yes, the first poster was right. People who should read every line of thier logs should read every line of their logs.

--
.\.\att Clare

Use grep by ufnoise · 2004-12-29 18:10 · Score: 0, Offtopic

grep ssh /var/log/messages > ssh.log

grep -v ssh /var/log/messages > notssh.log

Also, my current installation pretty much breaks things down into /var/log/maillog, /var/log/secure, and /var/log/messages.

The reporting from the logwatch package also seems to be pretty good.

Here's how to make this a lot more useful by Julian+Morrison · 2004-12-29 18:21 · Score: 3, Interesting

Step 1: Allow the option to automatically discover and load canned training packages, eg: a directory under /etc. Make it automatically pick the right training file to use when called with a logfile (so eg: btail httpd.conf knows to look for the training for httpd.conf files).

Step 2: Include btail with major distros

Step 3: Any package for an app that generates logs can come with a ready-made canned training package, which gets dropped into the /etc directory.

That way, you could apt-get a package, start btail-ing its logfiles immediately without the need to tediously train the filter first. Training would still be possible, to personalise the filter.

Re:Here's how to make this a lot more useful by Anonymous Coward · 2004-12-30 08:34 · Score: 0

And then the malicious types will know what kind of log messages the (non-)admins are likely to ignore.

I know, it's stretching, but I couldn't resist.

Nor for Me by schestowitz · 2004-12-29 20:58 · Score: 1, Insightful

People who monitor log files know best where to look and what to ignore. It is better to incorporate filtering into the application that generates the logs.

--
My Linux - (L)ove (I)s (N)ever (U)tterly eXPensive

Re:This code belongs on by Juanvaldes · 2004-12-29 21:45 · Score: 1

If anyone else is interested in what getopt is gnu usage and example

throw out the baby with the bathwater, will you? by Roman_(ajvvs) · 2004-12-30 05:06 · Score: 1

I'm don't understand how your suggestion fits in with your initial statement, or as a comment on the usefulness of a bayesian log filter. It's true that people know best. But after scrolling through the same "Operation completed with errors" line time and time again, the minute effort required adds up. even a simple automated filter can assist, which this person has implemented. It is better to incorporate filtering into the application that generates the logs.

That's akin to only filling a dictionary with words that you use. Most people only use 5000-10,000 words. A dictionary contains a multitude more.
In most cases, I'll be looking to spell something mundane (to check it's not "mundain"). Occasionally I'd like to spell something a lot more difficult, eg: "disenfranchisement". (I wish firefox had a spellchecker...) I'd be out of luck, if some filter decided not to include it in the book... I hope you see my point.

--
click-clack, front and back. I'm not moving this car otherwise.

Re:This code belongs on by Anonymous Coward · 2004-12-30 05:10 · Score: 0

> Actually a better design would be to not use ANY default config. What's the point? Reduce [crunch] useless choices

You, my friend, have understood the the art of software design.

Too bad most people I work with have not.

Bayesian by inertia187 · 2004-12-30 06:00 · Score: 2, Interesting

Bayesian tail might be neat. I like the idea of broadening the use, but I'd much rather see bayesian filters used on my in-box for more than just spam. I envision a filter that would sort out e-mails based on subject matter. This would have the net effect of improving the filter technology because it's trying to sort e-mails you actually want to look at.

We all know that if the filter makes a mistake and hides a message in the Spam box, and chances are you'll might miss many of them, another the chance to train the filter has been lost. But if an e-mail that was intended to land in the Irate Customer box, instead lands in the Clueless Customer box, the likelihood of noticing it is much greater.

--
A programmer is a machine for converting coffee into code.

Spellcheck in Firefox by schestowitz · 2004-12-30 13:45 · Score: 1

Yes, I contradicted myself somehow. You need to look at the two parts of my reply separately. About Firefox, try what I do and use kedit or the like to run a spellcheker (ALT+T+S). It takes 2 seconds to copy and paste text and I invoke kedit using CTRL+ALT+E (xbindkeys).

--
My Linux - (L)ove (I)s (N)ever (U)tterly eXPensive

Bayesian AIM bot by duncangough · 2004-12-30 21:31 · Score: 3, Interesting

I love Bayes stuff - and there's a very nice Python module written by divmod.

I was playing around with AIML to cobble together a basic chat bot when I realised that I could use a Bayesian parser to radically cut down the amount of AIML that I needed to write. AIML is an XML style of chat bot repsonses, it's clever in that it's highly recursive but the downside is that you need to create a rule for every eventuality.

By adding in a bit of Bayesian guessing before the AIML parser got it hands on the conversation, I'm able to keep the AIML files very focused and give the chat bot a bit more sparkle - you don't have to train him about everything. After a while he realised that 'yo', 'hi' and 'hello' are all the same thing, so he just guesses that you're saying hello and pulls out the correct response from the AIML file (rather than creating an AIML rule to deal with all the variations on 'hello').

If you're interested I'd strongly recommend installing GrokitBot. You can get the source and a bit more explanation at my site, Suttree.com

Playaholics : Free Online Games

--
Suttree, a weblog about casual games development

63 comments