Slashdot Mirror
Text-Mining Your E-mail
Misha writes "There have been a number of weeks/months in anyone's life that called for a better organization of your Inbox. filtering and folders work, but it'd be nice to have an text-mining tool running in the background that categorized incoming messages by topic as they arrive. It's nice to see that besides NLP research, there are some great algorithmic advances being done, as seen in this paper. Perhaps even one of them Perl monkeys will quickly hack such a background tool." Note: it's a PostScript file.
Here's to the researchers. I would like:
* An email box that lets me extract the threads with my friends.
* An email box that automatically ages the files effectively archiving them. Some of my mail folders/files are huge now and it takes too long to append them when new mail arrives.
Yes, I realize I should get off my butt and do this, but it's faster to post on slashdot.
Here's a link to a terribly useful site for converting your postscripts and word docs into pdf or jpeg.
I was thinking of how to intentionally fail my drug test... It would make a good memoir story someday.
I'm sure I'm not alone in saying that having a good history of well filtered incoming, and especially just about all of my Outgoing (Outbox) available for searching. My Outbox has been a lifesaver several times when someone claims that they didn't have that (electronic) discussion with me. It's great to quote "in a message sent... ...I asked you to...".
It automagically does full text indexing of all specified databases. To it, your Inbox is just another database.
"History doesn't repeat itself, but it does rhyme." Mark Twain
This would be an awesome tool to block spam. If this program could look at the text of an email message and determine that it is a solicitation of some kind and then drop it into an email "pit" (you know, a folder mapped to /dev/null), that would make my life a LOT easier...
The Dopester
"Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
Personally, I'd prefer that I simply get less email. The fact that we need NLP tools to pre-screen our email for us just shows how information-overloaded our society has become. What I really need is a tool at the sender's end that can pre-screen my email and tell the sender "Don't send this. He just doesn't care!"
Now we all now that most email is delivered promptly by gremlins, but gremlins are hungry and will eat a few bytes here and there.
They also leave waste in the form of spam.
So, I propose that we turn to gnomes to deliver the mail instead, as they are much cleaner, and can be satiated by attaching a file like 'Hamburger.txt'.
"I only speak the truth"
Karma: null(Mostly affected by an unassigned variable)
Every time you sign up for some mailing list or discussion group, create a new e-mail account or alias for just those mailings. Bam, it's automatically sorted out by itself with extreme ease. If you have limited bandwith (or are checking, say, on your palm) sometimes, just check your important addresses frequently, and reserve your mailing lists for a once-per-day check.
If some site asks for your e-mail address to download a piece of software, or to register, make up a new alias and give that to them. If you start getting tons of crap at that address, you can just remove that alias, and they get it all bounced back in their stupid spamming faces.
Give one address to your cow-orkers just for work stuff. Give a different one to your Mom and other techno-nots that blocks all attachments. Give another one to your friends with brains that goes unfiltered. For people you don't want to talk to, give them the address of an autoresponder tied to Eliza.
Be a *Happy Camper* and let your addresses be *Bubbles* and you be just *You*.
... "Give me a woman who loves beer and I will conquer the w
Not to mention that a PDF would be 10x the size. I have no idea why Mac and Windows OS's are so baffled by postscript...half the printer drivers have to deal with it already, why not just bundle a damn interpreter with the OS and have a minimal frontend on it for screen viewing?
Karma: Good (despite my invention of the Karma: sig)
Damn you, Admiral Hopper! I've got a huge stack of COBOL listings on my desk that I've got to translate to, of all things, vbscript (damn you, Bill Gates!)
This is a little more in depth than just matching strings in headers. This is about determining the topic of and email.
Spencer
Spencer Ogden
It's more general than e-mail, but in the wearable computing community, there's a little application called Remembrance Agent, written by Bradley Rhodes that many folks use. In terms of stand-alone UI, it's still quite primitive, but that's because it was built around dynamic hooks into Emacs.
I've been playing around with some Java-based wrapper code, to wrap the ra-retrieve executable in a Server and allow clients to access the data via sockets. I have a Java-based client coded up that hooks into the System clipboard, but it's still in alpha-mode. All GPL'd of course, but needs a little time to mature. It's a proof-of-concept, work in progress. :-)
Check out Brad's site for more insight into the work he did and is doing.
--The more you know, the less you know.
Postscript has the best reproduction accuracy for the file size. Assuming it has any kind of figures or equations, the only other reasonable alternatives are dvi and pdf. I've never seen dvi files rendered in a decent amount of time, and pdf is too fat, esp. for a paper linked to by slashdot. :)
Karma: Good (despite my invention of the Karma: sig)
my $pr0n = "adult";
...
my $spam = "viagra";
my $urgent = "penis enlargement";
open (INBOX,/home/mail) or die "Damn! No fun for me:$!\n";
@list = readdir(INBOX);
foreach $ (@list) {
if (-f $spam) {
my $status = unlink($spam);
}
if (-f $pr0n) {
my @MUST_SEE = $pr0n;
next;
}
if (-f $viagra) {
my @RAINY_DAY = $viagra;
next;
}
}
# or something like that
OK, it's not a piece of Linux software, but it is a beautful idea:
http://www.creo.com/sixdegrees/
668: Neighbour of the Beast
Vintage computer games and RPG books available. Email me if you're interested.
Hoo boy. Here we go again. When are you kids going to get it straight?
- Choosing not to listen to somebody is *not* censorship.
- Throwing your mail away before you open it is *not* censorship.
- Choosing not to relay somebody's spam is *not* censorship.
- Choosing not to broadcast somebody's TV program, even if you own a TV network, is *not* censorship.
- Telling a movie producer you won't distribute his/her movie unless he/she makes cuts or changes to the subject matter is *not* censorship.
- Rallying your church group together to burn books is *not* censorship.
- Refusing to sell certain magazines or newspapers, if you own a newsstand, is *not* censorship.
The only way somebody can be truly "censored" is when there is no legal means for that person to get his/her speech/art/etc. produced and disseminated to the pubic. Generally speaking, the only body with that type of power is the government -- because they make the laws.
Everything else is merely an inconvenience. It may piss you off, sure, and you may wish things were different. But you can't force people to support you, encourage you, or fund you if they just don't want to. For example, people in this country (the US) *do* have a right to decide what material constitutes pornography, relative to their local community standards -- and if you don't like it, you are within your rights to move to another town.
"No censorship" does not mean being forced to look at every piece of crap that somebody wants to throw in your face, and god help us if it did.
Breakfast served all day!
Now I can automatically filter my barely-legal porn spam from my anime porn spam. Lets hear it for technology =)
Somewhat to my astonishment when I clicked on the link up popped a box asking me to confirm Postscript Renderer options! I had no idea that I had anything on this box that could read Postscript.
Some minutes of 100% CPU later up pops a PSP window, with the document rendered in a font about five pixels square. Fair enough, I suppose, for what's basically a photograph editing application.
But really, how bizarre, posting something in a low level printer file format. We'll have people posting documents in PCL5 next.
Actually, ghostscript created a PDF about half the size of the .ps file.
-rw-r--r-- 1 kz None 239121 Apr 24 14:13 bhs.pdf
-rw-r--r-- 1 kz None 433678 Apr 24 14:02 bhs.ps
Of course, the PDF is Flate encoded internally, and the ps is a big fluffy text file, so the ps file would compress to well below the PDF size.
I am not your blowing wind, I am the lightning.
First, I sort out mail from the mailingslists I read.
Then, mail from friends, and people I correspond with a lot.
Finally, I have a weighted scoring recipe:
:0 Bh
* -199^0
#Assign an initial value of -199, mail gets filtered, if the score is above 0, at the end of the recipe.
* 50^1 ^(From|To):.*@hotmail.com
* 50^1 ^(From|To):.*@yahoo.com
* 50^1 ^(From|To):.*@aol.com
* 50^1 ^(From|To):.*@msn.com
* 50^1 ^(From|To):.*@excite.com
* 50^1 ^(From|To):.*@netscape.net
* 50^1 ^(From|To):.*@yahoo.co.uk
#Most mail to and from these domains is spam, so score it.
* 100^1 opt-out
* 50^1 opt-in
* 200^1 OTCBB
* 50^1 viagra
* 50^1 zyban
* 50^1 propecia
* 75^1 FREE
* 75^1 GUARANTEED
* 75^1 LEGAL
* 50^2 MILLIONAIRE
* 50^1 100%
#Words I only see in spam.
mail/Trash
This works quite well for me. If any spam gets through, I try to find some words, that I don't get in normal mail, and add them to the scoring.
/Styx
Message rules are very easy to set up and manage. No agents.
DBMAIL looks cool, once it supports postgresql it would be awesome.
I have been dissapointed in general with most SMTP, IMAP and POP servers. A real database is the proper way to do things. Email is my #1 app and I want to do complex queries on my archives.
So last year I bit the bullet and wrote a 50 line python program which imported all my mbox and Maildir format archives into a simple postgresql database. 600 megs worth over the last 4 years.
And another simple 50 line php program gives me a web database query interface. It suits my needs now and is much faster than searching through a big (but much much smaller) imap folder with almost every mail program I've tried. With some good design it really shouldn't be too hard to make an industrial strength email database system and I am surprised that it hasn't happened sooner in the open source world.
I think that direct SQL access to the mail database is preferred over IMAP. SQL gives you more capabilities and I find it less problematic than all the various combinations of IMAP servers and mail programs.
Jeff
ipv6 is my vpn
Spam filtering is one possible application of this type of tool, but the more useful involves taking the mail you *do* want, and sorting it into logical buckets. For instance, let's say work on several open source projects, belong to a couple organizations, and have a real-life job. You could toss a filter in your email that scans each incoming message and throws it in the proper bucket. This allows you to logically separate your mail to reduce confusion of each non-overlapping category.
:-)
Procmail only goes so far, it's really only useful for simple header scanning.. I could really see a good scanner utility being a valuable tool. Maybe Google should share some of their technology..
Josh Woodward
Alas, no. Adobe wanted ridiculous prices to license Display PostScript (DPS), the engine that NeXT used in the NEXTSTEP display system. (NeXT is a company. NEXTSTEP is an operating system.)
Given the ridiculous licensing prices, Apple went a different way and created Display PDF for Mac OS X's drawing system.
Ghostscript works just fine, but the lack of DPS is one of the reasons I still keep a NeXT cube on/under my desk.
There should be a moratorium on the use of the apostrophe.
Max V.
NeXTMail/MIME Mail welcome
Why are there so many people complaining about a PS link? The answer is simple: ./ is news for nerds, not for geeks.
So while the average geek keeps his favorite postscript viewer handy, the standart nerd wonders about such an ancient format and does not know how to feed his acrobat viewer with it...
Here is the solution for those irritated ones: try this piece of ancient software on the ancient adobe format, and you can miracously view it's contents!
Have fun and keep your google handy!
I have enjoyed using the VM module for Emacs. It allows sorting your entire Inbox into separate categorized mail boxes via regular expressions. Basically with one shift-A keystroke, my entire day's worth of mailing list stuff gets whisked away into a half-dozen different files. After this, I feel really sorry for people trapped in the Outlook dungeons!
Healthcare article at Kuro5hin
you can. Pay for a mail server that let's you administrate the configuration of, and a domain name (don't start with the "I shouldn't have to pay for it" crap, you want free? you get spam!).
Set up your mail server so that all incoming mail to your domain goes to you. then only give out email addresses such as yourcompanyname@mydomain.com. If companyA.com sells your email address to spammers, you can shut that email address off. you can tell your mail server to reject mail sent to companyA.com@mydomain.com. This is no sure fire way to stop everything, and someone who really wants to send you an email, can make up any string of alpha/numeric charicters and send it to you at your domain, But it's a really nice way to monitor who is selling your address. and you cut the address off when you see that they have compromised your information.
(note: many companies filter out the name of thier company before they sell thier address list, so your email never really makes it onto the list that the company ships)
Just use GhostView...
What is the difference between postscript and DPS? Any reason why DPS can't be integrated into X? The only effects of a DPDF renderer in OS X that I've seen are being able to view .pdf's without Acrobat and having vector-based widgets.
Karma: Good (despite my invention of the Karma: sig)
"Perhaps even one of them Perl monkeys will quickly hack such a background tool."
Been done already. Check out Mail::Miner.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
NM, here is this project that seems to be just that. Apparently Display Ghostscript is dead, but DPS lives on. Still don't see what the big whoop is.
Karma: Good (despite my invention of the Karma: sig)
My reaction was to want an e-mail reading program that didn't require any filter configuration, though I imagined it would do well to be given a few hints, such as who my boss is, who my mother is, and who my wife is. Other than that, let the program figure it out.
Imagine the canonical, old-fashioned secretary temp. She ('cause that's what the canonical version was) didn't have to know anything domain-specific to sort the morning mail. Magazines go together, bills go together, personal letters go together, etc.
I imagine an automated version for my e-mail. Look at who it is "to" (am I on the list?), look at who is "cc"-ed (am I on that list?), look at who it is from (my boss, wife, or mother?), look at who else it is to (boss, wife, or mother?), look at the thread it is part of (is it responding to something I previously wrote?), look at the content (does it mention me, things I have written, my boss, wife, or mother?). Was it sent to a mailing list? Was it written by someone I have explicitly written to (once or many times?)? Was it written by someone who has previously sent me direct e-mail (once or many times?)? Those ideas are just the obvious ones, think of others. Think of more. (Does it talk about sex, credit card merchant accounts, stock tips, or Nigerian money?)
Now take that and sort it by importance and similarity. Look for a way to present me in a descriptive summary, arranged in a hierarchy with a top-level of, say, 3 to 9 categories, a greatest depth no greater than, say, 4, and keep the sub-branching at intermediate nodes between 3 and 5--but don't max out all those dimensions at once, try to keep the total number of leaf categories to under, say, two dozen. Try to make more important items land higher in the tree and with few siblings, grouped with siblings of similar importance. (Maybe give an importance weight to each e-mail and balance the tree on that scale, that would float e-mails to me from my boss about my mother and wife really high with few siblings.)
This summary needs to be integrated with a complete index of the e-mail so I can see how a message fits into a larger thread, how it fits into previous e-mails.
I (the user) would need to tell the program when to make me a summary of my e-mail (e-mail reading is different when a lot comes in or just a little), and I want to be able to browse through old summaries, including deciding to see composite summaries or, say, the last several days, a week (or three), month, year, or 400 days.
So I think it ends up being a 4-part user interface:
List of summaries (which can be manipulated).
A given summary.
Exhaustive thread/date/subject/sender list (analogous to what every e-mail reader seems to have now). Note that this view could effectively be turned into an exhaustive address book. Frequent (favored) correspondents could be highlighted by me for ease in sending a new e-mail, and also to provide importance hints to the program. This is where I might say who my boss/wife/mother is.
A body of a (or more) specific e-mail being read, written, or old e-mail (sent or received) being reviewed.
And I could go on, but I won't. If anyone wants to write such a thing and wants to hear more, send me an, um, e-mail.
-kb, the Kent who has been saving all his e-mail (including spam!) for a year or so, providing plenty of raw material to test any such program.
Well, the attentive reader would have noted that I pointed out that Adobe wanted a very high per-seat license. Apple wanted to pay a flat rate, IIRC, and the two companies didn't work it out. So Apple went a different way.
DPS was used in a more fundamental way in NEXTSTEP. It was really amazing. There was true WYSIWYG, as the code on the screen was what was literally sent to the printer. Layout was really improved as a result, and you could mix postscript code with your drawing program efforts and see it previewed in a live fashion on-screen. It was easy to save documents in a portable fashion (PS), and a dozen other things.
There should be a moratorium on the use of the apostrophe.
Max V.
NeXTMail/MIME Mail welcome
The main result in Kleinberg's paper relates to finding NEW topics that start to appear in the stream. Let's say you already have categorization filters (procmail, keyword filters, your own set of folder hierarchies, whatever...), but there's a new topic that starts showing up in your mail, or in your newsgroup feed, or on CNN. Klienberg's result is a way to find that the new stuff really is NEW, and you might want to group it up together, and make a folder for it. You could do that automatically, or by hand, but first you have to know that there's a topic.
there's a bunch of other work in this area, what the NLP types call TDT -- "Topic Detection and Tracking"
jzw of Mozilla/Netscape fame have a hypothetical program called Intertwingle which is (Score:5,Interesting) ....
RFC1925
There's been lots of work on auto-classifying email. I did my semester project in Machine Learning on this in 1999. It's a fairly simple study, but it seems like a Naive Bayesian classifier using word counts as features does a pretty decent job of classifying email, and does really well on spam.
The paper is here here.
J.
Maildir DOES work great, and I use it myself for non archived emails.
/bin/ls: Argument list too long
They work great until you have lots of messages.
My postgresql email database contains 54,244 email messages. Current filesystems do not like having that many files in one directory. A filesystem is NOT a database - it only has one field (filename) that you can do queries on.
The database allows me to properly index the fields such as 'date', 'subject', and 'from' - for instance:
select count(*) from msgs where msg_from = '<myemailaddress@domain>';
returns a result in a fraction of a second because it does not have to iterate through all my messages. Whereas a Maildir directory with 5000 files in it can not be groked with wildcards. Try it!:
$ ls *
bash:
A big problem with the concept of putting only metadata in the database and the content in the filesystem is that you end up making the system even more complex as you need two different ways of accessing the data and the data is split between two sources.
The two different ways of accessing the data is a problem when you want to access the emails from another computer. For me it is simple - my Mac OS X machine can make a postgresql connection to my linux server and do queries including message content easy and quick.
SQL databases nowadays handle large text fields and blobs just fine and make it dead simple to back up, process, or query all the data.
'Folders' can be just SQL VIEWs and are way more flexible than seperate Maildirs for each folder.
By the way, I would LOVE to see an imap server and email client program handle 54,244 messages in one folder that I can view and search different ways without it bogging down or trying to cache 600 megs of data locally or trying to make a single list box with 54,244 items in it (and usually crashing in the process)
Please show me one so I can use it instead of writing my own smtp to postgresql gateway.
--Jeff
ipv6 is my vpn
Reminds me of the maps for the 3D network game for the Mac that Ambrosia made...Avara, I think? The maps were vector graphics, where different shapes meant different things and text inside the shapes was code. Very cool idea. I think there's still a lot of potential in the idea that source code doesn't necessarily need to be a simple linear text file.
Karma: Good (despite my invention of the Karma: sig)
Haha, that is what I like to see. Some common sense once in a while.
Some other transgressions: the Mac OS has forced the Apple menu on its users for nearly 20 years. Why can't I have a 3rd party menu? And sure people could download an alternative to GNOME terminal, but realistically who will exert the effort? And why don't I have a choice of who provides me with a tea timer in KDE?
Karma: Good (despite my invention of the Karma: sig)
Looks great so far! However I guess I have to manually create the postgresql database tables....
jeff
ipv6 is my vpn
There are plenty of e-mail mining tools in development. This particular work takes one particular approach to mining the data. Whether this approach will turn out to be useful remains to be seen.
this is perhaps the greatest example of slashdotter myopia ever. I don't give a crap about my karma, I just have to laugh at this AC:
It shouldn't take two months to get ghostscript and ghostview. In fact, it comes with most modern operating systems.
Clue time: 99% of people who've ever used a computer have never heard of either. If they click on the link above, they get a windows file box for "open with" and they wonder why the author didn't inlude a warning of what this strange file format was and what, exactly, they are supposed to do with this file.
DO NOT DISTURB THE SE
158213 Apr 19 09:41 bhs.ps.bz2
t.
I do pay jackass. My email comes with my internet connection.
I'm sorry I didn't make this clear. I'm not saying that you need to pay for email, I'm saying that you will have to pay extra to get your email filtered at the domain level.
what you do about telemarketers is your business, and outside the scope of this thread
That sucks. While Louisiana is chock full of corrupt politicians, they occasionally make it work for their residents. Our no call list requires that business who make unsolicited calls to Louisiana residents subscribe to it (to the tune of $800). Hefty fines for those that call a number that's on the list.
I get really pissed off when I get a call on my prepaid, expensive per minute cell phone, especially while I'm driving/riding my bike/sleeping/whatever. I'll have to figure out a way to simulate a car crash sound, so I can scream in agony, and then hang up.
I'm really curious how the legality of this works...the state controls access to it's phone lines under their conditions? Will they actually have the power to impose these fines on businesses that don't pay what amounts to their telemarketing tax? Our state constitution is based on Napoleonic Code...
See also for the FAQ for weasel^H^H^H^H^H^Htelemarketers
...Time is the best teacher, unfortunately it kills all of its students.
Most of the major MTA's will do this nowadays, but with a + rather than a -. I know sendmail does this, and am pretty sure about postfix and exim as well.
Look at this reference, for example.
(darren)