Slashdot Mirror


Text-Mining Your E-mail

Misha writes "There have been a number of weeks/months in anyone's life that called for a better organization of your Inbox. filtering and folders work, but it'd be nice to have an text-mining tool running in the background that categorized incoming messages by topic as they arrive. It's nice to see that besides NLP research, there are some great algorithmic advances being done, as seen in this paper. Perhaps even one of them Perl monkeys will quickly hack such a background tool." Note: it's a PostScript file.

153 of 217 comments (clear)

  1. there is no way to win... by Anonymous Coward · · Score: 1, Interesting

    They'll end up finding a loophole in your filtering, or you'll end up filtering out real emails.

    Only way to win is to kill it from the source. End of story.

    1. Re:there is no way to win... by ldopa1 · · Score: 1

      Doesn't this effectively amount to censorship? I agree that we (as individuals) need to make spamming less cost effective, but just preventing people from emailing (which is what you'd have to do) is censorship, unless of course you're the govt, in which case it's "protecting national interests"...

      --
      The Dopester
      "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
    2. Re:there is no way to win... by plague3106 · · Score: 1

      Um i think i have a right to say 'No, you cannot snail mail me' and 'no you cannot call me.' So why don't i have the right to say 'no you cannot email me.' Its not censorship if the audience does not want to hear what you have to say. Yes they have a right to speech, but that doesn't mean they can call me if i do not wish to listen.

      Censorship requires a 3rd party between the person 'giving' the speech and the person 'receiving' it.

    3. Re:there is no way to win... by tenman · · Score: 2

      you can. Pay for a mail server that let's you administrate the configuration of, and a domain name (don't start with the "I shouldn't have to pay for it" crap, you want free? you get spam!).

      Set up your mail server so that all incoming mail to your domain goes to you. then only give out email addresses such as yourcompanyname@mydomain.com. If companyA.com sells your email address to spammers, you can shut that email address off. you can tell your mail server to reject mail sent to companyA.com@mydomain.com. This is no sure fire way to stop everything, and someone who really wants to send you an email, can make up any string of alpha/numeric charicters and send it to you at your domain, But it's a really nice way to monitor who is selling your address. and you cut the address off when you see that they have compromised your information.

      (note: many companies filter out the name of thier company before they sell thier address list, so your email never really makes it onto the list that the company ships)

    4. Re:there is no way to win... by ldopa1 · · Score: 1

      "Um i think i have a right to say 'No, you cannot snail mail me' and 'no you cannot call me.'"

      Actually, you DON'T have that right. You DO have the right to say "I won't read what you snail mail me" and "No, I won't listen to what you have to say when you do call me." You DO (in the U.S.) have the right to say "You cannot call me again." (see the TCPA, section 11)

      Next time you see your friendly postal carrier, ask him (or her) why you have to get all that junk mail. Ask him/her to just chuck it. Guess what, he/she can't. It's illegal for anyone except the sender or the reciever to trash ANY mail intentionally, and even the sender can't trash it after it's been sent. Once it's in the hands of the USPS, it's actually no longer the sender's property...

      --
      The Dopester
      "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
    5. Re:there is no way to win... by wolf- · · Score: 1
      I had it out on the phone with a "gentlemen" yesterday afternoon over the "do not call me". One of his employees, for the 3rd or 4rth time this week, rang ALL my potts lines, hanging up before we could answer them. Then when we managed to answer, asked for me. My wife said "he isn't available". So he talked to her about our being homeowners (we arent) and how his company (no good bastard telemarkters) can make us a deal on a new home or upgrades. She said she was not only not interested in doing business with him, she didnt want to be called again, and wanted on a no call list. She asked for his name, his phone number (at which point he hung up)...

      Got to love caller id. Got the name, checked the website, called up and asked to speak to the owner.

      I got David Parton on the line, who informed me that if I didnt want him to call me, that I should be put on the worthless GA No Call List. I informed him that pursuant to 47 U.S.C. Section 227. that the burden of compliance was on him and not on me. After being handled rudely, unprofessionally by the original caller, the founder had the nerve to DEFEND his people, their proceedures, and his attitudes.

      Why do I say the GA No Call List is worthless? It is the LINE OWNERS who have to pay to be on the list! $5 per line for 2 years. I now have to pay to NOT have my dinner rudely interupted by an annoying unprofessional telemarketer who doesnt even have his "market" correct?

      Sure, why not. I have to manage the loads of spam that comes through our company mail servers. I have to pay for the storage, the transport and processing of this junk. Why shouldnt I have to pay for telemarketers also?

      The use of "marketers" in regards to spammers and "telemarketers" has been used loosly. As their shotgun methods of product placement doesnt match nearly anything taught in a true marketing course.

      --
      ----- LoboSoft specializes in Digital Language Lab
    6. Re:there is no way to win... by plague3106 · · Score: 1

      (don't start with the "I shouldn't have to pay for it" crap, you want free? you get spam!).

      I do pay jackass. My email comes with my internet connection.

      Should i pay EXTRA to block telemarketer calls? I don't think so, and i dont' think i should have to pay not to be bothered. It is after all a right i have.

    7. Re:there is no way to win... by tenman · · Score: 2

      I do pay jackass. My email comes with my internet connection.

      I'm sorry I didn't make this clear. I'm not saying that you need to pay for email, I'm saying that you will have to pay extra to get your email filtered at the domain level.

      what you do about telemarketers is your business, and outside the scope of this thread

    8. Re:there is no way to win... by RennieScum · · Score: 2

      That sucks. While Louisiana is chock full of corrupt politicians, they occasionally make it work for their residents. Our no call list requires that business who make unsolicited calls to Louisiana residents subscribe to it (to the tune of $800). Hefty fines for those that call a number that's on the list.

      I get really pissed off when I get a call on my prepaid, expensive per minute cell phone, especially while I'm driving/riding my bike/sleeping/whatever. I'll have to figure out a way to simulate a car crash sound, so I can scream in agony, and then hang up.

      I'm really curious how the legality of this works...the state controls access to it's phone lines under their conditions? Will they actually have the power to impose these fines on businesses that don't pay what amounts to their telemarketing tax? Our state constitution is based on Napoleonic Code...

      See also for the FAQ for weasel^H^H^H^H^H^Htelemarketers

      --
      ...Time is the best teacher, unfortunately it kills all of its students.
  2. What I want by clion999 · · Score: 2, Funny

    Here's to the researchers. I would like:

    * An email box that lets me extract the threads with my friends.
    * An email box that automatically ages the files effectively archiving them. Some of my mail folders/files are huge now and it takes too long to append them when new mail arrives.

    Yes, I realize I should get off my butt and do this, but it's faster to post on slashdot.

    1. Re:What I want by quigonn · · Score: 2

      In fact, at least point 2 can be easily realized using mutt.

      --
      A monkey is doing the real work for me.
    2. Re:What I want by Col.+Klink+(retired) · · Score: 2

      Use nmh. Messages are stored in separate files rather than an entire folder in one file. You can then auto-archive by date with something like:

      refile `pick +inbox -before '1 apr 2002'` -src +inbox +archive

      --

      -- Don't Tase me, bro!

    3. Re:What I want by Jobe_br · · Score: 2, Informative

      For your second point:

      An email box that automatically ages the files effectively archiving them. Some of my mail folders/files are huge now and it takes too long to append them when new mail arrives.

      you could switch to using the Maildir format instead of the typical single-file 'mbox' format. Maildir is popularly used by the qmail MTA as well as courier-imap. I run all my email servers in this matter and I've noticed significant speed improvements in mailboxes that have many messages.

      Maildir maintains three directories, of which 2 are significant: cur and new. Any new messages delivered into the Maildir mailbox is placed in the "new" directory, once its been read, its moved into the "cur" directory. Each message is its own file, so no speed penalty is invoked for appending messages to mailboxes with many messages. Of course, all these different directories and such are transparent to the end-user, Maildir capable MUAs (for console users) and of course Maildir capable IMAP/POP systems are freely available (qmail does SMTP+sendmail wrapping and includes a basic POP3 daemon; courier-imap does IMAPv4 amongst other things; all the apps lend themselves to be used in an SSL via stunnel environment)

      Just a thought ... :)

    4. Re:What I want by Bamfsog · · Score: 2

      I know this is the wrong place to point this out, but Oulook does what you are asking for.

      You can sort a folder by user/subject/date, and there is a built in thread view. You can also use the autoarchive feature, or manually archive messages in X folder(s) older than Y date.

    5. Re:What I want by nosferatu-man · · Score: 5, Informative

      Welcome to Gnus. Have a sandwich.

      (jfb)

      --
      To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
    6. Re:What I want by swordboy · · Score: 2

      Here's what I want:

      A google plug-in for my mail client.

      Thanks in advance!

      --

      Life is the leading cause of death in America.
    7. Re:What I want by antiher0 · · Score: 2

      gnus has been doing this for years... as well as other neat things like mail scoring (similar to news scoring) so that mail you don't want to read gets filtered to the bottom of your list or (if you tell it to) doesn't even show up at all. Similarly, mail that you most want to read (based on past response) gets bubbled up to the top. gnus also supports mail expiry (once again, similar to news) so that old mail gets Handled(TM).

    8. Re:What I want by doom · · Score: 2
      Use nmh. Messages are stored in separate files rather than an entire folder in one file. You can then auto-archive by date with something like: refile `pick +inbox -before '1 apr 2002'` -src +inbox +archive
      Yeah, I was wondering a bit about what "text mining" your email is supposed to be about exactly...

      Personally, I use mh (using the emacs mh-rmail frontend). I refile stuff automatically typically just based on the '-from' (using commands much like the above pick/refile). And if I'm looking for something I remember seeing awhile back, a grep on one or two mail folders (which are just directories full of text files for us mh users) does a pretty good job...

      I won't say that there's no way to improve on this, but any fancy system that someone proposes has got to beat some pretty effective simple tools...

      I mean, if you're really after identifying a burst of activity on a given topic... wouldn't a combination of text searches and visual scans of subject headers sorted by date get you 90% of the way there?

      While we're on the subject, anyone taken a look at this old jwz idea: Intertwingle

    9. Re:What I want by doom · · Score: 2


      Postfix is also supposed to support the
      maildir format.

    10. Re:What I want by Mark+Wilkinson · · Score: 1

      To spur "enterprise Linux," Big Bang, the distributed two-phase commit.

      What does this mean?

    11. Re:What I want by nosferatu-man · · Score: 2

      You're the first person who's ever asked me that.

      It's a line generated by a text disassociator that I wrote and then pointed at a bunch of articles from those stupid rah-rah-rah Business 2.0 style rags a couple of years back. I wish I'd have saved the rest of the generated text: it was hilarious.

      Best,
      'jfb

      --
      To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
  3. PS-PDF Document format conversion by Misha · · Score: 5, Informative
    --



    I was thinking of how to intentionally fail my drug test... It would make a good memoir story someday.
    1. Re:PS-PDF Document format conversion by d3xt3r · · Score: 1

      If you run linux ps2pdf works nicely as well.

    2. Re:PS-PDF Document format conversion by DeadSea · · Score: 2

      PS-PDF is great for quickly mirroring webpages. I'm suprised that I don't see more people doing it here on slashdot to get some quick karma when sites get slashdotted. You have the webpage open in your browser (because you got there before the crowd). First you print it to a postscript file (netscape does this nicely). Then you run it through ps2pdf or some other tool like this and you have have the webpage (with all the pictures) mirrored in a single file. My friends were doing this on sept 11 when all the news sites were going down. Anything one of us saw, we all saw.

    3. Re:PS-PDF Document format conversion by KingKire64 · · Score: 1

      Here is a converted PDF from that website http://wheel.compose.cs.cmu.edu:8001/cgi-bin/brows e/objreal/BPtemp4701.1019671810,0

      Right Click and Save as PDF it has a temp File name

      --
      "All I can tell the "lesser of two evils" folks is that if they keep voting for evil, they'll keep getting evil."-Lp.org
    4. Re:PS-PDF Document format conversion by billnapier · · Score: 1

      Why not just read it using a PostScript viewer?

    5. Re:PS-PDF Document format conversion by daeley · · Score: 2

      If using Mac OS X, one could do the same thing in any printing-capable browser (or any other program). Use the Print command and click on the "Preview" button in the dialog. This automatically creates a PDF version of the document, which can be saved and uploaded.

      --
      I watched C-beams glitter in the dark near the Tannhauser gate.
    6. Re:PS-PDF Document format conversion by ravenwing_np · · Score: 1

      Alright, that explains why the load average on my server is up to 30 and 40. There is just an old Sparc4 doing the bulk of the converstion. If things go slowly, now you know.

    7. Re:PS-PDF Document format conversion by gasull · · Score: 1
      Here's a link to a terribly useful site for converting your postscripts and word docs into pdf or jpeg.

      Just click here (Converting postscript to pdf).

  4. The importance of E-mail history by Phred_Johnston · · Score: 2, Insightful

    I'm sure I'm not alone in saying that having a good history of well filtered incoming, and especially just about all of my Outgoing (Outbox) available for searching. My Outbox has been a lifesaver several times when someone claims that they didn't have that (electronic) discussion with me. It's great to quote "in a message sent... ...I asked you to...".

    1. Re:The importance of E-mail history by Liora · · Score: 1

      One question... I agree that it is useful to have an outbox archive, but what do you do if they still say they didn't get that or say that? Admittedly I can print out their email, but that can be fabricated and then printed, or I can forward them their own original message, but that can be fabricated, etc... Am I missing something?

      --
      Liora
    2. Re:The importance of E-mail history by ManDude · · Score: 1

      I'm on the other end of ""in a message sent... ...I asked you to..." IT, corporate, and sales send a lot of email and it becomes imposible to keep track of everything that they want you to do or not do. I am to the point where I skim very quickly anything that they send. I am expecting an email any time for one of them saying ""in a message sent... ...I asked you to..." and all I can say is "f?(k-you".

  5. Filtering by pyrrho · · Score: 1

    That feature in the description is not text mining, just filtering.

    --

    -pyrrho

  6. Yet another reason for.. by Dr+Caleb · · Score: 4, Informative
    Lotus Notes.

    It automagically does full text indexing of all specified databases. To it, your Inbox is just another database.

    --
    "History doesn't repeat itself, but it does rhyme." Mark Twain
    1. Re:Yet another reason for.. by LeeZard · · Score: 2, Insightful

      That's not the point. The paper is talking about modeling spikes in topic/content of data streams over time. This is the second layer analysis of the meta-data that gets stored in the database.

    2. Re:Yet another reason for.. by ConceptJunkie · · Score: 2

      Upside:

      Lotus Notes does all kinds of things automagically.

      Downside:

      It's _Lotus Notes_, the application that makes Microsoft Office look lean and mean.

      --
      You are in a maze of twisty little passages, all alike.
    3. Re:Yet another reason for.. by Dr+Caleb · · Score: 3, Informative
      How do you figure that?

      Lotus Notes (5.0.5), as installed on my system is 127M (no modem files etc) with 59M in help.nsf files, and my .NSF file and templates area hair over 12M. MS Office is over 160M, without PPT, and that's just the Program Files\Microsoft Office directory.

      Lotus Notes is pretty clean, so most of it's files are in 1 directory, not spread out over umpteen directories like Office.

      --
      "History doesn't repeat itself, but it does rhyme." Mark Twain
    4. Re:Yet another reason for.. by Dr+Caleb · · Score: 2
      Notes was not meant for email. Read the History of Notes.

      Yes, it can be bloated and slow, but what isn't nowadays, taking into consideration that people need things like calandaring, meeting scheduling and collaborative tools? What else can run on multiple platforms? Outlook?

      If you get to know it, understand it and use it, you never know, you might like it.

      --
      "History doesn't repeat itself, but it does rhyme." Mark Twain
    5. Re:Yet another reason for.. by nvainio · · Score: 1

      My mutt is 460 kb, my grep is 75 kb.

    6. Re:Yet another reason for.. by Dr+Caleb · · Score: 2
      You call Windoze and OS/2 "multiple platforms" Those are two. I also have it running on Unix, AIX and OS/400. Kinda nice since we're an AS/400 shop.

      I am forced to use it by my employer

      {a hush falls over the crowd} You mean your employer actually tells you what you can do at work, while they are paying you! How brutal it must be for you!

      (guess who) and no, I don't like it.

      I can guess, and I probabally used to work for them the past 10 years, which is where I got all my Notes training.

      No technical person I have ever talked to likes it.

      Strange. Any technical person I've ever talked to never turns their back on a good solution, and technical people who understand Notes love it. Especially if they have been "rescued" from Exchange Hell. Are you saying I'm non-technical?

      Are you management? Yes, MIS.

      --
      "History doesn't repeat itself, but it does rhyme." Mark Twain
    7. Re:Yet another reason for.. by Dr+Caleb · · Score: 2
      My car is red and I like Pizza and beer. So?

      --
      "History doesn't repeat itself, but it does rhyme." Mark Twain
    8. Re:Yet another reason for.. by Icculus · · Score: 1

      Believe it or not, I would use most of these words to describe Lotus Notes, or more precisely the feelings it evokes in me both as a user and a developer.

    9. Re:Yet another reason for.. by jonbrewer · · Score: 2

      All you have to do is use Lotus Notes for a few days on an aging PowerMac with 8 or 16MB ram, and you'll give up on it forever. You'll also tell everyone you know 1. what a horrible thing Lotus Notes is, and 2. what a horrible thing a Macintosh is.

      Those at Apple responsible for allowing PowerMacs to ship with System 7.5.x and less than 32MB ram should be banned from the industry. When an OS by default takes more ram than a system has, and is coupled with an application like Lotus Notes, which is hungry, nothing good can ever happen.

      This is, IMNSHO, a good part of the reason that so many corporations ditched their Macs in the mid-ninteys.

  7. Re:Link to a postscript file? by jordan_a · · Score: 1

    Yeah, a link to an standard document format that you can get viewers for on almost every platform. Damn that's soo inconsiderate, where's those word documents?

  8. The ultimate spam blocker? by ldopa1 · · Score: 2

    This would be an awesome tool to block spam. If this program could look at the text of an email message and determine that it is a solicitation of some kind and then drop it into an email "pit" (you know, a folder mapped to /dev/null), that would make my life a LOT easier...

    --
    The Dopester
    "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
    1. Re:The ultimate spam blocker? by set · · Score: 1

      SpamAssassin does this already, using a genetic algorithm.

    2. Re:The ultimate spam blocker? by jmb-d · · Score: 2, Informative

      This would be an awesome tool to block spam. If this program could look at the text of an email message and determine that it is a solicitation of some kind

      SpamAssassin will do this part for you.

      --
      In walking, just walk. In sitting, just sit. Above all, don't wobble.
      -- Yun-Men
    3. Re:The ultimate spam blocker? by ldopa1 · · Score: 1

      Do you have a link to this? Any more information you have would be welcome..

      --
      The Dopester
      "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
    4. Re:The ultimate spam blocker? by set · · Score: 1

      A simple google search would have found it, but it's spamassassin.org.

  9. a suggestion by 56ker · · Score: 1

    "that categorized incoming messages by topic as they arrive." - you can already sort messages into different folders depending on their topic by setting up rules.

    1. Re:a suggestion by spencerogden · · Score: 2

      This is a little more in depth than just matching strings in headers. This is about determining the topic of and email.

      Spencer

  10. Too much information. by abucior · · Score: 5, Funny

    Personally, I'd prefer that I simply get less email. The fact that we need NLP tools to pre-screen our email for us just shows how information-overloaded our society has become. What I really need is a tool at the sender's end that can pre-screen my email and tell the sender "Don't send this. He just doesn't care!"

    1. Re:Too much information. by ldopa1 · · Score: 1

      If our society has too much information, why did you just add that little tidbit to my stream of conciousness? Are you trying to make my head explode?

      Frankly, there is no such thing as too much information. Information is never created, knowledge is created. The information was always there (and always will be), you're just percieving more, creating knowledge of the information.

      --
      The Dopester
      "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
    2. Re:Too much information. by ConceptJunkie · · Score: 2

      Yes, but by your definition, information equals entropy, and there's definitely too much of that. We need some way to reverse that trend.

      I wonder what could be done with a really hot cup of tea...

      --
      You are in a maze of twisty little passages, all alike.
    3. Re:Too much information. by iabervon · · Score: 2

      There's plenty of information I want to get that I don't want to look at as email.

      For example, I'd like to get messages inviting me to events I'm unlikely to go to, and I'd like to have their dates get marked down so that I can see what is happening on a given day if I feel like doing something.

      I'd like to get new addresses for people, but I want to have my addressbook updated instead of seeing the message.

      It would be really convenient to have software that would figure out this sort of information from a human-readable message, since people are likely to want to send it in natural language (and the message probably includes more information that I might want to see if I decide I care.

    4. Re:Too much information. by j.e.hahn · · Score: 1

      You can't. The 2nd law of thermodynamics forbids it.

    5. Re:Too much information. by foniksonik · · Score: 1

      What about forcing marketers to register in a database? New legislation that states that any marketer not registered and still found to be actively marketing will suffer mandatory fines w/ a 3 strikes rule or something for repeat offenders.

      This would mandate and maintain opt-in and opt-out standards while still allowing initial marketing by the business. ie: they could send us one mail, we could decide if we wanted to ever receive from them again and then go to the database and choose to opt-out or later on choose to opt-in again.

      This would also give consumers recourse against 'rogue' marketers. Businesses who are legitimately marketing thier product would not have any problem with this and 'rogue' marketers could be foudn guilty of invasion of privacy, which is what this all amounts to...

      Mod this up if you think it's a legitimate idea, or maybe even start an 'Ask Slashdot' article to see if we could hash out a workable version and push for legislation somehow.

      --
      A fool throws a stone into a well and a thousand sages can not remove it.
    6. Re:Too much information. by phaedrus · · Score: 1

      ahhh, _bored of the rings_

  11. Sort it by domain... by qurob · · Score: 1

    I can sort reports from devices, co-workers, clients....each goes in its own folder....

  12. What's wrong with IMAP ? by MagicFab · · Score: 1

    IMAP (Internet Message Access Protocol) was designed to centralize email information, I believe. If stored/implemented with a database, what more would you need ?

    I think querying through SQL would satisfy most of us.. and be very useful in corporate environments (for example, query all email sent from a user to support), and it's already done by some projects like DBMAIL.

    Anybody out there with experience using these ?

    BTW, there's an extensive database of IMAP products including some that make the data accessible via LDAP... hours of fun!

    --
    Notepad specialist & FAT administrator, group training available
    1. Re:What's wrong with IMAP ? by statusbar · · Score: 5, Interesting

      DBMAIL looks cool, once it supports postgresql it would be awesome.

      I have been dissapointed in general with most SMTP, IMAP and POP servers. A real database is the proper way to do things. Email is my #1 app and I want to do complex queries on my archives.

      So last year I bit the bullet and wrote a 50 line python program which imported all my mbox and Maildir format archives into a simple postgresql database. 600 megs worth over the last 4 years.

      And another simple 50 line php program gives me a web database query interface. It suits my needs now and is much faster than searching through a big (but much much smaller) imap folder with almost every mail program I've tried. With some good design it really shouldn't be too hard to make an industrial strength email database system and I am surprised that it hasn't happened sooner in the open source world.

      I think that direct SQL access to the mail database is preferred over IMAP. SQL gives you more capabilities and I find it less problematic than all the various combinations of IMAP servers and mail programs.

      Jeff

      --
      ipv6 is my vpn
    2. Re:What's wrong with IMAP ? by ryochiji · · Score: 1

      Personally, I find the search command (part of the IMAP specs) to be sufficient for finding old messages. I usually have a general idea of which folder a particular message might be and some other key word or information, which is all I need to find the message in question.

      Only times I have problems with this is when I have to search through a mailing list archive containing several years' worth of digests... so, I guess a more "intelligent" solution would have it's uses.

    3. Re:What's wrong with IMAP ? by Tracy+Reed · · Score: 1

      Maildir has done very well for me. Fast and reliable. A filesystem IS a database. I don't think there is much need to put email into any other sort of database. It just adds another unnecessary layer of complexity. I wouldn't mind seeing a database used to store metadata so the mail can be quickly searched but I'd prefer to leave the emails themselves in the filesystem and have the database of metadata contain pointers to the filenames in the filesystem.

    4. Re:What's wrong with IMAP ? by statusbar · · Score: 2

      Maildir DOES work great, and I use it myself for non archived emails.

      They work great until you have lots of messages.

      My postgresql email database contains 54,244 email messages. Current filesystems do not like having that many files in one directory. A filesystem is NOT a database - it only has one field (filename) that you can do queries on.

      The database allows me to properly index the fields such as 'date', 'subject', and 'from' - for instance:

      select count(*) from msgs where msg_from = '<myemailaddress@domain>';

      returns a result in a fraction of a second because it does not have to iterate through all my messages. Whereas a Maildir directory with 5000 files in it can not be groked with wildcards. Try it!:

      $ ls *
      bash: /bin/ls: Argument list too long

      A big problem with the concept of putting only metadata in the database and the content in the filesystem is that you end up making the system even more complex as you need two different ways of accessing the data and the data is split between two sources.

      The two different ways of accessing the data is a problem when you want to access the emails from another computer. For me it is simple - my Mac OS X machine can make a postgresql connection to my linux server and do queries including message content easy and quick.

      SQL databases nowadays handle large text fields and blobs just fine and make it dead simple to back up, process, or query all the data.

      'Folders' can be just SQL VIEWs and are way more flexible than seperate Maildirs for each folder.

      By the way, I would LOVE to see an imap server and email client program handle 54,244 messages in one folder that I can view and search different ways without it bogging down or trying to cache 600 megs of data locally or trying to make a single list box with 54,244 items in it (and usually crashing in the process)

      Please show me one so I can use it instead of writing my own smtp to postgresql gateway.

      --Jeff

      --
      ipv6 is my vpn
  13. look by Joe+the+Lesser · · Score: 4, Funny

    Now we all now that most email is delivered promptly by gremlins, but gremlins are hungry and will eat a few bytes here and there.

    They also leave waste in the form of spam.

    So, I propose that we turn to gnomes to deliver the mail instead, as they are much cleaner, and can be satiated by attaching a file like 'Hamburger.txt'.

    --
    "I only speak the truth"
    Karma: null(Mostly affected by an unassigned variable)
  14. The joys of owning a domain by CaptainPhong · · Score: 5, Insightful
    I've found the most joy from owning my own domains, and a lot of it has to do with e-mail sorting/filtering as much as the traditional benefits (a permanent www.yourdomain.com web site address and yourname@yourdomain.com e-mail address).

    Every time you sign up for some mailing list or discussion group, create a new e-mail account or alias for just those mailings. Bam, it's automatically sorted out by itself with extreme ease. If you have limited bandwith (or are checking, say, on your palm) sometimes, just check your important addresses frequently, and reserve your mailing lists for a once-per-day check.

    If some site asks for your e-mail address to download a piece of software, or to register, make up a new alias and give that to them. If you start getting tons of crap at that address, you can just remove that alias, and they get it all bounced back in their stupid spamming faces.

    Give one address to your cow-orkers just for work stuff. Give a different one to your Mom and other techno-nots that blocks all attachments. Give another one to your friends with brains that goes unfiltered. For people you don't want to talk to, give them the address of an autoresponder tied to Eliza.

    Be a *Happy Camper* and let your addresses be *Bubbles* and you be just *You*.

    --
    ... "Give me a woman who loves beer and I will conquer the w
    1. Re:The joys of owning a domain by berck · · Score: 1

      I use sneakmail.com for giving my address out on webforms where I'm worried about spam. Easy for those of us without FQDNs.

    2. Re:The joys of owning a domain by UnknownError · · Score: 1

      Small amendment to CaptainPhong's three categories of mail senders ("Mom and other techno-nots"; "friends with brains"; and "people you don't want to talk to"): Couldn't you just say "techno-nots" and leave motherhood out of this? Giving birth doesn't lead, necessarily, to a life of spewing unwelcome email attachments. In fact, I've heard some mothers even write code....

      Thanks, though for the autoresponder-tied-to-Eliza idea.

  15. Uh... by willis · · Score: 1
    I'm not sure if this was for karma or not, but you might want to format text before dumping it like this. Just looks more professional.

    --

    there is no thing
    what else could you want?
  16. Re:Link to a postscript file? by Pfhreakaz0id · · Score: 1

    How about plain text, HTML, RTF, or PDF? Every person who's been on the internet longer than two months has Acrobat.

  17. Re:hi, i need to ask a question by ldopa1 · · Score: 1

    I used to know how to do this, but I forgot it as soon as I figured it out. Go figure...

    What was I talking about?

    --
    The Dopester
    "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
  18. Re:Link to a postscript file? by tps12 · · Score: 2

    Not to mention that a PDF would be 10x the size. I have no idea why Mac and Windows OS's are so baffled by postscript...half the printer drivers have to deal with it already, why not just bundle a damn interpreter with the OS and have a minimal frontend on it for screen viewing?

    --

    Karma: Good (despite my invention of the Karma: sig)
  19. Re:she said this, and she created COBOL too by Rorschach1 · · Score: 2

    Damn you, Admiral Hopper! I've got a huge stack of COBOL listings on my desk that I've got to translate to, of all things, vbscript (damn you, Bill Gates!)

  20. Re:text inside the PS file - references cut by GutBomb · · Score: 1

    well, i tried to paste it in here but the lameness filter would not let me. try this.

  21. Been there, done that by ansonyumo · · Score: 1
    I worked for a dot-com that built an application to do just this. Our focus was on product management, to automatically classify incoming email about products to specific categories in a hierarchy. The classification part was very general and could have been used for just this purpose.

    I say could have because it got sucked down the drain in late 2000 with all of the companies that didn't have a damn thing to offer. Lesson #1: make sure your CEO gets along with your venture partner.

    No, I'm not bitter. Much.

  22. how 'bout common-sense? by jptwo · · Score: 1

    a nice methods paper, but mr. kleinberg doesn't use any of the free metadata that comes with email and news: to, from, subject.

    i use outlook, and cluster my mail by sender... most of the time, that tells me pretty easily whether a given piece of mail is a work email, a personal email or a mailing list. from there, i check the subject line of work emails, just to confirm my categorization of work/humor/administrivia.

    i'd want to see a comparison between a metadata-only method (rules and filters on the RFC 822 header) and mr. kleinberg's method before i'd consider using it.

  23. Remembrance Agent by Tekmage · · Score: 5, Informative

    It's more general than e-mail, but in the wearable computing community, there's a little application called Remembrance Agent, written by Bradley Rhodes that many folks use. In terms of stand-alone UI, it's still quite primitive, but that's because it was built around dynamic hooks into Emacs.

    I've been playing around with some Java-based wrapper code, to wrap the ra-retrieve executable in a Server and allow clients to access the data via sockets. I have a Java-based client coded up that hooks into the System clipboard, but it's still in alpha-mode. All GPL'd of course, but needs a little time to mature. It's a proof-of-concept, work in progress. :-)

    Check out Brad's site for more insight into the work he did and is doing.

    --
    --The more you know, the less you know.
    1. Re:Remembrance Agent by jamieo · · Score: 1

      Yes I've used this a little and it's *very* nice. You're writing an email, or any document, and the bottom of your screen has a list of other emails and documents related not only to the mail/document you're writing, but the part of the document you're writing!!!

      I first saw this a few years ago and when you first use it it blows you away. Why this hasn't become wildly popular I don't know.

      Check it out.

      Jamie

    2. Re:Remembrance Agent by xyzzy · · Score: 2

      Perhaps because it's only compatable with RMAIL through a one-of-a-kind elisp interface?

      The correct question is, why hasn't someone taken it and tried to hook it into a somewhat more common platform.

    3. Re:Remembrance Agent by Tekmage · · Score: 2

      Which would be why and what I'm playing with... :-)

      I have it hooked into the system clipboard, so getting info to and from the RA is easy via that mechanism. The true power will come when/if there happens to be a way to "watch the keys". Kind of like a key-logger, but I'd rather not have it watching from the keyboard side; it should only be watching what's visible, not everything including passwords.

      The challenge (learning curve on my part) is getting deep enough access to system level interfaces via Java... Focus-independent access to mouse and keyboard input streams. Also have to work up an ra-index wrapper; more a function of the JRAServer than JRAClient classes.

      --
      --The more you know, the less you know.
  24. Re:Link to a postscript file? by tps12 · · Score: 2

    Postscript has the best reproduction accuracy for the file size. Assuming it has any kind of figures or equations, the only other reasonable alternatives are dvi and pdf. I've never seen dvi files rendered in a decent amount of time, and pdf is too fat, esp. for a paper linked to by slashdot. :)

    --

    Karma: Good (despite my invention of the Karma: sig)
  25. One use to rule them all by Col.+Panic · · Score: 3, Funny

    my $pr0n = "adult";
    my $spam = "viagra";
    my $urgent = "penis enlargement";
    open (INBOX,/home/mail) or die "Damn! No fun for me:$!\n";
    @list = readdir(INBOX);

    foreach $ (@list) {
    if (-f $spam) {
    my $status = unlink($spam);
    }
    if (-f $pr0n) {
    my @MUST_SEE = $pr0n;
    next;
    }
    if (-f $viagra) {
    my @RAINY_DAY = $viagra;
    next;
    }
    }
    # or something like that ...

  26. Six Degrees from Creo by PHAEDRU5 · · Score: 2


    OK, it's not a piece of Linux software, but it is a beautful idea:

    http://www.creo.com/sixdegrees/

    --
    668: Neighbour of the Beast
  27. Re:Link to a postscript file? by SuiteSisterMary · · Score: 4, Funny
    why not just bundle a damn interpreter with the OS and have a minimal frontend on it for screen viewing?
    Gee, wouldn't that be illegally using their monopoly to muscle out third party developers? Why, if the OS had a PS viewer built in, nobody would every buy one! Businesses would go bankrupt!
    --
    Vintage computer games and RPG books available. Email me if you're interested.
  28. PDF File by bheilig · · Score: 1

    Here is a PDF version.

    Boy could I use the Karma!

    1. Re:PDF File by bheilig · · Score: 1

      Crappy geocities... Is there someplace else I can put it?

  29. Censorship? by PCM2 · · Score: 2

    Hoo boy. Here we go again. When are you kids going to get it straight?

    - Choosing not to listen to somebody is *not* censorship.
    - Throwing your mail away before you open it is *not* censorship.
    - Choosing not to relay somebody's spam is *not* censorship.
    - Choosing not to broadcast somebody's TV program, even if you own a TV network, is *not* censorship.
    - Telling a movie producer you won't distribute his/her movie unless he/she makes cuts or changes to the subject matter is *not* censorship.
    - Rallying your church group together to burn books is *not* censorship.
    - Refusing to sell certain magazines or newspapers, if you own a newsstand, is *not* censorship.

    The only way somebody can be truly "censored" is when there is no legal means for that person to get his/her speech/art/etc. produced and disseminated to the pubic. Generally speaking, the only body with that type of power is the government -- because they make the laws.

    Everything else is merely an inconvenience. It may piss you off, sure, and you may wish things were different. But you can't force people to support you, encourage you, or fund you if they just don't want to. For example, people in this country (the US) *do* have a right to decide what material constitutes pornography, relative to their local community standards -- and if you don't like it, you are within your rights to move to another town.

    "No censorship" does not mean being forced to look at every piece of crap that somebody wants to throw in your face, and god help us if it did.

    --
    Breakfast served all day!
    1. Re:Censorship? by alouts · · Score: 2, Interesting
      Very valid points but:

      The post you're ranting against was a reply to one that suggests filtering is not what we should do. That spam needs to be "killed at the source". Which means legally preventing someone from creating any mail in the first place.

      Say what you will about spammers, but that IS censorship.

      ('Course there's plenty of people here who believe that censorship is fine in this case, but that's not what you're arguing, so I won't either.)

    2. Re:Censorship? by ldopa1 · · Score: 1

      I was saying that killing it at the source is censorship, not filtering it out.. So, I am in complete agreement. If I wasn't clear enough on that, I'm sorry..

      --
      The Dopester
      "Yes, I'm a Karma Whore, but I'm doing it to pay my way through school."
    3. Re:Censorship? by t · · Score: 2
      It is only censorship if "killed at the source" means to literally kill the fucknut sending the spam. Death is the most convienient and widely implemented form of censorship in places like China.

      Preventing someone from sending emails is NEVER censorship by definition. They can always go to Kinkos and make plain old paper mailings and then mail them to everyone on the planet.

      t.

  30. Finally... by Aiku1337 · · Score: 2, Funny

    Now I can automatically filter my barely-legal porn spam from my anime porn spam. Lets hear it for technology =)

  31. Postscript document by Tim+Ward · · Score: 3, Interesting

    Somewhat to my astonishment when I clicked on the link up popped a box asking me to confirm Postscript Renderer options! I had no idea that I had anything on this box that could read Postscript.

    Some minutes of 100% CPU later up pops a PSP window, with the document rendered in a font about five pixels square. Fair enough, I suppose, for what's basically a photograph editing application.

    But really, how bizarre, posting something in a low level printer file format. We'll have people posting documents in PCL5 next.

    1. Re:Postscript document by rgmoore · · Score: 2
      But really, how bizarre, posting something in a low level printer file format. We'll have people posting documents in PCL5 next.

      What's so strange about it? Postscript has the great advantage that it's actually designed to describe exactly what's on the page. That lets you produce very nicely formatted documents that will render exactly the same way on any computer, which makes it the output format of choice for programs like TeX. It's great because it's easy to print, so people who prefer to see things in dead tree format can do so easily. It can be processed into PDF very easily, too, so people who like PDFs won't have any problems. Sounds like a good choice to me.

      --

      There's no point in questioning authority if you aren't going to listen to the answers.

    2. Re:Postscript document by ansonyumo · · Score: 1

      Most academic publications are delivered in either Postscript or TeX, it's not unusual.

    3. Re:Postscript document by ansonyumo · · Score: 1

      Furthermore, PS isn't a low-level printer file. It is a page description language, and a very powerful one at that. It was the language used to implement GUIs for NeXT's NeWS, and its offshoot PDF is used on the Aqua GUI.

    4. Re:Postscript document by Permission+Denied · · Score: 1
      But really, how bizarre, posting something in a low level printer file format. We'll have people posting documents in PCL5 next.

      Not bizarre at all. Is it bizarre that people post things in PDF format? Should they drop PDF and just post using MS Word instead, since Word is a far more universal and portable method for distributing professionally typeset documents?

      What's the relationship between PostScript and PDF? Look into it and you'll see PDF was created just to deal with a couple of issues that distributing PS files has, and PDF is not far removed from PS.

      Search the web for any mathematics papers, and you'll find most of them in PostScript. Recently, I've seen some people using PDF for this purpose, but PS is far more prevalent in the math and CS communities.

      Years ago, there was a move to make the IETF standardize RFCs in PostScript format since it was almost as universal among the intended audience as plain text (the current format for RFCs).

      PostScript is not a stupid low-level printer language like PCL. PostScript is a beautiful, full-fledged powerful programming language, and contains programming constructs that are far more "high-level" than, for instance C or C++ (like equivalance of data and code, something you usually don't find in procedural languages). It's been loved by computer professionals for years.

      If you're interested, do a google search and you'll find the "blue book" the "red book," etc. Learning PostScript will change the way you think about programming, which should really be the important reason for learning a new language.

      Don't diss PostScript.

    5. Re:Postscript document by jonbrewer · · Score: 2

      .ps generally sounds like a good idea to many science-types.

      I think it's rather tiring.

      If I didn't have a full install of Acrobat on my system, I wouldn't have bothered with it. (It configured itself to handle .ps documents by converting them into .pdf.)

      .pdf has been around for as long as the commercial Internet, and is understood by every computer I've used in the past five years. It can be created by innmuerable commercial and free (as in beer and as in speech) tools. It can be read by Acrobat reader, a fantastic free (as in beer) tool from Adobe.

      There really are no reasons to publish in .ps other than whim, eliteism, or ignorance. All of those being sins in my book.

    6. Re:Postscript document by t · · Score: 2
      The reason is ignorance but not on the part of the publishers.

      Acrobat is shit.

      ggv will view .ps, .pdf, .ps.bz2, .ps.gz, probably others. Works great. There is no reason to differentiate between any of them. And if you really must ps2pdf works quite well.

      t.

    7. Re:Postscript document by Tim+Ward · · Score: 2

      Yes dear, I have written code in PostScript, both hand-coded programs (to generate forms etc) and machine generated (ie I've written PostScript printer drivers). In the mid 1980s IIRC.

      But it's still not an appropriate language to distribute documents that you want anyone other than Unix users to read. This is a historical accident. For anyone who is too young to remember, this came about because the first decent laser printer happened to be a PostScript machine, and Unix didn't develope a printer driver model - instead everyone just emulated, one way or another, the LaserWriter. (I don't know if this has changed, I haven't found it profitable to do much work on Unix graphical apps the last few years.)

      PDF is vastly more sensible as a general distribution format.

      I usually take the distribution of a document in PostScript format as a message that I'm not part of the intended audience, and I don't read it. If I'm really not part of the intended audience then that's fine, of course, and everybody's happy; but if I was intended to use it then they got the format wrong.

    8. Re:Postscript document by Permission+Denied · · Score: 1
      Yes dear, I have written code in PostScript, both hand-coded programs (to generate forms etc) and machine generated (ie I've written PostScript printer drivers). In the mid 1980s IIRC.

      Fair enough; you gain my respect. However, your original post made it seem like this was something new (We'll have people posting documents in PCL5 next, perhaps a subtle troll?).

      Like you said, whether posting PostScript is appropriate depends on your audience. This paper was from a guy in the CS department at Cornell, so I'd say it was altogether appropriate. I'd say less than one tenth of the posters here on slashdot read the paper, since it talks about automata theory and does not at all concentrate on spam busting. Those versed in automata theory have probably been through a traditional CS/Math program which means they've certainly seen papers in PostScript.

  32. Re:Link to a postscript file? by Kiaser+Zohsay · · Score: 2

    Actually, ghostscript created a PDF about half the size of the .ps file.

    -rw-r--r-- 1 kz None 239121 Apr 24 14:13 bhs.pdf
    -rw-r--r-- 1 kz None 433678 Apr 24 14:02 bhs.ps

    Of course, the PDF is Flate encoded internally, and the ps is a big fluffy text file, so the ps file would compress to well below the PDF size.

    --
    I am not your blowing wind, I am the lightning.
  33. procmail! [Re:The ultimate spam blocker?] by Styx · · Score: 5, Informative
    I use procmail, with weighted scoring
    First, I sort out mail from the mailingslists I read.
    Then, mail from friends, and people I correspond with a lot.
    Finally, I have a weighted scoring recipe:

    :0 Bh
    * -199^0
    #Assign an initial value of -199, mail gets filtered, if the score is above 0, at the end of the recipe.
    * 50^1 ^(From|To):.*@hotmail.com
    * 50^1 ^(From|To):.*@yahoo.com
    * 50^1 ^(From|To):.*@aol.com
    * 50^1 ^(From|To):.*@msn.com
    * 50^1 ^(From|To):.*@excite.com
    * 50^1 ^(From|To):.*@netscape.net
    * 50^1 ^(From|To):.*@yahoo.co.uk
    #Most mail to and from these domains is spam, so score it.
    * 100^1 opt-out
    * 50^1 opt-in
    * 200^1 OTCBB
    * 50^1 viagra
    * 50^1 zyban
    * 50^1 propecia
    * 75^1 FREE
    * 75^1 GUARANTEED
    * 75^1 LEGAL
    * 50^2 MILLIONAIRE
    * 50^1 100%
    #Words I only see in spam.
    mail/Trash

    This works quite well for me. If any spam gets through, I try to find some words, that I don't get in normal mail, and add them to the scoring.

    --
    /Styx
    1. Re:procmail! [Re:The ultimate spam blocker?] by bruckie · · Score: 4, Informative

      Or you could just use SpamAssassin, which is designed specifically to do this and has many more rules that have been created by others.

      --Bruce

      --
      There are 10 kinds of people in the world: those who understand binary, and those who don't.
  34. Since 5.0 it can by barzok · · Score: 3, Informative

    Message rules are very easy to set up and manage. No agents.

    1. Re:Since 5.0 it can by Milalwi · · Score: 2

      Message rules are very easy to set up and manage. No agents.

      Yeah, but you don't get any visual indication that there is new mail in your folders. You get told that there's new mail somewhere, but you have to go through your folders individually to find it. I have over 60 folders... do you think I'm going to use the message rules to automatically file them when I might note notice that they were there?

      If there's a way to provide visual indication of where the message got filed, I'm listening.

      Milalwi
  35. That's not necessarily the point.. by cheesyfru · · Score: 2, Interesting

    Spam filtering is one possible application of this type of tool, but the more useful involves taking the mail you *do* want, and sorting it into logical buckets. For instance, let's say work on several open source projects, belong to a couple organizations, and have a real-life job. You could toss a filter in your email that scans each incoming message and throws it in the proper bucket. This allows you to logically separate your mail to reduce confusion of each non-overlapping category.

    Procmail only goes so far, it's really only useful for simple header scanning.. I could really see a good scanner utility being a valuable tool. Maybe Google should share some of their technology.. :-)

    1. Re:That's not necessarily the point.. by 4of12 · · Score: 2

      Dynamic folders or views of your email would be a Wonderful Thing.

      I can't say how constraining it is to have statically defined folders which I have to move mail into based on my selection.

      Procmail helps to do this dynamically based on simple criteria, but when you want to have a particular piece of email show up in multiple views without having multiple copies, it really calls for associating named "views" of the whole mess with specific search and sorting criteria.

      That way, one view is "Latest Unread Messages" which has a particular message in it that might also show up in "Most Recent Messages about Project X" and in "Most Recent Messages from Boss".

      I'd love to have my email client show multiple views this way.

      --
      "Provided by the management for your protection."
  36. Re:Link to a postscript file? by nphillips · · Score: 1

    -rw-r--r-- 1 phillips cmb 181384 Apr 24 14:59 bhs.ps.gz

    And the winner is...gzipped postscript, which needn't be ungzipped before viewing.

  37. Re:Link to a postscript file? by MaxVlast · · Score: 2

    Alas, no. Adobe wanted ridiculous prices to license Display PostScript (DPS), the engine that NeXT used in the NEXTSTEP display system. (NeXT is a company. NEXTSTEP is an operating system.)

    Given the ridiculous licensing prices, Apple went a different way and created Display PDF for Mac OS X's drawing system.

    Ghostscript works just fine, but the lack of DPS is one of the reasons I still keep a NeXT cube on/under my desk.

    --
    There should be a moratorium on the use of the apostrophe.
    Max V.
    NeXTMail/MIME Mail welcome
  38. News for Nerds by lydon · · Score: 2, Informative

    Why are there so many people complaining about a PS link? The answer is simple: ./ is news for nerds, not for geeks.

    So while the average geek keeps his favorite postscript viewer handy, the standart nerd wonders about such an ancient format and does not know how to feed his acrobat viewer with it...

    Here is the solution for those irritated ones: try this piece of ancient software on the ancient adobe format, and you can miracously view it's contents!

    Have fun and keep your google handy!

  39. VM & EMACS by pmz · · Score: 3, Informative

    I have enjoyed using the VM module for Emacs. It allows sorting your entire Inbox into separate categorized mail boxes via regular expressions. Basically with one shift-A keystroke, my entire day's worth of mailing list stuff gets whisked away into a half-dozen different files. After this, I feel really sorry for people trapped in the Outlook dungeons!

    1. Re:VM & EMACS by brer_rabbit · · Score: 2

      could you give an example of your vm-auto-folder-alist? I've been using VM for quite awhile but I haven't tried this feature yet. Just curious how to set the variable to something useful.

    2. Re:VM & EMACS by pmz · · Score: 2

      The vm-auto-folder-alist is basically a list of which fields to scan and what to do with classes of entries in those fields. A simple example is:

      (setq vm-auto-folder-alist ("Sender:" ("mailing-list@domain" . "mailing-list.saved" ) ("mailing-list2@domain" . "mailing-list2.saved" ) ) ( "From:" ( "user@domain" . "user.saved" ) ( "your-e-mail@your-domain" . "sent_mail.saved" ) )

      A more powerful example using regular expressiongs:

      (setq vm-auto-folder-alist ("From:" ( "^.*@dot[.]bomb$" . "dot.bomb.saved" ) ) )

      This will take every e-mail whose From field matches the expression and save it into the file, dot.bomb.saved.

      I think this is by far the most useful and time-saving feature in VM, especially when subscribed to a high-volume mailing list.

  40. PDF Available Here by Jeffster98 · · Score: 1

    For those without postscript readers, a PDF version is available here.

  41. Enfish by cnladd · · Score: 1

    Check out Enfish Onespace for those of you running MS Outlook. Not only does it do great text mining of all e-mails, it does the same with contacts and with files on your hard drive (the professional version handles network files, as well).

    It's got a clean UI, but it is a bit hard getting used to. I've found it to be a great tool for finding info in a snap - I just enter a search phrase and instantly get a list of relevant e-mails, Word docs, spreadsheets, contacts, and even websites.

    And nope, I'm not associated with them in any way - I just like the product. :)

    --

    --
    Welcome to the land of the easily amused...

  42. I do this too + questions for other domain owners by Anonymous Coward · · Score: 1, Interesting

    I also use this technique for my externally hosted domain...I get all the mail addressed to any user in the domain, but its easy to set up mail client filters to remove those with are addressed To:, say, potentialspammer1@mydomain.

    So, if there's any possibility of SPAM, I just invent a new user. Unfortunately, I didn't figure this out quite soon enough and I have some users which get spam and real mail, which I can't afford to filter to trash - people buying their own domains (come on, its like $15 a year) should be thinking ahead.

    Also, its not as neat a set up as having my own POP server bounce back the message (which might mean you get off the spam list one day!). More importantly, filtering the To: field, doesn't help me most times, since spammers set the To: to "READTHIS" and use Bcc: for their spammies (is that a word!).

    ALSO

    Here's an unrelated question for anyone else who owns a domain like me, where they get a catch all POP box.

    How do you guys make sure people USE your nice domain name?

    In other words, its okay having a POP box, mail.mydomain.com, but you never seem to get offered the services of an SMTP server through which you can send your messages From: this nice address.

    I would hazard that most people rely on Reply-To:, which is all very well, except that not all mail clients respect it, and you may want to entirely obscure the actual From:.

    Of course, mail clients like Emacs and Mozilla make it easy to arbitrarily set your From:, however you then have to get this through whatever SMTP server you have available (and in order to block spammers and other pranksters, you will increasingly find that most will only send mail if the From: agrees with your user name).

    One of the reasons I moved to linux was so I could run sendmail and not rely on other peoples SMTP servers. The is okay at work, since we have direct internet access, but from home when I dial up, it doesn't work.

    I don't think my ISP likes to have people sending mail from their own computers, I get name resolution errors from sendmail when attempting to send email (but have no problem with DNS for web), so I think that perhaps the ISPs DNS servers refuse to give up MX records.

    Anyone else in a similar boat?

  43. What's wrong with PostScript? by Anonymous Coward · · Score: 2, Insightful

    Just use GhostView...

  44. Re:Link to a postscript file? by NewbieSpaz · · Score: 1

    Here ya go, in PDF format: http://www.kevindustries.com/bhs.pdf

    ps2pdf bhs.ps worked fine for me...

    --
    ------
    Random, useless fact: I type in startx entirely with my left hand.
  45. Re:Link to a postscript file? by tps12 · · Score: 2

    What is the difference between postscript and DPS? Any reason why DPS can't be integrated into X? The only effects of a DPDF renderer in OS X that I've seen are being able to view .pdf's without Acrobat and having vector-based widgets.

    --

    Karma: Good (despite my invention of the Karma: sig)
  46. pipe to mysql db? by yerdaddy · · Score: 1

    I've often thought it would be great if I could save my email to a mysql (or postgres, if you prefer) that would automatically parse the header and body into table fields. Then when you want to search it you can use SQL queries instead of the covoluted grep commands I use now.

    Doesn't seem hard to write. Anybody know of such a thing?

  47. Done already by Matts · · Score: 5, Informative

    "Perhaps even one of them Perl monkeys will quickly hack such a background tool."

    Been done already. Check out Mail::Miner.

    --

    Matt. Want XML + Apache + Stylesheets? Get AxKit.
  48. Re:Link to a postscript file? by tps12 · · Score: 2

    NM, here is this project that seems to be just that. Apparently Display Ghostscript is dead, but DPS lives on. Still don't see what the big whoop is.

    --

    Karma: Good (despite my invention of the Karma: sig)
  49. I Want Fewer Filter "Features" by kentborg · · Score: 2, Insightful
    Once I was at some internet tradeshow in Boston and every other booth seemed to be showing off their e-mail filtering features, each with one or more enormously complicated dialog box. Features! Features! Features!

    My reaction was to want an e-mail reading program that didn't require any filter configuration, though I imagined it would do well to be given a few hints, such as who my boss is, who my mother is, and who my wife is. Other than that, let the program figure it out.

    Imagine the canonical, old-fashioned secretary temp. She ('cause that's what the canonical version was) didn't have to know anything domain-specific to sort the morning mail. Magazines go together, bills go together, personal letters go together, etc.

    I imagine an automated version for my e-mail. Look at who it is "to" (am I on the list?), look at who is "cc"-ed (am I on that list?), look at who it is from (my boss, wife, or mother?), look at who else it is to (boss, wife, or mother?), look at the thread it is part of (is it responding to something I previously wrote?), look at the content (does it mention me, things I have written, my boss, wife, or mother?). Was it sent to a mailing list? Was it written by someone I have explicitly written to (once or many times?)? Was it written by someone who has previously sent me direct e-mail (once or many times?)? Those ideas are just the obvious ones, think of others. Think of more. (Does it talk about sex, credit card merchant accounts, stock tips, or Nigerian money?)

    Now take that and sort it by importance and similarity. Look for a way to present me in a descriptive summary, arranged in a hierarchy with a top-level of, say, 3 to 9 categories, a greatest depth no greater than, say, 4, and keep the sub-branching at intermediate nodes between 3 and 5--but don't max out all those dimensions at once, try to keep the total number of leaf categories to under, say, two dozen. Try to make more important items land higher in the tree and with few siblings, grouped with siblings of similar importance. (Maybe give an importance weight to each e-mail and balance the tree on that scale, that would float e-mails to me from my boss about my mother and wife really high with few siblings.)

    This summary needs to be integrated with a complete index of the e-mail so I can see how a message fits into a larger thread, how it fits into previous e-mails.

    I (the user) would need to tell the program when to make me a summary of my e-mail (e-mail reading is different when a lot comes in or just a little), and I want to be able to browse through old summaries, including deciding to see composite summaries or, say, the last several days, a week (or three), month, year, or 400 days.

    So I think it ends up being a 4-part user interface:

    List of summaries (which can be manipulated).

    A given summary.

    Exhaustive thread/date/subject/sender list (analogous to what every e-mail reader seems to have now). Note that this view could effectively be turned into an exhaustive address book. Frequent (favored) correspondents could be highlighted by me for ease in sending a new e-mail, and also to provide importance hints to the program. This is where I might say who my boss/wife/mother is.

    A body of a (or more) specific e-mail being read, written, or old e-mail (sent or received) being reviewed.

    And I could go on, but I won't. If anyone wants to write such a thing and wants to hear more, send me an, um, e-mail.

    -kb, the Kent who has been saving all his e-mail (including spam!) for a year or so, providing plenty of raw material to test any such program.

    1. Re:I Want Fewer Filter "Features" by Radix42 · · Score: 1
      I agree, I've been thinking about building such a "just do the right thing like a secretary" system for quite a while now.

      I was gonna email you about it, but your address isn't displayed (duh). Drop me a line at slashdot@arkhein.net

  50. It's already been implemented! by nikko · · Score: 1

    Please check out:

    http://homepage.mac.com/zoe_info/

    Zoe is way ahead of this curve.

  51. Check Out Phorecast by ciaweb · · Score: 1

    Phorecast downloads all your email into a database of your choosing; it is database abstracted using PHP's PEAR DB library.

    Phorecast is a web application written in PHP that combines email, calendar, and address book functions. It is language abstracted, so you can write a .tsv translation file for any language you like. Version 0.5 (on the way) improves these functions, and adds a todo list as well.

    Full disclosure: I wrote it, and I use it as my primary email client.

    --
    Try out Phorecast, open-source email, calendar,
    1. Re:Check Out Phorecast by statusbar · · Score: 2

      Looks great so far! However I guess I have to manually create the postgresql database tables....

      jeff

      --
      ipv6 is my vpn
  52. Re:Link to a postscript file? by MaxVlast · · Score: 2

    Well, the attentive reader would have noted that I pointed out that Adobe wanted a very high per-seat license. Apple wanted to pay a flat rate, IIRC, and the two companies didn't work it out. So Apple went a different way.

    DPS was used in a more fundamental way in NEXTSTEP. It was really amazing. There was true WYSIWYG, as the code on the screen was what was literally sent to the printer. Layout was really improved as a result, and you could mix postscript code with your drawing program efforts and see it previewed in a live fashion on-screen. It was easy to save documents in a portable fashion (PS), and a dozen other things.

    --
    There should be a moratorium on the use of the apostrophe.
    Max V.
    NeXTMail/MIME Mail welcome
  53. Re:I do this too + questions for other domain owne by Permission+Denied · · Score: 1
    I don't think my ISP likes to have people sending mail from their own computers, I get name resolution errors from sendmail when attempting to send email (but have no problem with DNS for web), so I think that perhaps the ISPs DNS servers refuse to give up MX records.

    This may also be a reverse DNS resolution problem. Check that your IP resolves to your hostname and that your hostname resolves to your IP. If not, some sendmail installations will reject your mail. Also, make sure your sendmail is sending out the correct hostname - eg, you can set up your machine so that it thinks its hostname is something.domain.com instead of some-long-crap-dsl-023-094.domain.com where something.domain.com is not an actual DNS record. This works fine for everything except when sendmail starts sending out emails claiming something.domain.com as originator.

    Another thing you can do is configure sendmail to send all mail addressed to "user+any_arbitrary_string@domain.com" to "user@domain.com". This is useful since I don't have to do anything to generate a new email address. Search google.

    I'll add that giving out a separate email addy for every company works beautifully. It also lets you know when some company sells your email address, something they will never admit to doing otherwise. I now get zero spam in my inbox.

  54. finding NEW topics by tswaterman · · Score: 2, Informative
    Many of these comments are missing the point. The paper is not really about categorizing your email.

    The main result in Kleinberg's paper relates to finding NEW topics that start to appear in the stream. Let's say you already have categorization filters (procmail, keyword filters, your own set of folder hierarchies, whatever...), but there's a new topic that starts showing up in your mail, or in your newsgroup feed, or on CNN. Klienberg's result is a way to find that the new stuff really is NEW, and you might want to group it up together, and make a folder for it. You could do that automatically, or by hand, but first you have to know that there's a topic.

    there's a bunch of other work in this area, what the NLP types call TDT -- "Topic Detection and Tracking"

  55. This can be done for free! by gregstoll · · Score: 1

    Check out sneakemail.com - it does basically this, but at their domain name, and you can set filters of particular addresses, or just delete them. Very useful idea, I'd definitely be willing to pay for it though...

  56. Intertwingle by geirt · · Score: 2

    jzw of Mozilla/Netscape fame have a hypothetical program called Intertwingle which is (Score:5,Interesting) ....

    --

    RFC1925
  57. Not new, but cool. by jefferson · · Score: 3, Informative

    There's been lots of work on auto-classifying email. I did my semester project in Machine Learning on this in 1999. It's a fairly simple study, but it seems like a Naive Bayesian classifier using word counts as features does a pretty decent job of classifying email, and does really well on spam.

    The paper is here here.

    J.

  58. wtf??!! by Anonymous Coward · · Score: 1, Funny

    who the hell gets so much email they need to
    mine for text, christ ??!! dont change your email filtering, change your pathetic life !!
    there are plenty of other things far more worth mining than TEXT

  59. Re:there is no way to win...Maybe there is... by wolf- · · Score: 1
    That all being said...
    As much as slashdotters hate lawyers...
    And most politicians...
    Maybe we should learn how to use them...

    Use the laws in place. Sue them for the costs associated with spam. Sue them when they break the laws that exist to protect consumers. Whether it is spam, telemarketers, or Best Buys. Stiff it to them. Out smart them. After all, arent we geeks and nerds? By the world's definition arent we all supposed to be smarter, or a step above average person?

    --
    ----- LoboSoft specializes in Digital Language Lab
  60. View the document online by Hew · · Score: 1
    --
    /cj
  61. kleinberg is rebel king!!! by complete+fallout · · Score: 1

    i'm taking a class with kleinberg right now and he's a great lecturer. if anyone is interested in algorithms of any kind, go read his papers.

  62. Re:Link to a postscript file? by tps12 · · Score: 2
    That sounds awesome. In my college days I always embedded latex in my xfig diagrams. I don't know if I'd necessarily want to go any less abstract than that (which isn't saying much, I know) under most circumstances, but it's cool that it's there, I suppose.

    Reminds me of the maps for the 3D network game for the Mac that Ambrosia made...Avara, I think? The maps were vector graphics, where different shapes meant different things and text inside the shapes was code. Very cool idea. I think there's still a lot of potential in the idea that source code doesn't necessarily need to be a simple linear text file.

    --

    Karma: Good (despite my invention of the Karma: sig)
  63. I do this too + questions for other domain owners by AFreeman · · Score: 1
    This may also be a reverse DNS resolution problem. Check that your IP resolves to your hostname and that your hostname resolves to your IP. If not, some sendmail installations will reject your mail. Also, make sure your sendmail is sending out the correct hostname - eg, you can set up your machine so that it thinks its hostname is something.domain.com instead of some-long-crap-dsl-023-094.domain.com where something.domain.com is not an actual DNS record. This works fine for everything except when sendmail starts sending out emails claiming something.domain.com as originator.

    Aha, is this why it works at work (where my hostname is correct and resolvable), but not at home with my ISP, where my hostname remains the same but could not be looked up?

    I just dial of over a modem, and its possible I configured that kind of perculiarly, because of wanting to switch between the LAN at work and my dialup at home.

  64. Re:Link to a postscript file? by tps12 · · Score: 2
    Gee, wouldn't that be illegally using their monopoly to muscle out third party developers? Why, if the OS had a PS viewer built in, nobody would every buy one! Businesses would go bankrupt!

    Haha, that is what I like to see. Some common sense once in a while.

    Some other transgressions: the Mac OS has forced the Apple menu on its users for nearly 20 years. Why can't I have a 3rd party menu? And sure people could download an alternative to GNOME terminal, but realistically who will exert the effort? And why don't I have a choice of who provides me with a tea timer in KDE?

    --

    Karma: Good (despite my invention of the Karma: sig)
  65. plenty of e-mail mining tools by j09824 · · Score: 2

    There are plenty of e-mail mining tools in development. This particular work takes one particular approach to mining the data. Whether this approach will turn out to be useful remains to be seen.

  66. Re:Link to a postscript file? by Pfhreakaz0id · · Score: 2

    this is perhaps the greatest example of slashdotter myopia ever. I don't give a crap about my karma, I just have to laugh at this AC:

    It shouldn't take two months to get ghostscript and ghostview. In fact, it comes with most modern operating systems.

    Clue time: 99% of people who've ever used a computer have never heard of either. If they click on the link above, they get a windows file box for "open with" and they wonder why the author didn't inlude a warning of what this strange file format was and what, exactly, they are supposed to do with this file.

  67. Re:and idea by ziplux · · Score: 1

    http://www.rhyolite.com/anti-spam/dcc/

    This and other projects do what you're talking about but most are not in wide use.

  68. Sneakemail: was "The joys of owning a domain" by mysta · · Score: 1

    I also have my own domain name but I'm limited to 5 forwarded email addresses. I wanted to do what you suggested a while ago but couldn't. Then I stumbled across Sneakemail and it basically did everything I had intended anyway.

    In a nutshell, you sign up for an account, giving only a contact email address (I use spam AT threewordslong DOTTY com). Once logged in you can create a new, randomized email address for each new web service that needs an email address. If one of these services spams or sells your sneakemail address you: a) know exactly who did it and cease further business with them and b) can filter on that specific email address.

    It's a great service and no, I don't work for them...

    --

    "Where is the wisdom we have lost in knowledge, and where is the knowledge we have lost in information?"-T.S.Eliot
  69. Re:Link to a postscript file? by t · · Score: 2
    Get current man!

    158213 Apr 19 09:41 bhs.ps.bz2

    t.

  70. Re:there is no way to win...Maybe there is... by foniksonik · · Score: 1

    There are several articles on /. concerning lawsuits against spammers.

    http://slashdot.org/search.pl?query=sue+sued+sui ng &op=stories&author=&topic=111&section=&sort=1

    --
    A fool throws a stone into a well and a thousand sages can not remove it.
  71. qmail dash-ext by main() · · Score: 1

    Qmail is good for this sort of thing. By default, a user receives everything at username-*@domain.com.

    So, I subscribe to amazon with username-amazon@domain.com, slashdot with username-slashdot@domain.com etc.

    You can then control the delivery location of mails to these recipients using .qmail-amazon and .qmail-slashdot files in your home directory and have a .qmail catch-all.

    This comes in handy for filing mailing-lists away, filtering out spam etc. Its also interesting to see who's sold you down the river to spammers, I recently started receiving spam to username-bsdtoday@domain.com... bastards!

    Cheers,
    Si

    1. Re:qmail dash-ext by dlc · · Score: 2

      Most of the major MTA's will do this nowadays, but with a + rather than a -. I know sendmail does this, and am pretty sure about postfix and exim as well.

      Look at this reference, for example.

      --
      (darren)
  72. Horrible interface. by arafel · · Score: 1

    I could list it all here, but it's much more efficient to just point people at:

    http://www.iarchitect.com/lotus.htm

    (Which is a site that everyone should read before doing UI stuff.)

    Sample of one of the "best" bits:

    Judging from the number of visitors who have mentioned it, the process of copying messages in Notes is perhaps its worst interface "feature". Apparently, when mail messages are copied from one folder to another, the message itself is not copied; Notes creates a "reference" to the message. Unbeknownst to the user, if you delete the reference, Notes will in turn delete the message itself. Similarly, deleting the message will cause all references to it to also be deleted.

  73. Text-Mining your bookmarks? by lvirden · · Score: 1

    I'd like to find something like this - where
    as I browse across a web page I could active
    a program that would look at the page, and suggest a series of folders that appear to be relevant.
    If I agree, I click okay and go on. If I think a category is unnecessary, or missing, I would
    have the option of adding a category.

    I see this as a parallel need to the mining
    of the email.

    --
    URL: http://xanga.com/lvirden > Quote: Saving the world before bedtime. Even if explicitly stated to the contrary, n
  74. SWISH++ is a good mail indexer by pauljlucas · · Score: 1

    SWISH++ (my search engine) specifically knows to index mail/news files (including text, HTML, RTF, LaTeX files) and attachments of any of those (in quoted-printable or base64 encodings). It can also index any other kind of attachment via external filter programs. A procmail recipe for auto-splitting incoming mail is included in the distro. I also believe that my statement of SWISH++ being the fastest open-source indexer is accurate.

    --
    If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
  75. Re:Link to a postscript file? by ansonyumo · · Score: 1

    Yeah sure, embed it in the OS. More proof that any notion of what constitutes an "Operating System" has been completely destroyed by Microsoft's marketing department.

    Dude, it's real simple. Install ghostview, set up as helper app in the browser of your choice for application/postscript.

    Bundle with the OS. Feh!

  76. SpamBouncer by gasull · · Score: 1

    SpamBouncer is a set of procmail recipes to filter spam.