Slashdot Mirror
DSPAM v2.10 Released
Nuclear Elephant writes "DSPAM v2.10 is finally available, after four months of development. This is the first stable release to include Bayesian Noise Reduction which was recently mentioned on Slashdot and in Wired News as an algorithm providing accuracy levels as high as 10x that of a human. Some other new features include Neural Networking - which finds nodes in a network that are contextually similar to form a decision matrix, Global Filtering - which provides SpamAssassin-like out-of-the-box type filtering for new users until they build up their own wordlist, Automatic Whitelisting - which automatically learns who your trusted senders are, and many other optimizations and enhancements. Head on over and download the latest tar ball."
now the question is.. how hard is it to get it to work with cpanel
Selling software wont make you money, selling a service will.
I tried several incarnations of dspam over a period of about 6 months. It was a pain in the butt to install, required a massive amount of training, and required you run a web server in order to have the point and click training capability.
I eventually gave up and tried the CRM114 Discriminator:
http://crm114.sourceforge.net/
It was MUCH easier to install, MUCH easier to maintain, and has the same or better level of accuracy. I used to get 100+ spam messages a day and now I'll get maybe 1 or 2 a week that sneak through (after only a few weeks of training on errors only).
Well I haven't used it in a year or two. But I had require authorization on from day one and still got request for authorization spam. Where some pr0n/webcam botperson requests authorization with a little ad in the request.
I don't have any clue what the solution to the spam email problem is but I believe it'd have to be a pretty major evolution.
FYI, modern MRI scanners use bayesian noise reduction during image processing. I used to work in a MRI research laboratory, and our director had pioneered the application of Bayesian noise-filtering algorithms in post-processing of image data.
Oddly enough, our director of research was notoriously difficult person to schedule a meeting with. Makes me wonder about 'unsupervised learning'...
Okay, so filtering on the receive end is fairly commonplace - but what about filtering close to the sender?
(1) Force all ISP customers to use their own SMTP server (block all port 25 access to external addresses).
(2) Set up an outbound SMTP server for all ISP customers to use - but include a spam filter that rejects sending the message if it considers it to be spam? It would also give instant feedback to the user - the mail client would immediately report the error.
Then the spam wouldn't even be transported over the net, saving vast amounts of traffic on the internet backbones. This action could also potentially kill spam overnight.
Nice idea.
Make another email worm like MyDoom(call it MyDick/MyAss etc), with misleading title/body that sounds like those spam mail that enlarge/shrinks various human anatomy.
People who reply those mails will be activated the virus and make his/her computer unuable. Soon nobody will have the gut to open spam mail anymore.
DSPAM is one of these statistical filters (like spamprobe and CRM114) that can perform virtually perfect filtering of spam/non-spam you receive.
Now that you are free of spam yourself, may I suggest that you take it one step further and share your data with the anti-spam community; the WPBL project lets many users report the IPs sending them spam and non-spam in realtime using a couple simple scripts installed in procmail.
Our central database then publishes a real-time list of spam sources (the IP blocklist). Unlike spamcop, WPBL is entirely based upon automatic decisions made by statistical filters, 24/7. The resulting blocklist is already used by many ISPs; and you can also use it to block spamming IPs at your own server.
I see all my fellow slashdotters saying (over and over again) that spam filters should be server side, because otherwise you are still paying for the wasted bandwidth. This is a very powerful argument, and I tend to agree.
However, there are two things that make me nervous about this approach. First of all, if I miss even one email, no matter how innocuous, because my ISP installed filters, I am going to be pissed!
"Man, you missed it, the party was a blast!"
"What party?"
"Didn't you get the email?"
With a client side filter, at least I can look through the 'spam' and find the gold nuggets. If my ISP filters for me, and I miss a legit email, I'm just SOL.
Secondly, all of the best filters are for linux. Ask me if I run Mozilla (for windows). I will tell you, "HELL yes I do". Is it anywhere close to 90% effective for filtering spam? Not for me! Is it 100% effective in letting my legit mail come through? Not for me! The browser has stopped 99.9% of the popups tho.
Anyway, long ramble short, give me something that's good on windows. Do I have to write it myself? I've been thinking of altering Mozilla to incoporate the latest anti-spam technology, but, man, I just never have the time these days.
Anyway, good work on the part of D-Spam, nonetheless. Kudos to your bad selves.
WWJD? JWRTFA!
Exactly my thoughts! Can we include something in Slashcode that automatically filters the GNAA and goatse trolls? Perhaps as a user-configurable option.
... I wouldn't want jokes based on the "$PROJECT is dying - Netcraft confirms it!" troll to be filtered out!
Some Bayesian approach ought to do it
The solution to the spam problem is simple yet elegant - gambling.
:)
Every time you send an email you place a small wager on the line that the recipient wants to read your message. Something like 1 cent. If the recipient doesn't mind your message then they don't redeem your offer and it doesn't cost you a thing. However, if you're sending spam then the recipient cashes it in (or perhaps it is used to cover overhead costs of this system).
If you send a legitimate email and somebody decides to be a jerk and cash it in then you're only out 1 penny. However, if you just sent 2 million of those unwanted emails you're screwed.
This is better than the "small price" schemes because it doesn't cost anything. Well, unless you're A) a spammer or B) sending email to dickheads.
This wouldn't replace SMTP, it would just be a layer on top. If you sent an email and you participated in this system then a third party would sign your messages and you'd be get a special verifiable header that the recipient could then treat as "likely ham".
Anybody have a better idea? I didn't think so.
No matter what technology it uses, neural nets, b-trees, recursion, tinkertoy logic, smell-emitting diode, leaky junction zener transistor, steam-powered aeolipiles, it only automagically presses delete, which is a pretty lame way of fighting spam.
It's a lame way of fighting spam, because, we STILL have to pay for the fucking spam bandwitdh; we STILL have to pay for the goddammed disk space used by the spam; we STILL have to pay for the bloody time lost transmitting the spam; we STILL have to pay for the extra ISP infrastructure to carry those spams.
Naaah. Spammers should be eradicated from the Internet, and the best way to do so is to completely BLOCK networks who host spammers (no matter what service), in order to force the collateral damage to whine to the ISP or simply vote with their feet.
The last two weekly builds have had this turned on. Further information is in this thread.
The bugzilla number for this feature evades me at the moment. I've only used the windows builds provided, but it shouldn't be too difficult to make your own linux build with this stuff turned on.
... if there was some way to plug tools like this into Mozilla directly so that you could expand on its built in junk mail detection with something more powerful.
File under 'M' for 'Manic ranting'
The authentication is useless even if implemented - you want to receive email from strangers, that's what all businesses are doing. If you are not one of them and only converse with your buddies, make a whitelist and be done - no spammer will guess your friends' emails.
Permissions to send email are also troublesome. If they are automated, then spam robots will be written to ask for permission first. If they are not automated... but how would you know if some random "John X. Frisby" <jfrisby@big.provider.net> is really who he is, and the matter he wants to discuss with you is not a bug in your Loafizer 0.99 script for your bread making machine, but a placebo enlargement pill. Additionally, permissions delay the mail exchange, which is bad for business.
There are ways to block anyone you don't want, and all other senders are legit (until they spam you, that is.)
So the problem is quite different, as you can see. There is a free channel of marketing, and spammers will be using it until it remains a) free and b) channel. Remove any one of those two, and they will close up the shop.
Actually, there is another solution. Everyone could simply respond to the spam they get. That'd quickly ruin the ecomonics of spammers.
I don't get SPAM. I don't have SPAM filters. How is this possible? Simple. I create a different e-mail address for any new untrusted entity that I have to provide one for. In the beginning I took advantage of being able to alias all e-mail for non-existent mailboxes (basically, *) at my domain to my primary account. It seemed to me an obvious and simple approach. Whenever I needed to provide an e-mail address, I just made one up, and it was forwarded to my regular Inbox. In my opinion, at that time my ISP was more "sophisticated" than most. Since then I have moved to hosting all of my domains on my own co-located server which runs Exchange 2000, thus complicating things. Now I have to actually add any new aliases that I want to use into my user account. I know of at least one product out there that can handle non-existent addresses and forward them to a specific account, but it is rather expensive for a feature that should have been built-in from the beginning (althought I'm not aware if the new Exchange can do this out of the box). Not to mention that someone with the proper knowledge and skills could make a similar add-on in relatively short order, but who ever has the time? The point is that you have to consider when and where you give your e-mail address out, and the possible consequences therein. It's not altogether different from giving out your phone number (especially if you are unlisted) or even your SSN.
Ah, but every now and then I get a "User has requested to add you to their contact list..." in my ICQ and they just put the spam in the notification reason box. I see the same thing with automated request system; they'll use the request process to pass the advertisements in to you.
...
Call me a cynic, but I think we're dealing with an inherantly unsecure system. As long as you have one mail server out there forging message headers, you can't trust the path back to the sender. Like abstinance, Whitelisting may be the only way to block 100% of what you don't want. But then you might be blocking an email from your third cousin someday who decides to email you out of the blue. The happy medium is the automated filter, like Yahoo's... but I'm noticing that this past week spammers have figured how to slip message through that one too
Not to underestimate the effort, but with extensions this has got to be easier than I think it is. Ruven Gottlieb's Purity-of-Email project is out there to integrate Mozilla mail with CRM114.
You *WILL* get spam my friend. I've been doing this for almost 20 years (admin) now -- and have specifically used aliased accounts for various reasons over the years as you are doing.
... it only a matter of time before you're screwed.
:). Bill's idea of email stamps, well, hahahahaha...
:).
:)
Wait... You'll be interested to know that the biggest problem with the spam coming in comes from virus infected Windows boxes. They send it. They harvest the users Outlook address book. If you ever end up in somebody's Outlook box
I chuckle at the whole Exchange thing. You pay for that?
I personally pay to have a fixed IP @ home and run a old Linux box. A lot of aliases I've used over the years (and some blatantly used to harvest) all go to some local account that processes the spam. Upon receipt -- mail the wrong account and sorry, but you're blocked (unless white-listed). White-listing can come from valid already received email -- but I work everything based off of IP. My hope is that the registered MX host(s) or any valid listed server by the authenticating DNS server will be the type of scheme that's re-implemented (or more to the point SHOE-horned in real soon
Over the last decade I've now got 380 aliased harvesting spam address' in use -- two valid email accounts @ home (my wife and myself) which is on my own IP with my own domain. I pay $5 extra a month above my broadband (10Mbit [yeah, solid] wireless) -- how much do you pay for that Exchange box?
I've run this type of setup through many offices scaled to dozens of email servers -- and the beauty is they also talk to each other sharing block/white-listed address' as needed. Wait -- you will get spam. Filtered through my account to I'm seeing 80 something that got in -- 2,164 blocked IP's [today], 380 harvested address', and 48 for various other infractions (attempts to relay through me, from a country where I know nobody, etc
Statistically (yeah, they all get nmap'd back)? 96% Windows based.
I give my email to friends. I have a work email that anybody that knows how to call me can have it. I even print it on my business card. No, I wouldn't post it to USENET or even here -- but it's still "out there". My unlisted phone number, OTOH, anybody can have. 847.854.0048. It's always busy and one channel of my ISDN home line. The other channel routes to the house for two phone lines (or Internet backup if and as needed) and is automatically unlisted and unpublished (at no cost since it is a "data circuit") -- and no, I'd rather not post that either.
Exchange? Never!
You are -truly- naive to think this kind of solution would even be possible to implement
Don't be so stupid. This solution would be entirely possible to implement. Large email providers - such as Hotmail, Yahoo mail and Earthlink - already have spam filtering controls for inbound email. It's not much of a leap to filtering outbound, especially considering that providers as large as Earthlink already have port 25 blocking in place (mail goes through their servers, just nothing intelligent happens to it on the way).
The big question, and the one that you should really have emphasised rather than throwing a temper tantrum, is this: would it would work and would it be effective?
Would it work? Yes. It's using the exact same filtering methods that have been invented for client-side - just doing the filtering in a different place - and pushes the responsibility to not send spam back onto the sender. If the scary filtering works for receiving, it should work for sending too.
Would it be effective? Indeterminite.
The biggest problem I can see is that it could break the feedback loop on what is spam. At some point the user needs to say "yes, this is spam", or "no, this is not spam". I'm not sure how that piece of the puzzle would be solved - but technology for the solution is a damn sight more complete than your blanket "it's absolutely impossible, the world is flat" statement.
And once completely working, it would definitely prevent abuse of any systems that it was attached to - and probably save the large networks time and money that they'd otherwise perhaps spend chasing spammers down.
If the idea became popular - there would be undoubtedly occasionally be problems with false positives - but if it's on the sending side, the sender can be notified immediately by the server giving an error message back before the client has even finished sending the message.
It would then be up to the smaller ISPs to implement the solution themselves - or face slowly being cut off from everyone else who disallows spam onto their networks.
The bottom line is, "No software can ever be better than a human in defining Spam".
That is true if the human is looking at a single email. Now give the same human a mailbox with 2000 messages, 1000 of which are spam (by his standards). He won't be thinking twice about calling the message spam and getting rid of it, so he's bound to makea couple of mistakes (happend to me a while ago, one of my friends has her email @ladymail.com and the Subject was in Latin - random to me. I called it spam befere even reading Hello,...).
The claim that is being made is that if this poor man overlooks 10 spam emails, dspam will only overlook one. Whether that's true or not is another thing, and would again depend on the circumstances, but I believe it would apply to me.
I recently started using bogofilter as a replacement to spamassassin. The reason for doing this was curiousity and the fact that the spamassassin regex process will always be following the spammers, not preceding them. The result is packages supplied by distros are quickly outdated and ineffective.
I have been using bogofilter for one month and have trained it to such a point that my weekly spam misidentification is well below 0.1% with proper training and configuration. And it's processing time is well below 1 second per message on a VIA EPIA 533 cpu (slow, ok?)
The net outcome of this is that I have found something which is highly adaptive to new spam techniques, extremely effective, very fast and light on the resources, and is at the point now where if just works.
The idea that they, DSPAM, will provide you with a pre-defined training set. That's damaging. What if you are an oral surgeon? You'll never get any email!
I've been working intensively on spam and have come to a few conclusions about spam filtering and such that I just have to share.
It will never go away. Even if you can proper regulate and control it, spam will never go away. No matter what anyone does. If the US constitution is to remain intact you can't remove spam just as we haven't been able to remove advertisements from radio, telephone, or television. And just like you can't get rid of pornography. It's all Free Speech.
It's also carrying a lot of money.
What will happen is that corporations, in the name of reducing spam, will lock up mail servers such that you have to pay them a service fee to send email on top of your connection fees paid today. Microsofts recent movement into the arena shows that thier is a motivation to make money out of spam/email.
In a few years, we'll pay for our email and we'll still get spam