Slashdot Mirror

SpamCop is not dead. It is still up and running and the free blocklist is a great part of your anti-spam arsenal. Compare RCVD_IN_BL_SPAMCOP_NET to the other free options using SpamAssassin rule vetting stats and you'll see it's among the top performers. ("S/O" is a measure of relative precision, "SPAM%" is recall.)

Unlike the other DNSBLs, SpamCop also reports spam back to the networks that sent it (with filters to deal with spammer-friendly and negligent network operators, either of which might ignore or even pass on the heads-up to spammers rather than disciplining them).

In particular, SpamCop did well against this Necurs attack but it does not fare as well against hailstorm/snowshoe spam attacks (which IP reputation doesn't help combat). IP-based DNSBLs aren't anywhere near as effective today as they were ten years ago, but they're still quite worthwhile. That said, you're right in that the best ones cost money.

I feel happy, oh so happy. I don't want to go on the cart.

Does your product even filter spam? How? When a virus is delivered via email, your tool does nothing. When an IP link is clicked, your tool does nothing. Against a 419er's solicitations, your tool does nothing. When faced with a fresh malware link, your db may be up to 12h out of date and lack an entry for it.

You claim your product is accurate. Prove it. Spamassassin is free and open, you can look at the source. Its efficacy is demonstrated daily. There are plenty of third party reports over the last 10+ years demonstrating its utility. SA has millions of users and hundreds (at least) of contributers. How about your tool?

You claim your product is fast. Perhaps it is leaner than Spamassassin (with SA's Bayesian filters and regular expressions, as you note, it almost certainly is), but Spamassassin runs on servers. It consumes zero disk, zero memory, and zero CPU on user systems.

I put my Spamassassin developer status on my resume. Sure, I'm not the sole dev of SA, but that's a feature. I can work on (and lead) a team. If I go away, the project will continue without me. Do you put APK Hosts File Engine on your resume? Whatever would people think when they search for it? ... oh, never mind, a quick search yields your full name attached to these shenanigans anyway. How then could my Fortune 100 employer hire you into any team?

Oh, and so you know it's "me" to the same level of clarity as I have for you,
Khopesh

I am subscribed to a couple of mailing lists, I could never manage all that traffic with webmail. Instead I use a full-featured MUA called mutt with procmail to sort things out and handle spam with SpamAssassin.

And wherever I am on the Internet, I can always ssh home and grep around all my mail archives to find something back. It's all searchable online, so to speak.

The message was undeliverable due to the following reason(s):
Unsolicited Commercial Post

X-Spam-Flag: YES
X-Spam-Report: Detailed Report
SPAM: -------------------- Start SpamAssassin results
SPAM: This post is probably spam. The original post has been altered
SPAM: so you can recognise or block similar unwanted mail in future.
SPAM: See http://spamassassin.org/tag/ for more details.
SPAM:
SPAM: Content analysis details: (11.099 hits, 5.0 required)
SPAM: BILLION_DOLLARS (1.0 points) BODY: Talks about lots of money
SPAM: HTML_MESSAGE (0.001 points) BODY: HTML included in message
SPAM: STRONG_BUY (2.498 pints) BODY: Tells you about a strong buy
SPAM: RISK_FREE (1.0 points) BODY: Risk free. Suuurreeee....
SPAM: LOW_PRICE (1.0 points) BODY: Lowest Price
SPAM: FROM_HAS_MIXED_NUMS (0.3 points) From: contains numbers mixed in with letters
SPAM: ONCE_IN_LIFETIME (1.8 points) BODY: Once in a lifetime, apparently
SPAM: HOME_EMPLOYMENT (0.6 points) BODY: Information on how to work at home (2)
SPAM: SPAM_PHRASE_21_34 (1.9 points) BODY: Spam phrases score is 22 to 30 (high)
SPAM: DATE_IN_PAST_24_48 (1.0 points) Date: is 24 to 48 hours before Received: date
SPAM: [score: 11.099]
SPAM: -------------------- End of SpamAssassin results

Yes there is an apparent SA bug that occasionally causes the SURBL lookups to FP. Strictly speaking that's almost certainly not the fault of SURBLs which are the data source in RBL form. If you are using Windows, you may want to upgrade to the latest version of ActiveState PERL, which apparently fixes a possibly related Net::DNS buffer overflow bug. I don't have a reference for the PERL bug, but here's a SpamAssassin bug mentioning the fix.

I'm not sure this (SURBL) is very well implemented, though. I get a few false positives b/c my DNS flakes out or something and then non-BL'd sites are reported. Bug report here.

Linux and FOSS is affected by Windows viruses.. Lets see.. because of Windows viruses, my Linux based mail servers have had lots of great FOSS software developed to help combat the issue. On the down-side, many of these Windows viruses have also greatly affected my Linux systems due to DDOS attacks that have origins pointing back to viruses and other malware that has infected Windows boxes.

However, in my opinion one often overlooked problem is what to do with messages that are flagged as spam. You don't want to generate a bounce message, because 95% of the time it will go to an innocent third party (spam is forged). On the other hand, silently discarding it or putting it in some spam folder one never looks at is not a good idea either. Lately I've been just refusing to accept mail from the client if the message is identified as spam. Mail Avenger lets me do this, and it seems like a good compromise.

MailScanner

MIMEDefang

SpamAssassin

You can look at the other options we considered here.

Having green and red messages, while symbolic, makes for a much less pretty logo. The green, blue, orange colours are nicer. Furthermore, you can't necessarily tell good from bad at a first glance from the envelope. :-)

And yeah, the arrow only pierces one of the three envelopes.

Once you decide to set up a mail server, you'll probably want to add spam and virus scanning. Both are CPU- and memory-intensive. A consumer router probably won't have the horsepower for that. (I'm using MIMEDefang (a sendmail milter), SpamAssassin, and ClamAV on my box.

I'd like to see something like FOAF used for whitelisting. I posted a SA bug about it, and then there's things like Trust and Reputation in Web Based Social Networks . I think this looks like a workable approach.

...is google@yahoo.com, just because i think it sounds ridiculous.

This talk of email abuse reminds me of the spam filtering system of a person i go to school with:

you send him mail. If not on his "Acceptable Mail" list your are directed, via an auto response to his webpage, whereupon, you submit your email address for his approval.
The problem exists when 2 people employ this method, as if your auto-response's email address is not in the original sender's acceptable mail list, an endless (or quickly auto-abandonded) chain of emails back and forth betwixt auto-reponders is created.

I thought this was a bit silly.
Waive That elaborate nonsense. This works wonders for me

--Cheers

Heck I get at least 900 virus emails everyday sometimes over 2000 a day.

Thanks to the guys over at Clam Anti Virus and MailScanner most of these get caught at the mail server.

We have a daily humor mailing list with a few 100,000 subscribers and every time a new virus comes out we get blasted from all the unprotected windows/outlook express users.

To make sure we don't get infected and send out virus to all the users we use FreeBSD for our desktop OS and Evolution as our email client.

Oh and then there is all the spam we get sent, thanks to SpamAssassin for filtering most of this out.

Furthermore, SPF enables domain reputation systems such as GOSSiP (currently under design) which enable domain's to be given a "spaminess" score based on their previous behaviour.

That's interesting! I'd like to plug two bug reports of mine (I wish I had time to hack, but I haven't). Friend-of-a-friend makes great start for a reputation system, at least for whitelisting people you know well.

So, I there's a Spamassassin bug on this, and it has generated some interest.

Now, the problem is to generate FOAF-records easily and reliably, and for that, I suggest for example enabling KAddressbook to export them.

not all bulk mail is spam. spam assassin gives 2.4 points if it finds anything that looks like a unique identifier for X-Sender, and another 1.4 points for anything that looks like a tracking image or tracked link.

that plus the points for any non-safe html colors or any html at all, SA effectively tags ANY bulk mail as spam!

I don't agree with all of its default settings either, but then, it's simple to adjust the scores in any way you like.

3.0.0pre1 was made available last week.

It will apparently take another month or so to finalize the weighting of the rules.

I've put 3.0.0pre1 on a production system that filters ~350k messages per day. With some tweaking of the RBL, bayes, and AWL rules, it is much (~10%) more efficient at tagging spam than 2.63, which I'm running on a parallel server that also sees ~350k messages/day (load balancing is your friend).

More info: http://www.au.spamassassin.org/full/3.0.x/dist/bui ld/3.0.0_change_summary

For those looking for the official spam assasin site here it is

The link in the text goes to some search page

Like the AC said, put Exchange behind a proper MTA. Keep your exchange server inside the firewall for the suits to fiddle with their calendars and crap. Setup Postfix, Qmail, Sendmail, Exmim or some other MTA as your internet-facing email server. I use Postfix with Amavis forming a nice interface to Clam-AV and SpamAssassin. I don't run exchange though. Can't help you there.

CRM-114 and DSPAM exhibit substantially inferior performance to the other filters, regardless of threshold setting. Both exhibit substantial learning through outthe email stream, leading us to conjecture that their performance might asymptotically approach that of the other filters. From a practical standpoint, this learning rate would be too slow for personal email filtering as it would take several years atthe observed rate to achieve the same misclassification rates as the other systems.

This is interesting considering the harsh words the DSPAM author directs towards SpamAssassin in the DSPAM FAQ. In contrast, I think, the SpamAssassin developers say they are interested in testing the "dobly" noise reduction technique that DSPAM employs, see SpamAssassin bug 3078.

And then be prepared to continue filtering out spam (although with my setup, of the 100+ daily messages that would get into my inbox without filtering, I now get about 10, all marked as spam, with the rest getting blocked by the rbl lists and some custom rules).

Every day I get dozens of delivery attempts at an address I used to run a listserver on, which has been invalid since 1998. No human has *ever* been behind that address. The spambags do not care about invalid addresses.

How true. But you can use that against them!

I have several addresses like that. Some were accidentally created for me on other sites by scripts like wpoison. Others are spammer-specific mutations of my real address. And I have a number of old addresses, like special ones generated for Usenet News posts five years back.

Now I feed them all into SpamAssassin's Bayesian classifier. I even looked in my logs to see the 100 most common choices for dictionary attacks and feed those in, too.

Now, thanks to the spammers, I get a lot less spam in my inbox! Yesterday's score was 356 messages fed to the trap, 145 spams in my spam folder, 1 spam in my inbox, and no false positives.

Regrettably, there are no adequate instructive manuals for http://www.spamassassin.org

Apparently, to be certain about the possibility of false positives you have to go through all the messages at some point anyway.

I admin a small mail server and trap mail with spamassassin. On average we get about 100 spams per day for only 7 active email accounts.

To check for false positives, I review all of the trapped mail from time to time, and I'm starting to get discusted with the whole spam thing. Here is some of the crap that I get:

GET VALIUM AND MANY OTHER DRUGS 4 L j
China World Trade Corp making major breakthroughs
GET YOUR UNI.VERSITY D|PLOMA tqlylsrvi
Take advantage of low interest-rates!
Powerful weightloss now available where you are.
Fwd:re:Home del.very on all meds.


I'm also starting to get amused at how easy it is to identify spam with enough rules in spamassassin. These guys suck at sending mail.

I just can't believe that some people actually respond to some of the mails. The ones that get me are the mortage and loan ones. Who in their right mind would give all of their financial information to someone who randomly emails you with junk like this:

HellWo dear hom5ke oUwn5er,

We have b\eeQn notified that y<oiur mortgMage rate is fixed at a
vet6ry hoyigh interesNt rate. Therefore yhqou are current overpay[ing,
whick7h sumsRs-up to twhXo+usaEAnds of dollXLarws an5RnudPal0ly .

Lugckily for you we can [1]guoGaranteze th@e lowest r{ates in the U.S.
([2]3.50%). So hSVu=rry beQ0caNuse the ratHe f.orecast i|s not
l;oobrkincNg good!

Thesgre is no obligations, an6d i^t FykREAE

Locnk on the 3.50%, even wHSith bad credit3A!


Where all of the urls are behind a yahoo redirector, and its barely legible from all of the obfuscation techniques?

Fuck spam specific laws, it just should be illegal to try to get money from someone under the pretense of deception. Clearly mails like this are deception, and its getting out of hand.

http://www.spamhaus.org/sbl/howtouse.html
http://www.spamassassin.org/full/3.0.x/dist/rules/ 25_uribl.cf

Once an actual human person has read and acted on the mail, they should be able to mark it "official business" and/or move the email into an "official business" folder which does get kept as required.

SpamAssassin Prehistory: filter.plx

Before there was SpamAssassin, there was Mark Jeftovic's filter.plx. This was a 'context/keyword spam filter', which came up with the basic scheme of what we use in SpamAssassin: namely, named rules identifying spam-like 'features' of the mail, each rule has a score, and once the number of 'strikes' goes above a certain threshold, the mail is marked as spam. And written in perl, of course ;)

I (Justin Mason) used this for several years, adding a few small snippets of code; eventually though, the code was getting a bit stale, and Mark seemed busy on other stuff, and I had a few thoughts about some improvements I could do with a total rewrite ;) -- so I decided to recode from scratch under an open-source license, and that was SpamAssassin.

Unfortunately the original site at http://antispam.schmooze.net/filter/ is no longer up, but the Internet Archive has a snapshot of it from December 1998 here.

Also courtesy of the Internet Archive, the change log of filter.plx is here, spanning June 1998 to August 1997.

Finally, Mark was kind enough to dig up a source code tarball for filter.pl-1.02d.tar.gz (20k). This is the 1.02d release, February 1998.

Whatever you do, don't actually run the code -- spam nowadays looks nothing like spam did back then, before e-mail clients grokked HTML. Plus I don't think Mark wants to get bug reports at this stage, it's been 5 years ;) This page is here instead to document the history of this project.

--j. Jul 14 2003 jm

The only argument that I've heard that makes any sense is if someone is against Gmail beacuse of this ad thing, so they dont sign up for the service, but then all their friends do so when they send email tot hem, their emails are scanned for content, even though they're not signed up with the service.

Hmm, messages scanned for content by the receiving mail server... nothing at all like these MailScanner and SpamAssassin packages that we have installed on our mailservers, that scan every piece of received mail for content. :)

(and we don't even tell the sender that we're "reading" their mail!)

So many people want capital punishment for murderers, yet the real serious crimes are commited by corporations. You'll never see people demand that a CEO of a major corporation be executed for knowingly distributing unsafe products that resulted in the death or serious injury of many people. Now playing devil's advocate one could argue that spammers and virus writers provide jobs for many people

The real problem is compliance, until 99% of mail servers provide this data, I can't reject mail from non SPF listed domains.

There aren't many tests which are perfect spam identifiers with no false positives. You should use the SPF compliance as part of a scoring scheme. Messages that fail SPF are more likely to be spam than messages that pass, so they get a higher spam score. If the score exceeds a threshold, mark it as possible spam. If it exceeds a higher one, delete it unread.

This is the strategy Spamassassin allows, and it works really well, especially if you let the scoring be adaptive (i.e. use "Bayesian tests").

> Not that any of this is going to stop me from getting a gmail account with my favorite username once it goes live. Be nice to have a big name webmail account that doesn't have a bunch of numbers in it :)

Especially since spamassassin takes points off for those numbers. ;)

Actually, (after quite a bit of searching, mind you) according to this Fees the fine, while small, would not be insignificant.

They're talking $20 per complaint, after your "free" complaints per month. Which, for the "low" volumne bulk sender( less than 1,000,000 per month), is 1 complaint per month.

So, for the above example, 10 complains - 1 free complaint * $20 is $180. The sign up costs are $375 Application, $500 license, $500 bond.

So after your first month, you've spent $875, bonded $500, e-mailed 500,000 messages, and lost $180.

And somewhere else, I thought read that if your bond drops below half, you have to replace it. So they've effectively created a charge system for spam.

This would be quite nice if they donated some of the bond money to, say, the SpamAssassin Development Team, or maybe SourceForge.

The SpamAssassin test USER_IN_DEF_WHITELIST checks to see if the sender is in the list of companies that are on its built-in white list. Network Solutions, internic, register.com, nytimes.com, amazon.com, mypoints, paypal, the FT, Palm, Handspring and others are all on it. They don't sell access to it, so it is not the same as what Microsoft is doing. It is similar, however, in that some companies get a free pass (well, up to -15) for any mail that they send out.

1. Is The Subject Capitalized Like A Headline?
2. Does the subject contain too many non english-alphanumeric characters?
3. Is the subject a duplicate of another subject in the same POP retrieve job?
4. Does the subject contain 4 or more spaces anywhere?
5. Is the subject more THAN HALF CAPITAL LETTERS
6. Does the mail have no subject at all?
7. Does the su-bject con+tain obvi!ous obfuscation?
8. Finally, does the subject hit on the blacklisted words?

If everyone followed your moronic thinking then cops would be pulling over every ryder truck thinking that there was a bomb on it!

Stops unwanted mail dead.

Finally be able to stop bitching about your inbox.

100% Free.

Small catch: you need your own mailserver. Answer: add procmail to your recipie. Ha, get it?

MailScanner
SpamAssassin
ClamAV

It's relatively simple to keep your parents surfing in safety. As many people have already mentioned, Firefox is a good start. But that's not where you need to stop. While Thunderbird is stil in alpha, it makes a nice email client, and has fewer glaring security holes than some of the more popular clients.

But where everything comes together is with the last two important pieces of software. I used to be a strong supporter of The Proxomitron, but it's very difficult to find now, and is no longer supported, so I've switched over to Privoxy which runs on most platforms, incidentally.

Privoxy is a local proxy that does filtering on all web content that you view, removing things like some ads, and all unrequested pop-ups. It filters virtually all malicious content I have seen.

A personal firewall is important to have now, and there are some reasonable free ones around. The ones I like take a bit of configuration, but they sure beat Zone Alarm. The two I use are Kerio Personal Firewall and Sygate Personal Firewall.

Sadly, both these products used to be completely free, but the same is no longer completely true.

Essentially, it is important to use a good browser, mail client, local proxy and firewall. With those in place a virus scanner is often somewhat redundant, though one of those might be a good idea too.

On the spam prevention front, I find Popfile to be an invaluable tool. It is, however, a wee bit advanced. I suspect that most parents wouldn't quite grok it. I've heard good things about SpamAssassin, though, and it might be worth the effort of teaching parents.

It's true. We've been using a combination of MailScanner, Spamassassin, and ClamAV on ours and a number of customer mailservers for a little over a year now. Don't seem to remember any viruses getting through, and many times Clam has an update before the commercial vendors. It's also got _great_ support through the mailing list(s). I would recommend ClamAV wholeheartedly.

here is a fine guide to build a Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC.

You can follow the steps and build it with Linux too. This entire procedure has been developed with security as a primary focus. These are the main tools it shows:

• Amavisd-new (www.ijs.si/software/amavisd) is the main filter which processes email from postfix and ensures that we don't lose any mail. Amavisd-new is an huge improvement over the original amavis which was a simple virus scanner, and I think it is the best way of implementing SpamAssassin (www.spamassassin.org). SpamAssassin is the main anti-spam component which works by comparing messages to a ruleset and by using a statistical analysis that is custom built based on your email. In addition to the SpamAssassin spam detection software, we will be using 2 online SPAM databases: DCC (www.rhyolite.com/anti-spam/dcc) and Vipul's Razor (razor.sourceforge.net).

Could this be a sign of the beginning of the end of spam?

Dunno... but it could be the beginning of the end of sendmail. Not that it would be a bad thing...

There's much better software out there.

In other words, this would work ok as an element of a larger system that could deal with the other 50%.

Interestingly, this is EXACLTY how SpamAssassin works, running a variety of testing engines from simple header-regexps to basian analysis to DNS-blacklist checks. SA is essentially just a framework for pulling diverse testing schemes together and using all of their input to evaluate mail. This technique seems like it would be a good addition.

I never thought that Slashdot would help me find papers relevant to my research!

I think that their idea is good from a technical point of view, but very bad from a privacy point of view. I am of the opinion that gathering social network information is extremely dangerous. A pertinent example: If your friend is branded a "terrorist," then "they" can exploit the information that you have voluntarily provided to then put you on a "terrorist" watch list.

Another example: Say that someone who knows someone that you know actually buys something from a spam. If the spammer can access the social network information, suddenly your little niche of the network is going to be aggressively spammed. After all, like minds congregate.

There is no doubt in my mind that the black hatters will infiltrate the social network communities and use that information to spy on potential viewers. See this bugzilla thread where the folks from Atriks Professional Email Deployment Service follow SpamAssassin's development and adapt their "ratware" tool accordingly.

The biggest problem with collecting social networks is that once the data has been gathered, it is very hard to control. Those of you using Orkut should think long and hard about it.

In conclusion, I think that this is technically a good idea but it opens a Pandora's box.

Possibly too late in the thread to be read, but things I would like to see to make it an IE killer:
* SVG support
* Privoxy available as a plug-in

For Thunderbird:
* spam-assasin and dspam available as plug-in options

Phillip.

Why do solutions always have to cost money or put control is some company's hands? I call bullshit. So here, people, are your solutions to spam:

User-level: spamprobe, bogofilter, spamassassin and spambayes are all very effective statistical filters with bayesian components. Train them well and you will see next to 0 spam, with just about no false positives. I dare say these will filter mail better than a human could do visually.

Those statistical filters aren't scalable. Running a large ISP is more your thing? Then install DCC at your site and enable greylisting on top of it. This will catch nearly all your spam, and false positives are rather rare.

All this software is free and actively developed. There, I've just saved you from spam. Where's my 200 USD consulting fee?

Yep. Amavis-New on Postfix with NOD32 and SpamAssassin for us.

I've said it a thousand times.

1. Mutt
2. Spamassassin
3. Greylisting
4. Profit!

If it weren't for /., I'd have never noticed.

Are you people really getting so much spam every day that the "delete" button just doesn't do it for you?

SPAMASSASSIN flags close to 1000 e-mail messages PER DAY as spam for the four e-mail addresses that I use. Unfortunately, two of those are 'well known' support addressess for some web sites I maintain. I've had to scan through those flagged messages looking for support requests from users, only to give it up as a lost cause.

Bottomline; if a user sends in a support request that, because of a false positive, gets flagged as spam, they will not get an answer.

Running the 'd' key over 1000 spam messages per day is not what I consider to be a productive use of my time.

Sakshale

That last point is particularly good, since the PHB types freak if their email isn't exactly the way that they're used to... and they also freak when implementing new technologies.

I don't know about that. Ever since I installed SpamAssassin & MIMEDefang on our incoming relays, there doesn't seem to be anything I want for stopping spam that the PHB's won't let me have. They bought me seven more IBM x335 machines just for handling mail relaying. They're ecstatic that all I want is more hardware, and not an expensive license and software maintenance contract from NAI or some outfit like that.

We just reached a milestone of having 12 million spams rejected in a month (with score >= 10.0). That's about 400 per minute, and it doesn't count emails rejected by sendmail (sender domain must resolve, access_db entries, malformed address, etc.)

Only about 1.5 million emails a month are legit messages that an employee wanted to receive. Do the math folks: 7 out of 8 emails presented to us for delivery are spam.

There's lots of great filtering technologies available out there, and the best ones are non-commercial in nature. Microsoft or Yahoo have not helped my spam situation; but spamprobe, bogofilter, spamassassin, and spambayes definitely have helped me, in very real terms: > 99% accuracy, with (generally) zero false positives depending on the quality of configuration.

Now an appeal to you folks out there who use these filters I've mentioned with similar good results (w.r.t. accuracy): we no longer see spam thanks to our filters. How about taking it one step further? Join the WPBL project and help us centrally collect IP addresses of spammers. It's an automated system to determine real-time spam sources using reliable, trusted data contributors. We are currently tracking over 15,000 IPs.

ahem