Slashdot Mirror

I've faced the exact problems you describe, have tried the oft-touted solutions, and (since you're not averse to a commercial solution) I can tell you the answer:

http://www.cloudmark.com/businesses/

I love it because (a) it eliminates a very high percentage of SPAM, (b) it has an extremely low false positive rate, and (c) it requires no fiddling - one easy installation and then forget it aside from occasional updates.

A big part of what makes Cloudmark Server Edition effective is human feedback. When a user flags a message as SPAM this sends information back to Cloudmark which helps identify SPAM for other users. Votes from users with a proven track record of accurately identifying SPAM are weighted accordingly. Due to to the large number of CSE users the system works amazingly well. It is by FAR the best solution I've tested. Bayesian filters, for example, require endless tuning and are subject to poisoning attacks. Greylisting is helpful, but it works on the assumption that spammers will never attempt delivery twice. I don't know how valid that assumption still is.

The Cloudmark website talks only about Microsoft Exchange, but there are versions of CSE that work with other types of mail servers.

I'm blocking thousands of junk messages every day on several servers with almost zero time spent on administration. Do yourself a favour and check it out.

The best solution currently in the marketplace, *BY FAR* is CloudMark. http://www.cloudmark.com/

They have a desktop and a server version and charge per user. I think we pay about $1000usd per year for 50 users. They catch everything except the occasional backscatter Non deliverable report from when your address is joejobbed.

The way it works is they generate various hashes from message content and aggregate those in their central DB.

Mail (from what I remember) is never blocked until a sufficient number users, who are weighted differently based on trust (reporting history), mark it as spam.

This doesn't cause any delay as they have zillions of users, and I believe most of the reporting comes from users of their desktop versions. I don't believe I have *ever* had a false positive, as in zero in 2 years of use.

Can't recommend them highly enough. Software used to be a little crappy and would hang sometimes (runs as a service hooking to exchange...or maybe it's mapi), but they've fixed that earlier this year.

Any questions let me know

Well, have you ever heard about Vipul's Razor Open Source antispam? And what about the commercial version CloudMark? They are working right now and really fscking good! BTW, Abaca is kindda copycatting them... (check http://www.cloudmark.com/serviceproviders/technology/ and look for the "Trust Evaluation System" paragraph).

Another example, GMails antispam... even if it's not the same logic, a great part of its antispam algorith is based in the "intelligence" of us zillions of gmail zealots that provide feedback for every email we get.

Unless this is a highly targetted and customised phishing attack. Collaborative filtering like cloudmark works amazingly well. You can stop a phishing attack spread within a few minutes. Here is more info on collaborative filtering or google for it.

You can't fight spam with heuristic filters. The only way to go is to use collaborative filtering, where millions of users participate in the filtering process and the outbreaks of spam, virus or phishing are detected and caught in a matter of minutes. Companies like Cloudmark provide this solution for free to end users, and several open source efforts are available as well.

That's what he means by "fuzzy." Coming up with a hash that will catch *mostly*-identical mails (and, one hopes, not pick up too many false positives). Razor/Cloudmark does things this way, and has a fairly goot detection rate even though spammers have beentrying to use hash-busters to defeat it for at least 4 years.

As you point out, the big three provide a massive surface area to do mass mailing tests on but have no commercial interest in sharing this information.

A company called Cloudmark has, for several years now, been running a similar system for cross-provider spam pooling. It's an outlook plugin that weights user opinions against email voted as desirable or otherwise; pretty much exactly as the article describes only on a more centralised (commercial) basis. A friend who ran it a few years back reliably informed me it was very effective.

Disclaimer: No interest in the company. Just a satisfied customer.

Cloudmark has been doing this for over a year now, I love it and have gotten many "thank yous" form people I have told about it. It uses both Bayesian filtering and uses information gathered from its network of users to help identify spam. And it integrates with MS Outlook. This other eProvisia service forces you to an address on their email server, bogus.

And don't the major web-mail providers (Yahoo mail, Hotmail, etc.) use group-based filtering as well?

http://www.cloudmark.com/products/spamnet/

is a collaborative spam filtering solution:

Protect you, your friends, and family
Every time you block a spam message that slips through, your vote benefits the entire community.

A bit expensive though : 40 dolls per year.

This stuff rocks. Zero false positives. Easy setup and interface. http://www.cloudmark.com/

It transformed my e-mail from a Kafka/Milton-esque mill of grinding misery into the basically useful enterprise it oughta be. Simple as that.

All it does is check your incoming mail against mail reported as spam by 1.08x10^6 other users. A simple idea, well executed. No, I don't work for them or own stock. I'm just a regular schlub who was getting 1000 spams a day for a while, did some looking around, and came up with a total winner.

At first glance, this solution might look like one which can be trusted as a reliable method to avoid having unsolicitated mail in one's inbox, but it takes just a little curiosity to be somewhat put off the idea of paying the stated yearly fee for the service.

On the eProvisia LLC - Spam Eradicator presentation page, it is written that for 'a low yearly fee of $19.95', one can benefit of the enjoyment of a 'guaranteed 100% protection against spam [...] with the unprecedented reliability and simplicity of [their] all-in-one solution!' It first sounds like a rather interesting deal thus said, and one is incited to read on. After all, how do they obtain an unprecedented reliability against protection? It is written further on, that 'a dedicated team of over a hundred trained Screening and Preselection Specialists, working 24 hours a day, will begin manually reviewing, hand-picking and approving important correspondence, vigilantly discarding all junk mail.' Perhaps this company has 'thousands of satisfied customers' as it is said on the same page, but I would certainly not be volunteering to pay for some people I do not know reading my personal e-mail and sorting it according to what they think is or is not what I would like to recieve in my inbox.

After all, how could I know what are the ethics of my 'Specialist', what he or she considers to be correct and what he or she considers not to be, whether or not the 'Specialist' in question may just employ some subjectivity when sorting my e-mail? What's more, from what I understand basing myself on what is indicated on the given site, the client is not actually able to recieve the e-mail marked as unsolicitated by the 'Specialist', and hence cannot verify whether or not there are some false positives with a quick glance, as one usually can. I do recognize this is intentional so the client is free of the mail marked as unsolicitated, but it would be appreciated if there was the option of recieving the junk mail with a specific tag (such as 'X-Unsolicitated') which the mail client would filter so the user of the service could have the -mail marked as unsolicitated by the 'Specialist' sent in a folder, just so results can be checked.

If I want human-powered spam filtering, I would rather try Cloudmark SpamNet as a solution. It is a product based on the active participation of the community, and it works by marking as spam mail marked as such by users of the service. Users with a greater reliability in the past use of the product have a greater influence in the marking as spam of e-mail, and the general reliability of the program is said to be of 98% if the company producing it is to be believed. (It costs $39.95 per annum for the subscription to the community filtering, though a 30-day trial version is provided.)

I have tried just about every single anti-spam software out there, so I have some experience. After being fed up with getting false positives and having to deal with tons of spam getting past the spam filters I tried out Cloudmark's Spamnet - a community based approach to fighting spam. So far it has been 95-99% effective with 0 false positives which is the most important factor for me.

In the past couple of months it has blocked 19,221 spam messages. I don't even bother to send spam to a Spam folder anymore it just goes straight to the deleted items.

For those of you getting a lot of e-mail, the price of the subscription is definitely worth it.

URL: http://cloudmark.com/products/spamnet/

I am always confused by the omission from these tests of collaborative filters like Cloudmark's SpamNet, which I have used at work for a long time with a very high "catch" rate, no real processing time, and no false positives. Essentially, every email you get it hashes and checks with the server. If you get a spam, you right-click and report it as such. Then it pulls any messages from your inbox which enough credible people have marked before you. (A gross oversimplification, but close enough.)

I feel like at our current stage of technological development, you have to combat human-generated deception with human intervention.

(By the way, that cloudmark tool is Outlook-only, but contains some concepts I'd like to see in other filters...

how many aggregate "irreplaceable years of life" this scumbag cost others

According to my spiffy SpamNet statistics window; the collective community (~970K) would have lost a combined 8200 days (since the program began) dealing with spam ourselves.

I believe (and hope!) that this works in a similar way to a product like Cloudmark Spamnet,

This msg brought to you by SpamMark CloudNet because we can't afford any real advertising.

I have a gmail account, and they have a nice feature that actually lets you report a message as spam. I believe (and hope!) that this works in a similar way to a product like Cloudmark Spamnet, which basically lets the community of users determine what is spam. This then updates the community spam list, and affects the e-mail of all clients. As this gets more effective with more users, I hope gmail is very popular.

Not really. They'd probably use a reputation-based system like Razors' Truth Evaluation System. All that would happen is this:

1. Spammers sign up a new account and subscribe to mailing lists.
2. Spammers mark these ligitimate emails as "spam" in an attempt to poison the spam detection system.
3. Regular users notice this and mark it as non-spam.
4. The spammers' already-low (since they're new) trust level goes down so far that their future markings are ignored.

Spammers could abuse this by automatically signing up lots of new accounts. Perhaps new accounts could start out with zero trust, or maybe even a negative number. Then people would have to earn trust (by correctly marking any spam that gets through) before their marks are even used by the system. It would be tricky to manage and find a balance. Spammers are a determined bunch, but so are users. There's always a certain subset of users who are willing to be a little more vigilant in order to keep the spammers in line and help everyone else. Gmail is bound to have a huge number of users, so it should work out fairly well. Vigilant users + bayesian filter (with huge database) = almost perfect spam filter.

Instead of paying a penny, the sender would "buy" postage by devoting maybe 10 seconds of computing time to solving a math puzzle. The exercise would merely serve as proof of the sender's good faith.

Even if it takes 10 seconds to perform the calculation on todays hardware, within a few years we are talking about 0.1 seconds or less. So why go with a temporary solution, let's nip the problem in the bud!

Personally, I use the Cloudmark plugin for that nasty M$ program which effects a /.-like Karma system. Until we have a system with staying power, I don't see an end to this problem.

It's allready been done. See Cloudmark Spamnet.

Now...a free version would be even better.

A simple way to solve this problem is with p2p blacklists

It's semi-distributed, in that users install a small plugin for Outlook, adding "block" and "unblock" buttons to the tool bar. The entire community of users works against spammers.

It works well. When I check my mail, I can watch the 50 or so spams I get daily pop into my inbox, and then promptly fly right back out again.

(Blatantly stolen from Spamnet's Learn More page)
When the message comes in, SpamNet generates a unique fingerprint of that message. The fingerprint is a one-way hash, or unique string of numbers that represents the email and can absolutely NOT be decoded.

This unique fingerprint of the message is sent to the server where it asks the database if this message is spam. The server comes back to the client with a confidence level of how sure it is that the message is spam by checking it with the other fingerprints in the database.

If the same signature has been reported to the SpamNet database, this indicates that the message is spam and it is consequently moved from the member's Inbox to the Spam folder.

If a spam slips through, the SpamFighter can use the "Block" button (vs. delete) to remove it from their inbox and report it to SpamNet to help themselves and the community. Again a unique fingerprint of the body of the message is generated and sent back to the server. Here is where TeS, or the trust system comes into play to ensure that only valid spam messages are blocked. SpamNet looks at the reputation of the person that blocked the message and depending on their individual trust rating; a confidence level is applied to that message to decide whether it should be blocked for the entire community. Each person starts with a zero trust rating and generates trust based on several factors including how accurate their reports are and the number of reports overtime. This process happens instantly taking less than 3 minutes to stop a spam message that's new to the system for the entire community.

Cloudmark does this. I don't use it directly, but my installation of SpamAssassin checks the Cloudmark/Razor servers for the message signature.

Since my email is hosted on a Linux server, I use procmail (with SpamAssassin) to filter spam into a Spam folder.

In this type of case, one overpowering rule makes the program more harmful than helpful. Being too sensitive greatly decreases specificity as well... thus the system spirals into uselessness.

Anyway, Cloudmark is a commerical product based on vipul's razor. Sadly, vipul's razor will not compile in the window's environment yet.

God, I hate spam

Davak

Its called CloudMark, and here is a link: Download SpamNet

This program is great, you can mark new spam, and its added to their spam database to be filtered for everyone else. It auto-updates when you load outlook, and is very un-obtrusive. It simply moves spam to a 'Spam' folder in outlook, which you can clear whenever you want. Very reccomended, I like it! :)

Cloudmark's Spamnet is an extremely good application for this.

http://www.cloudmark.com/products/spamnet/

If the user is known (from having been manually entered or already recieved 1 registed email in the past, and not in the blocked senders list, the mail will again go into the users's inbox. All other mail is automatically placed in a folder of the user's choice. If that means the trash, fine.

There's a mob called Cloudmark who do basically the opposite with a product called SpamNet - known spams are sent to the trash, unknown spams are "fingerprinted" by a P2P kind of affair. No need to worry about registering, is usually quite effective and mostly transparent. The downside is that it's MS Outlook only and the free beta program has finished.

Works for me though, and beats a new tax...

How is this different from the open-source Vipul's Razor, Pyzor or DCC, all of which are already in wide use through their easy integration with SpamAssassin?

Clearly a proprietary system just won't be as good because it needs, by its very nature, a lot of subscribers to be effective. Having said this, Cloudmark seems to do alright by using Razor's network.

Just think if slashdot was filled with spam messages.

You're new here, right?

Try reading at -1 and you'll see a lot of spam. Not necessarily selling you stuff, but a ton of noise. Moderation improves the signal-to-noise ratio.

Perhaps there's some sort of moderation we could do to our incoming email? (Hint: try SpamNet if you're running Outlook.)

and if you are a windows user, if you couple SpamAssassin with SpamNet then you have the best protection around!

If you're interested, I now use a spam filter which works pretty good. It's a peer-to-peer filter, so the definitions you create for spam are shared across all users, and vice versa. Since I started using it, I've seen very little spam in my inbox. Unfortunately, I think it only plugs into outlook, but I'm not sure. Maybe somebody here can reverse engineer this baby and pump a new cross-client/platform net...

Here it is

Even at 'our' level of knowledge, we all give out our email addresses every day. And we're not neophytes by any stretch.

Every day thousands of people sign into various sites, drop their email addresses here and there, never thinking of the consequence of where thats going to go, and not seeing the connection to the increased levels of spam. I have one spam account that I use for any site I think is going to sell/lease/rent/whatever my email and I watch it to see when increases begin. I don't ever give out a regular account, because I KNOW I'm going to get spam.

If we could educate the 'regular' masses of internet users that send emails to their family and friends, and surf for news, we'd be ahead already. If we could show them that by giving away your email address you ARE going to get spam, they might stop. The example that works for me is 'do you stop and give out your address to every single store you walk into? to the guys trying to 'give away' free newspapers?' If people learn to control their email address as they do the rest of their personal/private information, there will be less targets for spam.

My 'theory' works in practice. I get about 5 spams a day on my main account, which I use for various mailing lists, websites etc. I selectively give out my 'good' account, and what crap I do get Cloudmark gets rid of for me.

So if we could educate our friends/family not to just give up their email address to every site that wants it, every program they install, every popup that comes up, they'd get a lot less immediately.

ive been using CloudMark for Outlook. I actually read about it in a /. post. Windows users, check it out. It's free

Try to use SpamNet from cloudmark!
This one bases on a kind of P2P system which allows users to block Spam while this is reported to the main servers.
So if someone has blocked the message before, every SpamNet user doesn't have to do it again, because Spam is moved to a different folder automatically (they using checksums and stuff i think)
A problem still might be then, that this software is for Outlook only.
But nevertheless a good (though not perfect) system. I'm pretty satisfied with it.

Hope that helped...

cyphem

I've been using Cloudmark's SpamNet for the past few months and it's been working quite well.

The smart thing that SpamNet does, is that it relies on its users to determine if something is spam or not. If some email lands in your inbox and a few hundred SpamNet members have proclaimed it spam, it most likely is, and it gets immediatly filtered out. This has the net effect of a few user's needing to filter out a few message ocassionally, while the vast majority of messages are filtered out for all users. Although SpamAssassin seems quite good, it's still based upon filtering rules and spammers are constantly tweaking their emails to try to get around them. Since people are still better at determining what's spam and what's not, I find that its accuracy is generally better.

SpamNet isn't perfect though, as far as I know, it only works with Outlook on Windows and doesn't have a Unix, Linux or Mac version. It also sometimes filters out valid bulk mailings, but overall, I would definitely recommend it.

When a spam message is submitted to Cloudmark SpamNet, the system generates a secure fingerprint or signature of each message. This unique, but indistinguishable, fingerprint can now be securely shared with all the other SpamNet users to identify the same spam message in their email. This system permits everyone to contribute to the fight against spam and ensures that all email remains private.

To ensure that you never lose email, no messages are ever deleted or blocked. If a message is identified as a known spam message, it is simply tagged as spam and moved to your Spam box. This process allows you to verify that all the messages in the Spam box are really spam.

The SpamNet system has been running smoothly for over a year. During that time, thousands of users have endorsed the system's effectiveness by processing millions of email a day through SpamNet. The proven system will immediately cut your spam dramatically, but you can also choose to turn the spam checking off at anytime. The Cloudmark SpamNet Outlook add-in won't interfere with your email if you choose to discontinue using the service.

So what do you get for your spam contributions to SpamNet? Well, you get free spam filtering. If that sounds good, download the Outlook plugin.

When a spam message is submitted to Cloudmark SpamNet, the system generates a secure fingerprint or signature of each message. This unique, but indistinguishable, fingerprint can now be securely shared with all the other SpamNet users to identify the same spam message in their email. This system permits everyone to contribute to the fight against spam and ensures that all email remains private.

To ensure that you never lose email, no messages are ever deleted or blocked. If a message is identified as a known spam message, it is simply tagged as spam and moved to your Spam box. This process allows you to verify that all the messages in the Spam box are really spam.

The SpamNet system has been running smoothly for over a year. During that time, thousands of users have endorsed the system's effectiveness by processing millions of email a day through SpamNet. The proven system will immediately cut your spam dramatically, but you can also choose to turn the spam checking off at anytime. The Cloudmark SpamNet Outlook add-in won't interfere with your email if you choose to discontinue using the service.

So what do you get for your spam contributions to SpamNet? Well, you get free spam filtering. If that sounds good, download the Outlook plugin.

Vipul's Razor, Cloudmark, etc. are collaborative filters that let humans mark messages as spam and share the spam ratings, so even Spam-Of-The-Future messages that evade filterbots are likely to get caught by humans. That means that if a roughly-identical message gets sent to N people, and sneaks by their spam filtersbots, the first few humans to read it send in ratings that let everybody else's filterbot kill it for them. They do some kind of hashing function to catch similar-but-not-identical messages, which is necessary because message headers will obviously be different for every recipient, but have useful information, and message bodies for different recipients may be identical, but often have some recipient-customization, like "Dear Bob" and "remove-2184242314231-Bob@spammer.com".

If you are using windows, and outlook, you can install SpamNet, made by Cloudmark.

I had to stop using Eudora, because I had so many filters (400+) to kill my spam that it took, literally, 5 minutes for my mail to appear in my inbox, which, needless to say was very frustrating and annoying.

Anyhow, I have been using Spamnet for about 7-8 months and, depending upon the time of day that I check my email it correctly blocked between 60% - 95% of my spam.

For example, since it is a peer based spam detection system, so the more users that vote that email from a particular sender is Spam, the more likely you will get it blocked. Eventually, it maps out and makes blacklists based on overall stats.

The point is, I took 2 days off for Xmas and when I checked my mail on the 27th, it filtered out about 295 of about 300 spam messages.

If you are using windows, and outlook, you can install SpamNet, made by Cloudmark.

I had to stop using Eudora, because I had so many filters (400+) to kill my spam that it took, literally, 5 minutes for my mail to appear in my inbox, which, needless to say was very frustrating and annoying.

Anyhow, I have been using Spamnet for about 7-8 months and, depending upon the time of day that I check my email it correctly blocked between 60% - 95% of my spam.

For example, since it is a peer based spam detection system, so the more users that vote that email from a particular sender is Spam, the more likely you will get it blocked. Eventually, it maps out and makes blacklists based on overall stats.

The point is, I took 2 days off for Xmas and when I checked my mail on the 27th, it filtered out about 295 of about 300 spam messages.

www.cloudmark.com

It uses a moderation system not dissimilar to Slashdot (but maybe without the weird 2+2=5 maths) and in my experience DOES work. YMMV. I've yet to have it filter a legitimate message, and it picks up about 70% of spam into my Inbox...

(Using Windows...) Install Outlook 2k or higher to download your Hotmail messages to your local machine. Install CloudMark and say goodbye to 99% of your spam. This will also remove any space limits imposed by Hotmail, as your mail will now be sitting on your own hard drive.

It seems that at some point ISPs/mailbox providers are going to wake up and see that they should filter out the junk mail for their users. But, since we're all still waiting for that to happen, I decided to try a little program called SpamNet that promised to block out junk email. After a few months of use I'm happy to say it works great. The premise of SpamNet is that all users of the software can block spam. This works well, and works even better as more and more people use the software.

SpamNet sends it to a little folder called "Spam", in case you want to double-check and make sure nothing you wanted got blocked.

The good parts:
- Automatically blocks about 95% of spam
- Small, fast, simple, FREE

The bad:
- Not at all configurable (just does what it's supposed to do...)
- Occasionally it will block something from Amazon.com or another large mailing list which isn't really spam.
If you're tired getting spam give it a try for yourself, here is the link:

SpamNet

System Requirements:
Outlook 2000/2002/XP
Windows 98/2000/XP

Maybe something like Cloudmark is the solution?

Try SpamNet it does something like that, only works for Outlook 2000/XP at the moment but they say outlook express support will come soon. It generates some sort of hash from the email and compares it to its database of known spam, you can also block spam that it did not filter so it filters it next time.

Cloudmark makes SpamNet, a P2P plugin for *gasp* Outlook, that allows users to submit spam messages to a database, where an algorithm integrates the submissions into a master spam list that gets published back to the clients, which then pull messages out of users' inboxes as they arrive. (Works pretty well, too.) I should think their DB would be a good place for this effort to begin.

Cloudmark makes SpamNet, a P2P plugin for *gasp* Outlook, that allows users to submit spam messages to a database, where an algorithm integrates the submissions into a master spam list that gets published back to the clients, which then pull messages out of users' inboxes as they arrive. (Works pretty well, too.) I should think their DB would be a good place for this effort to begin.

Doh! That was the pretty much the method I was secretly keeping to myself. =) They must already apply a complex algorithm, because I've had stuff blocked with my name stuck in it. But, if someone generated completely unique spam-mails for each person, it would be difficult to catch, though for spam to be effective as an ad, there would have to be some commonality between the messages. A large amount of random bytes at the end of each email would likely bypass the system, until you filtered for large amounts of random bytes. But if you tagged on, say, large chunks of Shakespeare at random...

Now if you get Hamlet with your next spam, don't blame me =)

Promising newcomers such as CloudMark, which taps the collective power of e-mail recipients to identify spam, may improve things for a while.
I've been using this for a while, and am catching like 80% with 0 false-positives so far. The only downside has been a few minor bugs, which is expected for a beta product and have more to do with Outlook than anything. I think the concept is sound, and would be pretty hard to circumvent. Basically, a fingerprint (one-way hash?) of the email (not just the header) is looked up in a database which contains reported spam. Reports are weighted for reliability, which prevents spammers from unblocking their own spam. I can think of only one way, besides a DoS, to get around it, but I ain't telling here =) www.cloudmark.com