Slashdot Mirror

The technical countermeasures used by TMDA to thwart spam include:

• whitelists: accept mail from known, trusted senders.
• blacklists: refuse mail from undesired senders.
• challenge/response: allows unknown senders which aren't on the whitelist or blacklist the chance to confirm that their message is legitimate (non-spam).
• tagged addresses: special-purpose e-mail addresses such as time-dependent addresses, or addresses which only accept certain kinds of communication. These increase the transparency of TMDA for unknown senders by allowing them to safely circumvent the challenge/response system.

This combination was chosen based on the following assumptions about the current state of spam on the Internet:

1. You cannot keep your email address secret from spammers.

2. Content-based filters can't distinguish spam from legitimate mail with sufficient accuracy.

3. To maintain economies of scale, bulk-mailing is generally:
* An impersonal process where the recipient is not distinguished.
* A one-way communication channel (from spammer to victim).

4. spam will not cease until it becomes prohibitively expensive for spammers to operate.

Probably the most reliable way to defeat the spammer business model is to use a whitelisting mail filter technique like TMDA. Spammers rely on 1. cheap and easy bulk email delivery (for them, at least) and 2. access to your mailbox by default. That doesn't work if mail is not delivered by default with a whitelisting system -- in that case, their mail waits in limbo for a confirmation response that will never come.

Well, if you use TMDA, you can configure it to avoid what you're talking about. With TMDA, it can detect whether or not an email was sent in response to an actual email that you sent. If so configured, then any challenges that you get from someone will only be delivered to your mailbox if you actually sent the original email. If a spammer, right now, sends an unsolicited challenge to my mailbox, I'll never see it.

So, exactly the contrary to what you're saying. The wider spread the use of C/R like TMDA, the less effective that your suggestion will be.

If a spam has a spoofed from address, then your C-R system will send a challenge to the spoofed from address

If I reply to an email you sent using a different email account than the one that you sent your email to, then my other email account won't be on your whitelist and I'll receive a challenge. By itself this is merely annoying

However if we both do it then our challenges never get through.

You might think that problem 1 can be solved simply by challenging only non-spam emails, but then you have the problem of spam filtering all over again. Most people who use TMDA do so specifically because they think filtering is ineffective.

But I eventually stopped using RBLs because they summarily cut off too many people without providing each individual a mechanism for getting legitimate email to me. And I eventually stopped using SpamAssassin because it was too effective. It marked a couple of emails that were NOT spam as spam and I ended up losing some contacts. It was a very low percentage, but enough to annoy me. So I switched entirely to TMDA. One could argue that a legitimate email that doesn't get confirmed is the functional equivalent of a "false positive", in which case I've had a few of those. But if you don't care enough to confirm your email, it must not have been something very important to talk about. Which is a very different situation than you sending me an email that, from your perspective, I completely ignore because my spam tool thought it was spam.

Of course, by all of this, I don't mean to suggest that C/R is the solution that everyone should use. I know it probably seems that way because I'm defending it. But I don't mean it that way. I really only want to understand why someone might not use C/R. I'm not trying to advocate that everyone choose C/R. If it doesn't work for you, I was just curious as to why.

If a spam has a spoofed from address, then your C-R system will send a challenge to the spoofed from address

If I reply to an email you sent using a different email account than the one that you sent your email to, then my other email account won't be on your whitelist and I'll receive a challenge. By itself this is merely annoying

However if we both do it then our challenges never get through.

You might think that problem 1 can be solved simply by challenging only non-spam emails, but then you have the problem of spam filtering all over again. Most people who use TMDA do so specifically because they think filtering is ineffective.

But I eventually stopped using RBLs because they summarily cut off too many people without providing each individual a mechanism for getting legitimate email to me. And I eventually stopped using SpamAssassin because it was too effective. It marked a couple of emails that were NOT spam as spam and I ended up losing some contacts. It was a very low percentage, but enough to annoy me. So I switched entirely to TMDA. One could argue that a legitimate email that doesn't get confirmed is the functional equivalent of a "false positive", in which case I've had a few of those. But if you don't care enough to confirm your email, it must not have been something very important to talk about. Which is a very different situation than you sending me an email that, from your perspective, I completely ignore because my spam tool thought it was spam.

Of course, by all of this, I don't mean to suggest that C/R is the solution that everyone should use. I know it probably seems that way because I'm defending it. But I don't mean it that way. I really only want to understand why someone might not use C/R. I'm not trying to advocate that everyone choose C/R. If it doesn't work for you, I was just curious as to why.

They where a great free email service ('whitelist') similar to the TMDA system.

I see quite a few posts suggesting that spammers are getting desperate, but brazen seems more appropriate. They are shutting down some of our most effective anti-spam tools and there seems nothing we can do about it. To me that looks more like their winning.

Everyone in the world must jump through the painful, non-functioning hoops of whitelisting...

Just out of curiosity, what about whitelisting do you think is non-functional? I've been using a program that, among other things, is an automated whitelist management program. It's called TMDA and it works fantastically. There are other similar programs.

I'm just curious as to why you think whitelisting is non-functional.

Bluebottle has found itself under constant attack from numerous sources over the past couple of months making it almost impossible to deliver spam free email to your account in a consistent and timely manner. We have therefore decided to cease offering protection for external accounts, and will be removing the verification protection from Bluebottle accounts.

This has not been an easy decision to make but has been necessary in light of the delays currently being experienced in email delivery. Whilst work is still being performed to address these issues, as it currently stands, Bluebottle is unable to ensure the timely delivery of mail for Bluebottle accounts. You are certainly welcome to continue using your Bluebottle account, although no verification protection will be applied to inbound mail.

We have done everything in our power to address these attacks although it has had little effect. We are obviously very disappointed that we cannot continue to provide you our service at this time.

Bluebottle's email verification system is best provided in a distributed manner making it considerably more difficult for these attacks to be effective. We will therefore be making our software freely available to any service provider or enterprise to protect their end users from unwanted email, and by doing so make it a more secure solution given that it is provided in distributed environment.

Please accept our sincere apologies for the inconvenience our decision will cause.

It leads to a domain squatter if you can't spell. If you can spell correctly it works a lot better.

Nice misspelling.

TMDA

Comparing it to (sorry, should have included a TMDA link for those not familiar with it) filtering and RBL's is not fair because unlike the latter two, it does *exactly* what its supposed to. I'll admit its a hack, but for the time being it is the best hack out there.

The reason I suspect you haven't used it is because you mention one of the same concerns I've had about it, mainly automated responses. Bluebottle's answer to this is in the form of a 'pending' list (which you can 'OK' emails from) and the ability to manually add specific email addresses or even whole domains.

Its really a pretty good system. I think almost everyone is clear now that RBL's are a potential nightmare and filtering only creates a new list of email to cull through (looking for mislabled email).

Personally, I currently use TMDA to protect some of my accounts. It's a challenge/response system that uses whitelist/blacklist technology and sends a challenge to the unknown senders. This is quite effective at combatting spam since the challenge to a spam message usually ends up bouncing anyways.

The only problem with TMDA is that some people consider the challenge/response method to be quite rude (click here and do a search for "gunfighter" to read the responses to my comment(s) about TMDA). TMDA, and similar technologies, definitely place the responsibility for ensuring message delivery in the hands of the sender and receiver. In addition, there are other considerations such as the additional overhead of the extra messages. There are even cases where people who haven't properly configured such technologies end up getting into confirmation loops and screw up by sending a challenge to a legitimate mailing list.

To overcome these problems I've actually concocted, in cooperation with a fellow developer, an automated means of verifying the authenticity of an email message. While this may not stop spam cold in its tracks from the get-go, it will definitely be a step in the right direction. Instead of blocking entire IP blocks (or even individual IP addresses), companies, ISPs, and individuals will soon be able to compare against blacklists of individual users.

By using this technology in cooperation with a challenge/response-type filter, only individual senders flagged as potential spammers will be a) blocked or b) flagged as possible spam or c) receive a challenge/response. This will completely obsolete any and all current methods of dns or IP based blacklist(s).

Time-to-market is up in the air right now, but hopefully we'll have a prototype ready here in the next month or two. Hope to see you then.

Along with what jovlinger said, you can create tagged addresses to give out to anyone you choose. You don't post these ones publicly because they require no "challenge-response".

A full working scheme for whitelisting and tagged addressing can be found here.

The only thing that can get through this kind of filtering is an extremely smart spammer who (as srw described above) can find the address of somebody on your whitelist to put in it's "from" field, or email worms which will most likely come from people on your whitelist.

Email worms are another problem altogether, but spammers smart enough to spoof your friends are something no filter can properly deal with right now. In the meantime, Dacin Santa is right. Whitelisting is more work (if you use the full scheme), but it's the best way.

Don't forget the excellent combination of Qmail with TMDA for flexible challenege-response spam blocking.

An interesting thread here about how TMDA, a C/R filter, used in conjunction with SpamAssassin, can provide the best of both worlds. While TMDA is by itself effective, there seem to be some humanistic issues involving the assumption that all e-mailers are spammers unless they prove otherwise. The thread explains how Bayesian filtering can be improved by using a decent C/R filter like TMDA without alienating people that send legitimate e-mail.

Personally, I figure anyone thin-skinned enough to be insulted by my C/R filter probably isn't worth talking to anyways, but I digress...

I finally managed to install TMDA. The installation is not for the faint of heart, but the end result: Usable email again -- after all these years.

After I blacklisted mail9.com (they confirmed every spam assigned to them), I have received zero spam. TMDA filters out a hundred messages per day.

say I purchased something on-line from a vendor I had never dealt with before. Their e-mail system may automatically kick out an e-mail

Using TMDA, you would generate a "keyword" address: A unique addressed, identified by a keyword embedded in the address, which would allow your vendor to bypass the C/R system. If they keyword address starts being abused then (1) you can easily disable it, and (2) you know not to do business with that vendor again.

As an example, another post here mentioned a system where the mail is held, not on your ISP or upstream provider's system until you download it, but rather is held on the sender's or sender's ISP's system.

This system quickly breaks down, though, as delays are introduced by having to wait to fetch each piece of mail. People bothered by such delays will write/obtain software that automatically fetches the mail at a predetermined time, which would then shift the bandwidth problem (part of it, anyways) back to the recipient.

The other problem with sender authentication is who, exactly, determines whether a sender is authenticated? I run my own e-mail server. Will I have to pay out bucks for an "authority" to confirm that my sending address is valid? Right now, some ISP's (notably Time-Warner offshoots) are denying access to their SMTP servers under the guise of reducing spam. If your IP happens to fall within a certain range, they simply don't allow you access. We will end up in the same morass RBL has put us in: Who plays God in determining whether a sender is truly "authentic" or "worthy"?

1. If the email is from someone in my whitelist, allow the mail to go through and feed it as 'ham' to the Bayesian filter.
2. If the email is not in my whitelist, run it through spam filtering software (Spamassassin works well) to determine if it is likely to be spam.
3. If it seems like spam, then use a challenge-response system (like TMDA) to find out if a human sent the email.
4. If the mail doesn't seem like spam, just deliver it. If I get 3 non-spammy messages from the same person (separated by a day or more) then add them to my whitelist automatically.
5. If someone responds to the TMDA challenge, put them in the whitelist and deliver the original email.
6. If no one responds to the TMDA challenge after a week, feed the mail as 'spam' the the Bayesian filter.

• Business mail I want (like receipts and newsletters from companies I do business with) get through always since the Sneakemail-type address is whitelisted. This solves the problem of businesses not responding to TMDA challenges.
• My real email address is protected from businesses who are likely to sell it and from people farming addresses from mailing lists.
• Personal email that the spam filter sees as non-spam gets delivered without bothering the sender with a challenge-response system.
• Personal email that does seem spammy by the filter still has a second chance to make it through the system with the challenge-response system. This should reduce false-positives to include only spammy emails from people who don't respond the the challenge.
• The Bayesian filter is automatically trained based on mails from people in my whitelist and mails from people who never respond to the challenge-response.

I don't disagree. I think that eventually we should move to a better email model - something like TMDA perhaps, where there is no guarantee that spammers can reach mailboxes. Or better legislation to make spamming punishable, controls on mail routers on million message mailouts, etc. Or djb's Internet Mail 2000, which moves the onus onto the senders network to store all 1m messages at a time, until people pick them up.

The other thing you can do is impose a microcost for mailing - at 1c/mail, spamming isn't economical any more. But then that is going to penalise the people who have legitimate reasons to send a million emails at a time - you'd have to have a very good micropayment system working on the Internet to do this.

However, those things need widespread change, and they need people in positions of power. Joe User at home can push for it, but they still get spam and they still want a short term solution. I suggest that even if they're filtering, the action of having to check their spam filter will make them irate enough. I see it as being like IPV6 - everyone would really have to change at once for the system to be most effective. (I use Freenet6, do you?)

Now that viruses are public, caught quickly, and Microsoft are being a lot less lax with security (I am in no way commending their effort, but they at least mostly fixed the Outlooks), you don't see people writing them nearly as often. I feel spam will get the same.

Working out the details of an appropriate certificate policy is not trivial, though.

The solution to this is tagged addresses. This is what TMDA uses (dunno anything about port995.com).

The basic deal is that you tell amazon.com that your email address is someuser-amazon-cryptochecksum@foo.net instead of someuser@foo.net. Any mail sent to that address gets right through to your mailbox. If Amazon ever starts spamming you, you revoke the address. TMDA has some front-end tools to make generating the addresses (handling the crypto) pretty painless.

My favorite solution is still TMDA, a free challenge-response auto-whitelist and complex filtering system for Linux. I realize you anti-challenege / response people won't hit the "R" key for me, but I consider that a useful filter...

I'm sorry that you aren't willing to push the "R" button to do you part in the war on spam...

But on the personal responsibility side, if one expects to receive private messages in response to a posting of some kind (be it mailing list or whatever), using TMDA you can set up addresses without filters to subscribe to the list, such as eli173-1-k-responses@biteme.org. That email address can be set up not to have an auto-challenger on it, but still deliver email to eli173@biteme.org.

Should a spammer harvest that tagged address, you can close it down and start up another one.

Moreover, TMDA filters can also use other filtering techniques (ala Procmail), such as looking in headers for a Mailing list name, and it can avoid auto-challenging emails with those headers.

You can check out all the filters here, and there are some common uses here.

To date, I am unaware of missing any non-spam email because of TMDA. Keep in mind that messages can be kept in a "pending" directory until their challenge is replied to. I (quickly) scan that directory once a week or so, in case I missed something.

But it turns out that most people worth emailing with are willing to press a single key for you...

I'm sorry that you aren't willing to push the "R" button to do you part in the war on spam...

But on the personal responsibility side, if one expects to receive private messages in response to a posting of some kind (be it mailing list or whatever), using TMDA you can set up addresses without filters to subscribe to the list, such as eli173-1-k-responses@biteme.org. That email address can be set up not to have an auto-challenger on it, but still deliver email to eli173@biteme.org.

Should a spammer harvest that tagged address, you can close it down and start up another one.

Moreover, TMDA filters can also use other filtering techniques (ala Procmail), such as looking in headers for a Mailing list name, and it can avoid auto-challenging emails with those headers.

You can check out all the filters here, and there are some common uses here.

To date, I am unaware of missing any non-spam email because of TMDA. Keep in mind that messages can be kept in a "pending" directory until their challenge is replied to. I (quickly) scan that directory once a week or so, in case I missed something.

But it turns out that most people worth emailing with are willing to press a single key for you...

After a while, SpamAssasin's false negatives and positives drove me to the Tagged Message Delivery Agent (TMDA).

TMDA has flexible whitelist and blacklist capabilities. But the big win is that it can be set to autoreply to anyone not on the whitelist, and require them to reply back before allowing the email to get through. Of course, very few spammers have valid return email addresses...

This may seem drastic, but in fact it has made life soooo much easier. It also helps you to "automagically" get off those email lists you signed up for a long time ago, don't really care about, and are too lazy (or lost the info) to sign yourself off ;)

The only sad thing is that no longer do Russian women want to extend my length or give me free money or viagra, and I am no longer in contact with Ms. Sesse Seiko from Uganda...

Man, I love TMDA. I'll just generate an expiring address. The one that I'm giving them will only work for 7 days.

A slightly different idea that I was considering today works as follows.

Take the Tagged Message Delivery Agent, a system that will send a challenge message to anyone it doesn't know (isn't in the whitelist), which you have to reply to.

Then change it so anything allowed through on the whitelist is added to the "Not Spam" category, and anything that is challenged is passed through the filter. If it passes, it doesn't get challenged (but also doesn't get added automatically to Not Spam), and if it _doesn't_ pass, then it gets challenged.

Few, if any, false positives, and challenges not sent where they don't need to be. Sounds foolproof enough...

In the few days I have been using TMDA, I have been exceedingly satisfied. It is a much better solution than SpamAssasin. You should try to whitelist most of the people you expect to receive email from ahead of time, but I haven't had any complaints from people having to respond to a message bounced back to them for authentication.

That, in combination with qmail's revokable dash-addresses (howard-amazon@cow.com, howard-slashdot@cow.com, etc.) make it an excellent solution not just for avoiding spam, but for tracking its sources as well.

I'm sure others out there have prior art to present in this - particularly TMDA.

My own anti-spam system, which will be launched very soon (sorry no link, my dev server couldn't survive a slashdotting), also uses challenge-response. The predecessor of this also used challenge-response, and has been in use for around three years, "publicly" at least in the sense of the many thousands of people and spammers who interacted with it.

Any suggestions how I ought to present this to the patent and/or lawsuit people? Sure it would be helping a competitor, but I feel it's the right thing to do.

TMS which became TMDA.

Besides, TMDA works, while Mailblocks doesn't. I grabbed a Mailblocks account while I could get a good username, and found that Mailblocks doesn't send out the challenge: it just discards my test messages as spam after 14 (?) days.

A few of my favorite examples are:

• MyNetWatchman, firewall incident reporting service. Helps to defray spam by finding and reporting compromised hosts internet-wide.
• SpamCon Legal Fund, to help them further the cause.
• TMDA, The GPL spamfilter that actually delivers on the zero spam, zero fasle positive promise.
• SpamHaus, who does a great job keeping lists of both servers and spammers, and is very dedicated
• Your Local Food Bank. courtesy of abuse.net who says: "If you feel that abuse.net has been useful to you, please make a contribution to your local food bank, which needs money a lot more than we ever will. Thanks."
• Distributed Intrusion Detection System, another firewall aggregator, maybe the biggest, free to all

A number of times somebody has posted to a mailing list asking for help. I've answered them privately, only to get a "please jump through the following hoops" message.

Assuming the poster asking for help has a degree of clue, TMDA copes with that. Clearly, in your experience, the poster did not.

You can configure the return address of the posted message to accept unhindered replies from _all_ senders to that particular address for a limitied period of time. Therefore, the window of opportunity for spammers to use that address is small.

This is discussed in 'Dated Addresses' under TMDA client configuration.

You haven't done your homework. See the FAQ

See the FAQ

Most ISPs are lazy and incompetant and only interested in collecting your money. The rest are in bed with the spammers.

Actually, there are a few of us that offer TMDA to our customers.

I also don't buy the argument that "Most ISPs are lazy an incompetant." Spam is a very real problem, an most of the big ISPs are already beginning to take action, both technically and legally.

Of course! Whitelisting (or SMTP2000) is the future, baby.

• dated addresses
• sender specific addresses (which only let a unique from addresses through)
• keyword addresses

I've got to concur. TMDA is the best Challenge/Response system I've ever used, and what beats all is that it's open source.

And it won't be long before spamers all spoof the source address of all e-mails as: dilbert@dilbert.com, or whatever their mailing list is...

Having looked at commercial and OSS systems, I reccomend TMDA over any other existing system. It has a great web interface for your n00bs, and way more features and temporary addressing tricks than anyone else. It's light years ahead.

Then, if you added a dozen more equally clever features, and a nifty web interface availible, you would have TMDA

:)

Almost all of the questions I have seen here about challenge/response systems have already been answered in the TMDA FAQ. If you have a question about how these systems work, try looking there first, you may find your answer.

The confirmation messages are sent to the Return-Path: header address. If you're using TMDA (or similar products) correctly, you will receive the user's confirmation request and be able to confirm delivery of your original message.

Click here for an explanation of the TMDA way.

-- Gun

I disagree. It's not too drastic. I work for an ISP, and we recently piloted a similar program using Tagged Message Delivery Agent. I must say that it works flawlessly with almost zero false negatives. We even have a web interface so that people can go and look into their pending queue to manually approve or reject messages. Unconfirmed messages are automatically deleted after a week. For the mailing list problems Mr. Minh mentions in the parent post, this has proved to work great. When one of our customers gets a bank statement, he or she can manually approve that email for delivery. The approval adds the bank's from address to the user's whitelist, and all subsequent emailed bank statements pass through without the need for confirmation.

Read the TMDA FAQ and you'll get answers to many questions about the process. In addition, it will explain to you how you can set up your list so that less than 10% of your legitimate senders never even see a confirmation message. It explains how to handle mailing lists as well.

This IS the current answer because it is a mechanism used for delivery once the mail server has receives the message. It does not require all participants use it, yet it performs beautifully for those who choose to use it. Until the SMTP protocol and related software are re-written (and everybody upgrades en masse), this is definitely the answer. I promote the solution anywhere and everywhere I have the chance.

-- Gun

I disagree. It's not too drastic. I work for an ISP, and we recently piloted a similar program using Tagged Message Delivery Agent. I must say that it works flawlessly with almost zero false negatives. We even have a web interface so that people can go and look into their pending queue to manually approve or reject messages. Unconfirmed messages are automatically deleted after a week. For the mailing list problems Mr. Minh mentions in the parent post, this has proved to work great. When one of our customers gets a bank statement, he or she can manually approve that email for delivery. The approval adds the bank's from address to the user's whitelist, and all subsequent emailed bank statements pass through without the need for confirmation.

Read the TMDA FAQ and you'll get answers to many questions about the process. In addition, it will explain to you how you can set up your list so that less than 10% of your legitimate senders never even see a confirmation message. It explains how to handle mailing lists as well.

This IS the current answer because it is a mechanism used for delivery once the mail server has receives the message. It does not require all participants use it, yet it performs beautifully for those who choose to use it. Until the SMTP protocol and related software are re-written (and everybody upgrades en masse), this is definitely the answer. I promote the solution anywhere and everywhere I have the chance.

-- Gun