Slashdot Mirror

• Actively manage my whitelist. For example, if I needed to send a resume, I would make darned sure that the prospective employer's domain was on the list.
• Use challenge-response only in conjunction with other antispam tools. My system is roughly: if I know it's spam (tagged address known to be in spammers databases), it gets trashed. If spamassassin or spamoracle thing it's spam, I refer to tmda for possible challenge/response. Otherwise, the mail gets delivered.
• Warn people about the system. If I know that someone new is about to send me email, I warn them: "You might get an autoresponse back. If you do, just hit 'reply'."
• Use some care in writing the challenge email. Trying to craft a letter that is understandable to non-geeks wasn't that easy.

> How do two people with challenge and response communicate?
> If the challenge always gets thrugh, then the spammer will just issue cahllenges as spam.
> If they don't get through, then you would have a nasty mail loop.

In TMDA (a challenge response system in python) at least, when you send a email to somebody, they don't get a challenge when they answer. It's logical because if you send him an email, you know he will not spam you :)
So i assume earthlink system will act the same.

Take a look at some of the projects that scare the hell out of professional spammers:

I honestly don't think the projects you mention scare spammers. There's plenty of ways of getting around SPEWS and spamhaus and other "realtime" (ha, yeah right) blackhole lists.

I think the only thing that would REALLY put a damper on a professional spammer's day is if a majority (or at least, all of the major mail sites such as AOL and Hotmail and Yahoo!, etc.) started using TMDA.

If it becomes impossible to send unsolicited mail from a forged return address that hasn't at one point been authenticated as a valid return address, because everyone is using TMDA-like mail exchangers, then spammers will get shut down pretty quickly.

-- Dossy

You should check out TMDA. TMDA offers challenge/response based whitelisting so that unknown senders can mail you after an initial confirmation. With a correct TMDA installation legitimate senders will only need to confirm once, after that they are added to the whitelist and future mail passes through automatically.

I think that it would be better to focus on server side protocol changes. The author of qmail hosts this site that talks about on alternate protocol to SMTP. Note that this would be entirely a server side change (it would affect relations between mail servers, while leaving existing protocols for client/server communication).

I sent you (Larry Seltzer) an email. Instead of using expensive certificates, I propose that we add a new type of record to DNS (call it an smtp record for now, someone can always come up with a better name later). The new record would tell what IPs (or FQDNs) are allowed to send email with a certain domain. For example, if an email address is me@slashdot.org, then only mail servers with SMTP records for slashdot.org would be allowed to send an email from me@slashdot.org. If a different mail server tries to send it, the receiving server can refuse the email.

Also check out tmda.net. It uses a number of methods to prevent spam, including temporary addresses and whitelists built by challenges (and client actions). Unlike the previous two proposals, this requires client changes (on the receiver's side), but it does not require others to change the protocols they use. Except for the challenges, senders and intermediary servers do not even need to know it exists.

All three of these proposals could be started more simply and with less additional infrastructure than the certificate idea. The first two require changes to the way things are currently done, but only on the server side. The third is even simpler, only the receiver has to make changes (btw, these are both client and server changes).

I disagree... After installing TMDA I have been getting more spam. Actually it is quite funny, since I don't see any of them normally. I just checked all my spams with tmda-pending (I was bored), and I got the Nigerian email a couple weeks ago...

So what does TMDA do? It sends a reply asking to confirm that address. I receive a follow up from the Nigerian people. TMDA sends another reply. I get the Nigerian peeps again (they have a ton of gold just for me). TMDA sends a reply... this happened eight times.

So you can see that this will only increase spam! ;)

Yes, not at all!

I can think of Active Spam Killer (ASK), TMDA and Qconfirm (QMail Only).

I personally use ASK and it works well. TMDA also works very well but requires complete control of your mailserver to install. Qconfirm is Qmail only .

All these are absolutely free.

This answers how TMDA deals with this problem. I don't know how other challenge/response email systems deal with it.

There is an IETF draft which describes how automatic, machine generated responses are supposed to be formatted, so that they can be easily identified as such. The primary author of TMDA (Jason Mastaler) has, I believe, incorporated all of the recommendations of that draft which are applicable to TMDA. So, if other automatic response systems (like TMDA) are also following the recommendations of that draft, then all of these auto responders should be able to interract.

This answers how TMDA deals with this problem. I don't know how other challenge/response email systems deal with it.

There is an IETF draft which describes how automatic, machine generated responses are supposed to be formatted, so that they can be easily identified as such. The primary author of TMDA (Jason Mastaler) has, I believe, incorporated all of the recommendations of that draft which are applicable to TMDA. So, if other automatic response systems (like TMDA) are also following the recommendations of that draft, then all of these auto responders should be able to interract.

With TMDA you can make a 'dated' address which would allow anyone who uses that address to send you an email for a certain amount of time.
Example from http://tmda.net/config-client.html

jason-dated-989108708.a17f80@mastaler.com

This particular address expires on Sun, May 6 00:25:08 2001 UTC, which is exactly 5 days after it was generated. TMDA time intervals can be set in years, months, weeks, days, hours, minutes, and seconds. Once a dated address expires, messages sent there must go through the confirmation process. Use of strong cryptography insures that the timestamp can't be modified.

If you are interested check out my website at http://www.standblue.net/.

There is an email address listed on the contact page, but expect an immediate challenge. ;)

BTW, mailblocks.com isn't free; it's $10/yr. However, that's still only half what fastmail.fm charges annually for their spam filtering service (with SpamAssasin).

Having all email routed to my inbox means that my figures above include dictionary attacks.

Using tagged addresses also runs up the total a lot. Every time I give out my email address, either on a registration form or in a public posting, I use a different tag.

I started tagging addresses in the early days of spam. Remember when we foolishly thought we could attach a disclaimer to usenet posts along the lines of "send me spam, and I'll bill you $50 under the anti-fax laws"? Well, I was dumb. I figured that in order to "prove" that unsolicited email was unsolicited, I had to have some proof of how the spammer got my email address, and that I had a clear disclaimer.

The good news: I have a pretty good idea of which of my online activities generate spam (e.g., posts to control.cancel and *.test, my NIC registrations, and usenet group-creation votes all seem to be popular for the spam-database trollers)

The bad news: I can easily get hit 30, 40, or 50 times for any one mass-spewing a spammer decides to do.

The totals above contain NO false positives -- they're all tied to tagged addresses which only produce spam. Not included are the 50 or so false negatives I get a day, which get tackled through other means.

Thank Ghu for tools like procmail, tmda, and spamoracle.

And how exactly do you propose to "inform the system that you are a real user"?

The answer to his question is, "By using TMDA, of course!". TMDA is an automated whitelist management program. I agree that manually managing whitelists is next to impossible, even at the individual level. But that's why TMDA exists, to automate that process.

And it's currently being tested on a large scale. GMANE is using TMDA as a mechanism of blocking spam for some 3500 mailing lists.

I wish I could rewrite the original question so that it was more clear that TMDA is an automated whitelist management program. Cuz I don't think the guy understood that. And he answered as if the question were suggesting that the ISP manage all their user's whitelists.

$.02

1. The type of system you describe (a whitelist-based system) has already been implemented many times over. TMDA, etc.

2. No. I want to be able to receive mail from people I don't know who have read my writing, used my software, or checked out my website. And I don't want such people to have to jump through hoops to become "designated friends" before they can do it!

What is the best way to discourage spammers from spamming?

I prefer postfix with Tagged Message Delivery Agent myself.

This can be done via TMDA, a whitelist-centric anti-spam package. You can create sender-based addresses so that the originating org has a direct pipe to you inbox, but that anyone else trying to use the address will have to confirm their identity first. You can remove that direct pipe at your leisure.

Moral: spammers hoover slashdot, so don't post your email here, ever.

Screw that. I refuse to hide or obfuscate my email address. I've been using the Internet for 15 years. I remember the time when the Internet was mostly spam-free, and people rarely forged email addresses even though everyone knew how to.

My real email address is deven@ties.org -- this is my primary personal email address, not a spam-trap address. I know that the spammers are harvesting address from Slashdot and everywhere else. I don't care. Let them have the address. I've never hidden it, and I never will. I'm stubborn that way. (It's akin to refusing to change your lifestyle in response to terrorism, even when you know you're at risk...)

Of course, since I don't hide my email address, I get tons of spam, along with "Joe job" bounces/replies for spams forged in my name, plus more bounces copied to postmaster, since I receive postmaster mail for several domains. Bring it on! It just provides me with a larger corpus of bogus email to use for Bayesian filtering, or whatever other technique I may experiment with...

I firmly believe that a technical solution will be required to solve the spam problem. Legislation won't prevent the virtually-untraceable international spams, and may not even prevent local ones if it's not zealously enforced. Social controls haven't been effective. We need to prevent the spam from being delivered in the first place, or at least mark it as suspicious so legitimate mail doesn't drown in the noise so easily.

Beyond basic filtering like SpamAssassin and Bayesian filtering, there are other technical solutions worth exploring. Human validation techniques like TMDA might help. Finding a way to punish spammers and drive up their costs, such as E-Stamps or selling interrupt rights (original paper: HTML or PDF), might be effective. (But likely a higher barrier to legitimate mail.) Some sort of PGP-style Web of Trust might be very effective if done well, but it would be difficult to build. Perhaps some "soundness" principles could be borrowed from Usenet II to create a similar system for email...

Let's cross our fingers and hope to find a truly effective solution (or combination of solutions) in the near future!

For what it's worth, there's another antispam technique that might merit some of your consideration. The technique is effective because spammers want anonymity. They want to continue to use fake return addresses. And that can be exploited. I use one such system called TMDA. As a consequence, I'm not afraid to advertise my email address on slashdot (for example).

TMDA isn't really an antispam system, per se. It's an automated whitelist management system, with a bunch of really useful extra features thrown in for the heck of it. But at its heart, TMDA forces you to have a real working email address in order to get into my mailbox. Now of course, spammers might choose to respond by using real return email addresses. Personally, I think that would be a very positive development. In the mean time, it's a pretty effective technique.

TMDA has some competition, too. Active Spam Killer does similar things, although I haven't used it.

TMDA

Since I first started filtering spam (several years ago), I have attempted to send mail (when it looked reasonable) back to the sender. Later I added a URL for the sender to use to send a plain message. More recently I just switched to TMDA. It's a much cleaner solution.

You can offload the burden of handling false positives by handing messages determined to be spam off to TMDA. Then the sender gets to tell you that it's not spam.

Sure, but wouldn't it be a whole lot easier to just use TMDA?

There are a number of things that you can do with mailing lists. But unfortunately, since there are so many different types of mailing lists with so many different configuration options, there typically isn't one single, easy to set up solution.

For what it's worth the TMDA documentation on the web site is pretty good. As for me, I tend to use TMDA sender addresses to subscribe to mailing lists. More details here. And, of course for anything more complicated than the docs can handle, there are the TMDA mailing lists, which can be accessed without subscribing through GMANE

Once you've gotten a little bit comfortable with TMDA, this is a pretty good reference for using it with mailing lists.

Good luck.

There are a number of things that you can do with mailing lists. But unfortunately, since there are so many different types of mailing lists with so many different configuration options, there typically isn't one single, easy to set up solution.

For what it's worth the TMDA documentation on the web site is pretty good. As for me, I tend to use TMDA sender addresses to subscribe to mailing lists. More details here. And, of course for anything more complicated than the docs can handle, there are the TMDA mailing lists, which can be accessed without subscribing through GMANE

Once you've gotten a little bit comfortable with TMDA, this is a pretty good reference for using it with mailing lists.

Good luck.

The main problem with laws such as these is that they just will not work.

Of course they won't work. But that's not the point. SPAM is not a technological problem. It's a social problem. Technology alone can't solve it. And, unsurprisingly, the law alone can't solve it. But combined, there's hope.

The reason that I want a law against spam is to continue to keep spammers using fake email addresses. To keep them from going to actual email addresses. If spam is illegal, then there's a disincentive to be easily tracked by using a real working email address. Why do I want spammers from using real email addresses? Because the most effective way I've seen to block spam is TMDA. And it's spam blocking ability depends on spammers not having working email addresses.

TMDA requires users to authenticate themselves by verifying that they can be tracked to a working email address. If they can, then they can send me email. Spammers don't do this. So they don't get into my mailbox.

But if everyone started using TMDA, then there would be an incentive for spammers to start using real mailboxes in order to get around the fact that everyone is blocking them. But a law which disincents getting tracked makes TMDA stronger.

The fact that a law, alone, won't fix spam doesn't mean that it won't help. And, frankly, I think the technology needs help, because this is a much harder problem than technology is accustomed to solving alone.

$.02

It sounds like you're using TMDA. Or, if you're not, you should be. :) Check out my related post on this story.

Until we have quantum computers, we're stuck with black lists, which work pretty well anyway.

... or software managed whitelists. This software assumes that everyone is blacklisted until they can prove otherwise. This system will work until spammers start using real, working return mailboxes. At which point, 99% of the battle will have been won.

I'm really excited about all of the neat stuff happening with Bayesian filtering and related technologies, but I just wanted to put in a plug for TMDA, Tagged Message Delivery Agent, which uses a whitelist-centric strategy. Since I began using it, the amount of spam I have to look at is virtually at zero. If you haven't read about it yet, check it out.

People, this is residential service you are getting here.

So?

If you need to run your own mail server you need to find out about that when you sign up for service.

I did, and it's allowed. And I don't need to run my own mail server. I want to run my own mail server. And I want to run my own mail server because TMDA is the most effective spam blocker that I've tried. It's a *ton* easier to use with your own domain and mailserver.

A typical residential user never needs to connect to any SMTP relay except the ones the ISP provides.

If you're responsible enough to run a mail server, and you really NEED one, get a real account.

A *real* account is not defined by the nature of the IP address. My service uses dynamic IP address only because static IPv4 addresses are in too high of demand. They simply require oversubscription. That's life. When (if) we ever move to IPv6, what will your solution be then?

The terms of service of my cable modem allow for running a mail server. That is a term that is between them and me. Stop trying to interfere with it, please... unless of course, you're willing to fund the difference in monthly costs for the static IP address. Didn't think so.

It's interesting to see that the talks focused on heuristics exclusively. The main problem with all of these techniques is that they may classify legitimate email as spam as well.

Since two months, I've been using the Active Spam Killer (ASK) now, and this has been mostly successful. In short: If a person writes me an email, they will have to confirm the mail, unless they are on my whitelist or the email contains a magic key (which is included in my sig and will thus be included in a reply). Confirmation also places a person on the whitelist, automatically. Since most spammers forge the From: address, they are not able to confirm their mail, even if they wanted... -> Pretty much no spam (dropped from approx. 20-30 spam-messages per day to 1-3 per week). Sure, if you order a book at amazon, their computer might not confirm. Thus I look into the confirmation queue from time to time whether anything in there is legitimate. Thus far it has not yet occurred that a person would not confirm his/her email, by the way. ASK is well documented, written in python and easy to setup.

There is another similar system (which I haven't checked out): TMDA.

I am wondering why big corporations, universities, ISPs are not providing such a (preconfigured) system as an option in their email packages ...

Of course, it requires some fiddling with MTA setups, but it could probably be done without if it got popular enough...

If this is the first email from the sender, the email is held in a temporarily holding location and the sender is sent an email with a challenge which only a human is capable of completing. The sender only needs to complete the challenge once, and their email, and all future emails that they send you, will immediately be placed in your inbox.

TDMA does that and it's free ;)

1. Some of my friends don't understand that they need to reply to the query message when they use a new address
2. my ISP doesn't allow me to run a mail server from my broadband line anymore

The problem with filtering is that, in essense, you've already paid for the email. Think about it, the email has already been received by your system, so you've paid the cost of the spammers email.

Using a "token" or "tag" based system means that spammers would have to request a token from you before they could send you an email. Their costs now go up (as does yours) as they have to filter your email (and the thousands of others replies to token requests) to get the token to send you. In the long run, companies could provide the service of token processing so that your personal email would only be accessible to white-listed addresses (thus moving your costs to the token-processing server).

See TMDA as one approach to this problem.

See the TMDA website.

and I'll tell you why. The only reason businesses don't get as much spam compared to home users is because of one difference. The average home user doesn't have an IT department at their disposal to help fight spam. At the company I'm at we still get tons of spam for the same reason home users do. Too many people treating their work account like their home account and signing up for lists and things they shouldn't be. Spam has gotten so bad that we're considering implimenting the silver bullet of spam filtering, TMDA. The only problem is that this is very difficult to impliment and it goes purely on a whitelist only basis. Spam is everywhere and anyone who says differently is either downplaying the problem, or living in a bubble.

Read the FFAQ.

Here's an anti-spam service I'd pay for.

• Have a system like Sneakemail that gives me a random email address to use for signing up for stuff on the web.
• Use Spamassassin (or a similar tool) to filter mail.
• Include all of the Sneakemail addresses in the whitelist automatically so they get through. If they start getting spam, there should be a way to set them to bounce. This makes sure that receipts and newsletters get through 100% of the time since they are often incorrectly tagged as spam.
• Anything that scores under 5 or so from Spamassassin gets delivered automatically.
• Anything else has to go through a TMDA like system to authenticate.
• A second (and third?) email address that always goes through TMDA. This would be used for mailing lists and Usenet.
• A web interface, IMAP, and POP3 for downloading mail. Allow people to forward their filtered mail to another email address also.

I think this system would have the best balance of a spam-free inbox and low risk of losing important mail. How much would I pay? I think I'd pay $5 a month if it was set up well. $10 a month is too much for sure though, unless you offer some other features. There is already a cool webmail service called MailSnare that uses TMDA for only $19.95 per year, so you'd have competition :-)

Not really something everyone can just implement for themselves, but what you are talking about requires no modification to SMTP or any other protocol. It's called Tagged Message Delivery. You can find out all about it here.
If all e-mail providers would implement such things, we would all see a lot less SPAM.

Welcome to TMDA. It even has a SMTP proxy for those email clients that can't call it directly.

You might want to check out TMDA - Tagged Message Delivery Agent. Seems to do pretty much what you're talking about.

SpamAssassin does fine for me, but if you want to go whitelist, then you can do a whole lot worse than TMDA.

Until I started using TMDA, just recently. 100% effectiveness, no more spam. It works on the whitelist-centric strategy of only allowing mail from known senders through, and allowing unknown senders to confirm themselves.

You may share my original fear: that important clients wouldn't be able to get through. The fact of the matter is that with a well-populated initial whitelist, you've already taken care of most of those scenarios.

For the remaining population of legitimate senders that aren't whitelisted, you may worry about them not taking the time to confirm themselves. But as the TMDA FAQ notes, we used to have the same worry about confirming mailing list subscriptions, and now that's completely standard. If someone took the time to write you an important message, they'll probably take the few seconds it takes to respond to a confirmation request once and for all. But, my friends, as the article notes, I think we've reached that point where such minor inconveniences are well worth the net drop in junk mail.

No, TMDA does not stop spam at the source, and it barely reduces the resouces required to receive spam, but it does address the most notable waste of human resources, because once you start using it, you don't have to look at spam any more. If you're an end user looking for a fix, check it out.

A better way to implement white lists is TMDA. If it don't know the one that is sending the mail, it automatically sends an email asking for a confirmation, so that defeats most spammers and gives normal people the opportunity to not be ignored by a plain white list scheme.

TMDA offers those who want it the ability to filter e-mail through a confirmation process (or, you can generate "keyword" or "dated" addresses for temporary use in newsgroups and other high-harvester areas). My spam went from several tens of spam messages a day to zero after spending a couple of hours with TMDA.

This solution doesn't do anything about bandwidth (since you will still get the same amount of spam traffic at your mail port), but it's a fuzzy-warm feeling to be in control of your own mailbox for once.

i use linux for many or the reasons he stated, though i came from vendor unicies, not windows. and i deal with pretty much no spam at all:

http://www.tmda.net/ takes care of spam for me.

Do we really want Big Brother stepping in to regulate our network activities? As much as I believe spam is an abhorrent practice (I am a devotee of TMDA), I don't believe the government is acting in my best interest. How, exactly, do they determine what is spam and what is not? How soon will it be before the feds come knocking at your door after unleashing a flood of e-mails requesting your money back from an on-line vendor?

The bad thing about all this is that the government is making the rules up as it goes along. Anybody who believes this is A Good Thing is deluding themselves. Every time cases like these are prosecuted successfully, a little freedom is taken away from the governed, and a little more power is granted to the governors.

Essentially, it throws the parsing problem right back in the spammer's faces: They must answer a fuzzy logic question in order to get into your inbox once and for all. It is similar to challenge/response routines in network connection code to prevent spoofing. The most interesting part from the intro:

The way TMDA thwarts incoming junk-mail is simple yet extremely effective. You maintain a "whitelist" of trusted contacts which are allowed directly into your mailbox. Messages from unknown senders are held in a pending queue until they respond to a confirmation request sent by TMDA. Once they respond to the confirmation, their original message is deemed legitimate and is delivered to you.

Bayesian filters to me, seem to work if you are a dull person without many changes in your life. For ex, if you constantly get spams with the word Madam in it and you later on get a sex change, you will need to recalibrate your filters. (Probably not the most pressing thing on your mind, so you'd lose a few authentic mails.)

Just some thoughts.