Postfix's Creator Outlines Spam Solution

SATAN writes "Wietse Venema started out as a physicist, but became interested in the security of the programs he wrote to control his physics experiments. He went on to create several well-known network and security tools, including the Security Administrator's Tool for Analyzing Networks (SATAN) and The Coroner's Toolkit with Dan Farmer. He is also the creator of the popular MTA Postfix and TCP Wrapper. SecurityFocus chatted up Venema to talk about software security, how to improve the code quality, what solutions we might have to fight spam successfully, the principle of least privilege, and the philosophy behind the design of Postfix. Venema is currently a researcher at IBM's T.J. Watson Research Center."

Comments

It's easy

[smartin]

Just get everyone to sign their mail including companies that send you receipts and opted in spam.

I would be happy if I could reject any mail that is not digitally signed and then manage the signed mail by signature.

Re:It's easy

[crunzh]

How would this signature fix anything?

Re:It's easy

[Xugumad]

As a big fan of signed e-mail, I see something like this:

Anything signed by someone I trust, arrives in my inbox. Anything signed but not by someone I trust, goes into a holding box from which I can fish e-mails I want. Anything not signed, or with a corrupted signature is rejected as unacceptable at the MTA level.

Now, anything arriving in my inbox can only be spam if someone I know has a hacked system, which should be rare AND I can contact them to tell them to fix it, because I know who it is from the signature (unlike e-mail viruses that could be practically anyone I know). This means that I know when I get e-mail in my inbox, it's worth me looking at.

Unexpected e-mails are still an issue, and may get lost, but frankly that happens anyway (I get somewhere over 200 spam per day, only a couple of dozen of which make it through enough filters for me to even glance at the subject line).

Filtering could be multi-stage, too; regular inbox for trusted people, a secondary inbox for people who I have been introduced to (for example, by a mailing list), then signed but unrecognised, and then everything else.

Re:It's easy

[crunzh]

You can do this with spf, but that have not solved the spam problem.

Re:It's easy

[Xugumad]

That's because there's very little actual use of SPF. I can do with it X.509 certs (Thawte do free e-mail certs at https://www.thawte.com/secure-email/personal-email-certificates/index.html - highly recommended), or GPG, as well, but the problem is getting uptake high enough for it to work.

Re:It's easy

[crunzh]

Its not only due to small uptake of spf, but your solution doesn't solve the spam problem. I (and many others) need to recive e-mail from stranger I don't know in advance, as I don't know their signatures, I cant filter on them and would have the normal spam problems. The problem with spam, is that most people need these email from stranger, and a solution that don't handle them does not solve the problem.

Re:It's easy

[Xugumad]

Maybe not solve, but I imagine most people get the vast majority of their e-mail, and ALL critical e-mail, from people they know in advance. This means that "uncertain" e-mail can be ignored safely for significant lengths of time, confident in the knowledge that if your boss e-mails you, you'll still get notification ASAP.

Make sense?

Re:It's easy

[Chrisq]

There are some people who receive most of their email from strangers, ans this would not work for them.

If you receive most of your email from people you know then this helps, as their emails can go directly as authenticated into the inbox. You would receive emails from new addresses in an unknown box, where once only for each address you would have to decide whether it was spam or not.

Id does not cure spam 100% but for most people it improves things.

Re:It's easy

[vidarh]

How often do you get spam where the "From" address is someone you know? Nothing is stopping you from doing this today - in fact there are many packages providing "greylisting" which improves on it by sending a message back allowing the sender to "prove" they are not a spammer - no real spammers take the hassle (if the from address is even genuine).

Re:It's easy

[antifoidulus]

Dude, if we could get everyone to do something then there would be a super easy way to stop SPAM: namely get everyone to stop clicking on stupid shit.

Not only does that action give spammers income, it is the #1 vector for the spread of botnets.....

Re:It's easy

[I'm not really here]

I get mail about 5 times per week where my email address is in the from email address... so I'm not sure this system would work too well anyways, especially since someone would just figure a way to "sign" the emails anyways. It would just be an arms race, and the spammers have lots of money to work with.

Re:It's easy

[theCoder]

If the email from the stranger is properly signed, then you look at it. If it is not spam, then you mark the sender as someone you trust. If it is spam, then you mark the sender as a spammer, and further messages from that identity are rejected. Spammers cannot afford to make a new key pair for every spam and ensure that the public key gets to the PKI that you use to validate messages.

Of course, there are challenges -- users misconfiguring their keys, the PKIs not being available, the PKIs themselves being overloaded with spammers trying to push keys there anyway, etc. Frankly, I think these are all challenges that can be overcome, and digital signing of email would go a long way towards helping with spam.

Oh, and someone else mentioned SPF. The problem with that is that it only signs and trusts the server. I don't care (or rather, I shouldn't care) about the server sending the email. I care about the user sending the email. If everyone digitally signed their emails, we could theoretically go back to open relays without any additional problems, since we would be verifying the actual person sending the email, not whatever machine they happened to be using.

Re:It's easy

[swillden]

Dude, if we could get everyone to do something then there would be a super easy way to stop SPAM: namely get everyone to stop clicking on stupid shit. Not only does that action give spammers income, it is the #1 vector for the spread of botnets.....

Actually, it doesn't give spammers income. Spammers don't care if you click the links. By the time you're deciding whether or not to click, the spammer has already done his job and made his money.

If you think not clicking links is gonna convince all the get-rich-quick scheming fools to stop paying spammers to send their crap then you sadly underestimate the supply of fools.

Re:It's easy

[seek31337]

Yeah, like customer support centers or contract companies already know everyone who will contact them.~

Re:It's easy

[smartin]

That used to be true. postgrey worked great for about 6 months, it no longer does much as the spammers adapted.

Re:It's easy

[Xugumad]

Customer support centres are going to get spam, nothing to be done about that, BUT it dealing with e-mail doesn't interrupt what they're doing, it IS what they're doing. On the other hand, if I'm coding and see the "Hey, you got e-mail" thing wave, I generally want to check it soonish to make sure it's nothing critical, which interrupts what I am doing.

Contract companies are about the worst case scenario, however that doesn't mean MOST people wouldn't benefit.

Re:It's easy

[Sloppy]

A signature means there is someone taking credit/blame for the message: an identity. Identities can accrue reputation. This guy is a spammer (and his messages can be discarded before wasting any human's time) and this guy is not a spammer, so go ahead and expose the user to his messages.

Re:It's easy

[mr_mischief]

It wouldn't have to stop all of them. A 70% reduction in spam would be a wonderful thing.

Re:It's easy

[fishbowl]

>How often do you get spam where the "From" address is someone you know?

I get a disturbing amount of spam "From" ***ME***.

But my favorite spam is an offer to refinance a post office box that has been rented under an assumed name. I have not yet identified the vector for the disclosure that was necessary to make that interesting to spammers, but I suspect amazon.com.

Re:It's easy

[nabsltd]

No greylisting implementation that I know of requires the sender to do anything special to "validate" their e-mail. What you are thinking of is a challenge-response system, and those suck because they create blowback spam.

Greylisting works on the principle that most spam comes from systems that don't follow RFC because they do not retry if they receive a temporary error. The MTA with the greylisting implmentation always returns a temporary "4xx" error code for any e-mail with a "new" sender/recipient/source IP triple and stores the information in a database. The greylist server keeps returning a temporary error for anything that matches this tuple for the configured timeout (usually about 5 minutes). After that, it lets the connection through as normal (where other anti-spam measures may be taken).

This stops most bot networks from sending spam. It still works remarkably well, as I only use that and SpamAssassin with a reject score of 10, and I see about 1-2 spam e-mails per week.

Re:It's easy

[Intron]

Most customer support is not direct email, it is done by filling out a web form to avoid spam.

Re:It's easy

[Anonymous Coward]

er... or just white list your friends - everything else that's not filtered by a ISP spam filter goes into a holding folder which you can selectively white list... oh wait you can already do that. I get no spam on gmail, btw.

Re:It's easy

[Xugumad]

Possibly it's mostly an issue because I get so much spam, but the most useful people for me to whitelist (work addresses) spammers frequently use to e-mail me, presumably because they figure they're on the same domain and there's a good chance I'll have whitelisted them.

Re:It's easy

[Loualbano2]

That makes sense most of the time.

Except when you are looking for a job.

ft

Re:It's easy

[turtleAJ]

This means that "uncertain" e-mail can be ignored safely for significant lengths of time

LoL

d00d, you are so going to miss out on big cash hand-outs from Nigerian families...

Re:It's easy

[Anonymous Coward]

That doesn't help. You still have to sort through the spam. When you waste that time isn't the issue.

Re:It's easy

[Sancho]

That's interesting, and something I hadn't heard much of before now. In fact, I can't recall ever hearing about someone receiving spam spoofed from someone they know except when the someone's computer was compromised. And digital signatures won't help, in that case (what, you think that the person isn't going to check, "Save the password to my digital signature?"

I don't think there's a solution to the spam problem--at least, not with e-mail as we know it. Any time you need to accept pseudo-anonymous connections from untrusted computers, and for as long as those connections are extremely inexpensive, you're going to get unwanted connections.

Re:It's easy

[Sancho]

Ah, but luckily, you probably don't need to white list yourself. If you do send yourself e-mail frequently, you could pretty easily require that mail from yourself be cryptographically signed. Unless your key gets hijacked, you're safe.

Re:It's easy

[Sancho]

That's a interesting point. If the signing keys were managed by a third party, and we could blacklist self-signed keys, then such a system might work out. That said, I don't know many non-techies that would want to pay yearly for a signing key just to send e-mail. Worse, you'd certainly see spammers getting multiple keys under fake names, so I'm not sure that this would do much at all.

Re:It's easy

[DamnStupidElf]

The PGP web of trust or a hierarchical PKI solves the spam problem. People will revoke their trust in a key used for spamming, and be much less likely to trust a key from the same person again, and Verisign and other companies will stop issuing certificates to entities that send spam.

Obviously some spam will be generated because it's always necessary for new people to enter the trust network, but ultimately there will form a core group of trust or certificates that never send any spam because they're trusted by people who don't like spam. Getting into that group will require the trust of enough other trusted people as to make spam almost nonexistent.

The problem with the idea is getting enough people into the trusted group to make it effective at both preventing spam and incorporating new legitimate users into the trust group quickly and effectively so they can communicate.

Re:It's easy

[WuphonsReach]

How often do you get spam where the "From" address is someone you know? Nothing is stopping you from doing this today - in fact there are many packages providing "greylisting" which improves on it by sending a message back allowing the sender to "prove" they are not a spammer - no real spammers take the hassle (if the from address is even genuine).

That's not graylisting, that's challenge-response.

Which is a fundamentally flawed system because the majority of spam uses a forged address for the from/reply-to. So all the C/R systems do is annoy the person who's e-mail address was used for the forgery. (In fact - it's easy to envision a DDoS attack by abusing mail servers that send out C/R e-mails.)

Re:It's easy

[swillden]

It wouldn't stop hardly any of them.

Re:It's easy

[swb]

I just implemented on my home email system and it cut spam remarkably. For me, this means somewhere around 10 a week, where I was seeing somewhere around 25 a day.

Re:It's easy

[Xugumad]

> what, you think that the person isn't going to check, "Save the password to my digital signature?"

I haven't. Erm. Yeah, totally agreed, most people will check that box :) Well, in reality, I think most people will pull the password off it, but same net result.

Re:It's easy

[spectre_240sx]

In the end, though, if most people benefit, that means that spammers aren't going to make any money and will hopefully quit anyway.

Re:It's easy

[jaxtherat]

I work for a scientific consultancy, and they receive a lot of their email from people they've not received anything from before.

Also, in regards to your "Hey, you got e-mail" thing wave , you can turn that off you know. Most dev shops I've worked at recommended at least 2 hours a day no email time, and during project crunch times all email would be off. Period.

Re:It's easy

[Xugumad]

Alas, I'm a strange developer/sys-admin/support person cross-over, so I can't...

Re:It's easy

[Skrapion]

That's not exactly a solution to the problem. I think we all agree that doing customer support through a web form sucks.

Re:It's easy

[jaxtherat]

Bummer :(

Re:It's easy

[Skrapion]

Scenario 1: You've published a web page that lists the email addresses of several of your coworkers. Perhaps it's a corporate directory, or contact information for a project you're working on, or posts by multiple coworkers on a forum. Whatever. An email harvesting bot sees all these email addresses in the same place and assumes they might know each other.

Scenario 2: The email harvesting bot finds different email addresses on completely unrelated web pages, but since they share the same domain name, it assumes there's a chance that they know each other.

Scenario 3: The email harvesting bot finds your email address (sancho@example.com) on a web page and spams you with a list of common names in the From field. Even if there's a 3% chance that you know john@example.com, that's good enough for the spammers.

In short, there's plenty of ways for spammers to fake emails from people you know. And what's worse, without some sort of authentication, we don't know whether or not that persons machine is really compromised.

Re:It's easy

[jonadab]

You don't need SPF to do this if you only need to receive mail from known parties and/or with known characteristics (e.g., a known mailing list you subscribe to in the To: field, a known flag string in the Subject field, etc). Just write your filters to pick up legitimate mail and plop it in the folders you read regularly, and the spam goes into the unsorted, unfiltered inbox, which you only look through when you have time to kill and/or a reason to believe you've missed a message. This is essentially what I do now. And it kind of works. Almost.

This kind of system (i.e., whitelist filtering) is a long way from perfect, though, especially if you'd like to be reachable by people whose messages you (and your filters) don't know to look for. And I'm not sure what the digital signatures would add, honestly, even if *everyone* used them. The spammers don't know my friends' email addresses and whatnot to fake them. Okay, yeah, I'm sure the spammers *have* my friends' email addresses, but they don't know that those email addresses are ones that *I*, in particular, normally expect to receive mail from, so they're no more likely to send me fake mail "from" those addresses than any of the hillion jillion other addresses in their vast databases. Occasionally a spam message does trip one of my subject-line filters by pure dumb luck, but the volume of such messages is very low (lower than the volume of legitimate messages), so it's not a very big deal.

The bigger problem is that I can't easily make myself publicly reachable under such a system. I can't just stick my email address in the signature/footer/sidebar/whatever of my website/blog/slashdot-profile/whatever and thus enable people who read it and want to respond to easily contact me. I mean, yes, I can tell them "put the word todhsals in the subject line", but that's an ugly annoying pain and has a lot of shortcomings. And it has to be different from what everyone else is doing, because if it becomes standardized the spammers will quickly implement it. But digital signatures don't solve this problem at all, because there's no way for my filtering rules to distinguish between the digital signatures of strangers and the digital signatures of spammers. Spammers can generate digital signatures just as well as anyone else. New ones, based on new keys, for each message, if necessary.

The article discusses pull technology, wherein the sending mail server would inform the recipient's mail server, basically, "I have a message for your user with such-and-such address, and he can retrieve it with this magic token message id", and then when the recipient checks his mail the mail client software gets these notices (basically a list of tuples, where each tuple has, at minimum, the sender's mail server, whence the message can be retrieved, and the magic token or message id that you ask for to get it). There could also be other metadata, such as the sender's return address. The client software then goes down this list and attempts to retrieve the messages; any it can't retrieve right now (e.g., because the sender's mail server is experiencing technical difficulties), it can save the metadata and try again later. This isn't perfect either, but it would have certain advantages, most notably that the sending mail server uses more resources than the receiving mail server, which shifts the economics in the recipient's favor. To my way of thinking, that would be a big deal. You'd still get some junk mail, of course. You get junk mail through the postal service, after all, when the sender pays postage. But there are limits to how *much* junk mail you get with sender-pays economics. It's a manageable flow. You could potentially actually take the time to glance at each subject line individually and decide whether to read the message, without having to quit your day job to keep up.

Of course, the existing email infrastructure (notably, SMTP and POP3) can't be retrofit with this, because it's different at the design level. We'd need new protocols and, t

Re:It's easy

[drinkypoo]

Let's not forget that using firefox with FireGPG you can sign anything, including a forum post (although the lameness filter will, of course, filter it out so it doesn't work so you'll have to use a site less lame than slashdot if you want to sign your comments. It also has gmail integration, which works very nicely thank you.

Re:It's easy

[crunzh]

That might work technically, as would a lot of other spam solutions. I see several problems, how would you get a critical mass that will make it a viable solution? People will need to get new email clients, webmail solutions will need to rewrite their existing solutions, and everybody needs a certificate. Thats a lot of work, what would make it get implemented? There are a lot of spam solutions that would work technically, but is impossible to get implemented due to social and technical barries.

Re:It's easy

[RichiH]

And how much longer would the spammers get paid by their clients if they had zero income? Exactly.

Re:It's easy

[totally bogus dude]

Actually, I think it'd work just fine. Just handle it how we handle secure web sites: your organisation pays for a certificate identifying them. This is trusted by everyone else because they buy it from one of the big CAs whose certificate ships with all OSes, web browsers, or in this case, mail servers.

While a small business and even an individual can easily afford $20 a year for an instant certificate with basic "domain control" validation, it will quickly become prohibitive to continually buy new ones because your certificate keeps ending up on blacklists (because you're spamming, or relaying mail for spammers). In reality, end users won't care, because their email provider (company or ISP) will buy the certificate.

The problem is getting people to do it. If nobody signs their messages, then there's no benefit to requiring signed messages. Since there's no benefit, nobody will sign their messages.

Re:It's easy

[phoenixwade]

That's not exactly a solution to the problem. I think we all agree that doing customer support through a web form sucks.

Not when the support staff responds to the ticket in a timely manner. I believe it's all about contact and response. Customers want you to fix whatever the problem is, but they also want to know you are working on it. The failure in most web form support systems is wrapped around a failure in the staff in updating the client at regular intervals, even when there is nothing new to say.

Re:It's easy

[phoenixwade]

(you may find the following useful)

Your post advocates a

( ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

( ) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Re:It's easy

[phoenixwade]

(In fact - it's easy to envision a DDoS attack by abusing mail servers that send out C/R e-mails.)

No need to envision it, either by design or by accident, it already happens all the time. This is almost exactly the same situation as Backscatter Spam except, since you are using the C/R concept to verify the recipient, it's not as easy to fix.

Re:It's easy

[swillden]

They do have zero income. Hardly anyone who pays spammers to send spam makes any money. But that doesn't matter, because they believe they're going to make money. When it doesn't work out, they go on to something else and the spammer scams some other idiot.

The spam won't end until the world runs out of idiots.

Re:It's easy

[Anonymous Coward]

I hear this argument but then, if the problem is fools paying people to send spam, why send the spam at all, just say it happened and collect the fool's money.

Most of what I see in my inbox is scams, not spam, and wordlist poisonings to help game the filters. Again I would suspect if nobody cares about those emails being delivered/clicked, there'd be no reason to add that overhead.

Re:It's easy

[swillden]

I hear this argument but then, if the problem is fools paying people to send spam, why send the spam at all, just say it happened and collect the fool's money.

Why? It's so cheap to send the spam, why risk getting accused of breaching contract?

Re:It's easy

[RichiH]

Last time I read about this, the report claimed that at least medication & enlargement vendors make a lot of money. If you have sources that claim otherwise, please link them. I would be interested in their numbers.

Re:It's easy

[Sloppy]

If the signing keys were managed by a third party, and we could blacklist self-signed keys, then such a system might work out. That said, I don't know many non-techies that would want to pay yearly for a signing key just to send e-mail.

Modern certification schemes have no arbitrary limits to the number of CAs that can validate an identity, and users aren't placed into a position of having to totally trust or mistrust any particular CAs. This gets rid of the need for any blacklists; you can use whitelists or a very simple algorithm to guess trustworthiness of CAs. This gets rid of the need to pay a lot for signatures (in fact, for most people, it's free).

But certification is secondary anyway, because..

Worse, you'd certainly see spammers getting multiple keys under fake names, so I'm not sure that this would do much at all.

That's not a problem either, because you don't care so much as to whether the name is fake or not, as you care about whether that identity has been used to send spam or not. A spammer can generate as many identities as he wants, but the only way an identity can accrue good reputation is to not send spam.

Re:It's easy

[Kent Recal]

There's a very simple solution for that particular use-case: Use a contact-form (with a captcha) and auto-whitelist all e-mails received via that form. Most support cold-contacts happen via a webform anyways so that's not much of a change from what we have today. Subsequent mails would then be picked up by the whilelist. Yes, spammers *could* hack the form and get themselves whitelisted, but the incentive is probably very low - who wants to spam a support channel that's most certainly not interested in your blue pills?

Re:It's easy

[Kent Recal]

Why do you need to receive mail from absolute strangers? I most certainly don't.
And if I had to then I'd probably offer some kind of webform in order to not drown in spam...

If you're talking about notifications from webshops and the ilk, I'd imagine there could be a simple way to whitelist certain addresses. E.g. during registration at a website there could be a button "allow e-mail from this site" that automatically imports the public key of that site. Not rocket science and certainly much better than what we have now.

Re:It's easy

[Kent Recal]

Usually you only make the initial contact through a form and subsequent communication is done by E-Mail. Furthermore I could very well imagine contact forms that additionally offer an option to upload my public key to their whitelist, after entering a captcha. That way I could even make the initial contact my normal E-Mail with very little extra-effort.

Heck, if something like this is standarized they'd be firefox plugins popping up left and right that automate the process of presenting your E-Mail public key to any website that you sign up with.

Re:It's easy

[Kent Recal]

If the email from the stranger is properly signed, then you look at it. If it is not spam, then you mark the sender as someone you trust. If it is spam, then you mark the sender as a spammer, and further messages from that identity are rejected. Spammers cannot afford to make a new key pair for every spam and ensure that the public key gets to the PKI that you use to validate messages.

Actually generating keys can be very cheap when you don't need them to be secure. Spammers would probably not even use a RNG for their keys. Nonetheless I still like the idea because *if* it was widely adopted (a pipe dream) it would quickly become common practice to only really look at mail that is signed with a key that is on your whitelist. There'd be easy ways to let people manually put themselves on your whitelist (e.g. with a captcha-form) for cold contacts, that stuff could even be built into mail-clients. And it would raise the bar for mass-mailings significantly since spammers *can* crack captchas - but they'd have to do so for each individual recipient.

Re:It's easy

[crunzh]

I am jobhunting at the moment, so somebody who want to hire me might email me. Old friends I have lost contact with might want to email me. People who have changed their email adresse since last time we talked. People at my university (or related to my studies) might want to email me, that might be exam information, people who want to discuss my papers or other stuff. I have a long list, and I would assume a lot of people would too. Your solution needs a critical mass before it is even remotely usefull, how do you propose that it is possible to get people to get the PK in the firstplace and change all their email clients?

Re:It's easy

[Kent Recal]

Well, we're obviously discussing a technical solution that would require everybody to change or upgrade their mail infrastructure - with all the caveats.
If we could roll out such a system then there are no inherent technical barriers to key exchange. For example mail clients could automatically detect when you send mail to an address for the first time and perform a key-exchange before sending the actual mail. The key-exchange could be along the lines of: The remote MTA creates a captcha for you to solve and after you have solved it, it would add your public key to the recipients whitelist and accept your mail. For the sender all of this could be wrapped up in a simple dialog box ("Enter these letters"). Ofcourse each recipient could configure whether it wants to allow that kind of auto-whitelisting at all.

This can be implemented today, all the technologies exist. The obvious problem is mass adoption...

On a side-note: Despite the mass-adoption problem I do think this would make for an excellent open source project. It should be possible to encapsulate the functionality into plugins and modules for the various popular MTAs and MUAs. The main problem would be to find people who are familar and comfortable with the different APIs of all the qmails, Postfixes, Exims, KMails, Outlooks and Thunderbirds of the world. This lends itself to the open source model; create a standard, a protocol and some API guidelines, then attract a herd of developers to build the various implementations incrementally.

Re:It's easy

[AmiMoJo]

The problem is that businesses don't want to loose a single important email, and often get them from people they don't know (i.e. someone passes on their address, or gets it off a business card or their web site).

Arguably if you get 500 spams a day you might miss some good stuff in the giant shitstorm, but I find that between SpamAssassin and gMail I have a 99.9999% filter accuracy and the only failures are always false negatives.

Spam is a solved problem as far as I can see, or at least it would be if everyone implemented good filters.

Re:It's easy

[DamnStupidElf]

(you may find the following useful)

Your post advocates a

(X) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses:

For encryption, PGP keyservers need to support email address lookup, but for signing it should be sufficient to work with the key IDs (hashes) instead. Looking up certificates or PGP public keys by their hash from public key servers turns email address harvesting into a brute force problem for spammers.

() Mailing lists and other legitimate email uses would be affected.
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks

PGP and X.509 certs are secure against cryptographic attacks.

( ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it

If people can deal with a green/red bar for their SSL connections, they could deal with color coding for email addresses.

( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once

Almost. It would require cooperation from at least all the major web mail vendors at once.

( ) Many email users cannot afford to lose business or alienate potential employers

They can still benefit from signed messages while doing traditional spam filtering for unsigned messages. If they sign their own messages, they will contribute to the solution.

( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

It would be difficult to get enough people to revoke trust in an individual to make a difference, since those revocations would be publicly visible and make retaliation revocation possible.

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email

Partially, although RFCs are the de-facto controlling authority for email, and S/MIME and PGP are already established standards. Simply using them would require some motivation.

( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes

It would make getting rooted much more damaging to individuals who had their email identity stolen for spamming. CRLs would fix that address, but make it a pain for individual users. That, hopefully, would be good selection pressure for secure PCs.

( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft

Trojans stealing private keys could make more convincing identity theft. That's a local security problem, not a spam problem.

( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical

Obviously the idea has been around for a while. Sometimes it just takes a long time for good ideas to adopted.

Re:It's easy

[seek31337]

I dunno. Have you seen the people that run support lines? TINY!

IMHO, complete replacement of the protocol is the only thing that will make a change happen. Until there is a wholesale replacement, no one is going to use cryptography.

I mean, who does and actually gets work done? I'd liek to see a testimonial of someone who has more than 30 contacts from more than 4 entities that reguarly contact them outside of their own entity. This technology has been around for a /very/ long time.

Will Not Work

[mfh]

Your post advocates a

(x) technical (x) legislative (x) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

(x) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
(x) No one will be able to find the guy or collect the money
(x) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
(x) Microsoft will not put up with it
(x) The police will not put up with it
(x) Requires too much cooperation from spammers
(x) Requires immediate total cooperation from everybody at once
(x) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
(x) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

(x) Laws expressly prohibiting it
(x) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(x) Asshats
(x) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
(x) Willingness of users to install OS patches received by email
(x) Armies of worm riddled broadband-connected Windows boxes
(x) Eternal arms race involved in all filtering approaches
(x) Extreme profitability of spam
(x) Joe jobs and/or identity theft
(x) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
(x) Dishonesty on the part of spammers themselves
(x) Bandwidth costs that are unaffected by client filtering
(x) Outlook

and the following philosophical objections may also apply:

(x) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
(x) Blacklists suck
(x) Whitelists suck
(x) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
(x) Countermeasures must work if phased in gradually
( ) Sending email should be free
(x) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(x) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
(x) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(x) Sorry dude, but I don't think it would work.
(x) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Re:Will Not Work

[Anonymous Coward]

I don't think that Postfix's creator will understand this list. You used prefix checkmarks.

Re:Will Not Work

[pla]

(x) Spammers can easily use it to harvest email addresses - How do digital signatures allow easy harvesting of email addresses?

(x) No one will be able to find the guy or collect the money - If the signature doesn't validate, the message never even gets to your inbox. Yeah, people can still send bogus-signature messages, but they wouldn't get to anyone.

(x) It is defenseless against brute force attacks - Of what nature? Few organizations on this planet have the resources to brute force a valid bogus digital signature, and no one can do it on the sort of scale you'd need to send spam.

(x) Microsoft will not put up with it - Microsoft actually suggested a variant of that approach, except server-signed rather than user-signed.

(x) The police will not put up with it - 100% traceability of every message? They've wanted that for years. Now, if enough people realize it takes no more effort to actually send encrypted mail over merely signed mail, we could have a problem, but the GP didn't go that far.

(x) Requires too much cooperation from spammers - How? It depends on the fact that spammers can't cooperate.

(x) Requires immediate total cooperation from everybody at once - Easy to obtain, in that we really only need the mail server admins to cooperate, then everyone (who wants to get their email) will play along pretty damned quick.

(x) Many email users cannot afford to lose business or alienate potential employers - So they would cooperate even quicker.

(x) Anyone could anonymously destroy anyone else's career or business - How do you anonymously send a signed message?

(x) Laws expressly prohibiting it - Clinton actually made digital signatures legally binding under US law... So quite the opposite.

(x) Lack of centrally controlling authority for email - There, we agree. This would require a community rather than central effort.

(x) Asshats - Simply wouldn't get (or receive) mail.

(x) Jurisdictional problems - The GP didn't suggest a legislative solution, so not applicable.

(x) Willingness of users to install OS patches received by email - No one can save those who lick plugged-in lamp cords.

(x) Armies of worm riddled broadband-connected Windows boxes - Can spam as much as they want, it will never get read.

(x) Eternal arms race involved in all filtering approaches - Unless someone finds a trivial crack to RSA, not applicable.

(x) Extreme profitability of spam - Irrelevant.

(x) Joe jobs and/or identity theft - Would require either knowing their private key, or even in the easiest case, physical access to their machine.

(x) Technically illiterate politicians - Have IT staff paid to make sure the bits flow.

(x) Dishonesty on the part of spammers themselves - Once again, irrelevant, this does not require any cooperation on their part.

(x) Bandwidth costs that are unaffected by client filtering - We already get tons of spam, that wouldn't really matter, but it would get better as the spammers eventually give up.

(x) Outlook - To repeat, MS already proposed something similar.

(x) Ideas similar to yours are easy to come up with, yet none have ever been shown practical - Because of naysayers, not because of any real barriers to implementation.

(x) Blacklists suck (x) Whitelists suck (x) We should be able to talk about Viagra without being censored - Which all have what to do with signed mail???

(x) Countermeasures must work if phased in gradually - Like IPV6 or changes to daylight savings time?

(x) Why should we have to trust you and your servers? - You shouldn't - So block me, you'll know with 100% certainty who you've blocked.

(x) Feel-good measures do nothing to solve the problem

Re:Will Not Work

[Undead NDR]

(x) Blacklists suck
(x) Whitelists suck

(x) Greylists suck as well

Re:Will Not Work

[Rinisari]

Best. Post. Ever.

Re:Will Not Work

[nabsltd]

How do digital signatures allow easy harvesting of email addresses?

Certificates must be centrally stored or related to a trusted central authority. With this, you only have to break that central authority to get all the valid e-mail addresses. In addition, if all e-mail had to be signed, then people wouldn't be able to use throwaway e-mail addresses as easily, so every "give us your e-mail" would mean that a valid e-mail address was being harvested.

Few organizations on this planet have the resources to brute force a valid bogus digital signature, and no one can do it on the sort of scale you'd need to send spam.

You are thinking about forging e-mail, which isn't the problem. Spam could have a valid signature without being from someone you know. Like current spam and phishing, it could be from someone you might know.

Easy to obtain, in that we really only need the mail server admins to cooperate, then everyone (who wants to get their email) will play along pretty damned quick.

Once you finish that "easy" task, could you get to work on the trivial problems of a portable fusion reactor, transporter technology, and faster than light travel?

Armies of worm riddled broadband-connected Windows boxes - Can spam as much as they want, it will never get read.

The bot that controls a Windows machine probably also would be able to control the certificates the user had for sending e-mail. Thus, signed spam from people you trust.

Joe jobs and/or identity theft - Would require either knowing their private key, or even in the easiest case, physical access to their machine.

Cruise on over to Verisign and see just how easy it is to get a certificate with an e-mail address that isn't yours, and Verisign is one of the better at checking it. Also, read what I wrote about bots infecting the machine.

I don't want the government reading my email - So add full-message encryption basically for no extra work. And they can read your email now, so how would this make it any worse?

Because today there isn't a central certificate authority that is required to be used by everyone sending e-mail. This idea would make that a reality. That CA would have all the private keys for all the certs in a one-stop shop for the government. Encrypting wouldn't do any good, because it would be done using one of the same keys that was available in the CA.

Re:Will Not Work

[mr_mischief]

Greylists suck more than whitelists (of which they're just a clever automated variation, after all).

I'm not a spammer, but I've fought lots of spam. To do so effectively, you need to know the weaknesses of different techniques as well as their strengths. To catch a spammer, think like a spammer. So here are a few of the many issues greylisting has that I find pretty funny.

Want to really have fun screwing with a mail server? Find two greylisting recipients on the server. Send a message to one "from" the other. Unless the greylisting software is very smart, it generates a new message for everyone not on your whitelist for every message they send rather than using a counted forwarding system containing the received headers. This means it's often actively fighting the loop detection software on the MTA.

Want to get a greylist user blocked on other people's blacklists as a spammer? Find a greylist user whose software sends the body (or at least the subject) of the original mail along with the greylist notification. Send a bunch of spam to the greylist user you want to get blocked as a spammer "from" a few hundred working addresses of people who contribute to public blacklists.

Greylists that include the original body also could be used for even more nefarious purposes.

Some greylist software uses a serial number or a poorly generated character string to confirm replies. If you send enough email to a poorly implemented one from enough different email sender addresses then you can sometimes find a pattern. Once you discover the pattern, you can render the greylist pretty ineffective. I've even seen one guy who was silly enough to just MD5 hash the concatenation of sender, recipient, and yyyy-mm-dd date format. If you fuzzed enough to know that (about 20 seconds worth of it), you could hit his accounts with any kind of spam as a legitimate sender. What's worse is this guy was offering this for free download as a server-wide software package with the only configuration option being a per-receiving-domain "on" or "off". I feel really sorry for anyone who has their business emails depending on that for anti-spam.

Assume the greylisting is working perfectly and 80% of your email is spam, which you don't get now. You've whitelisted the people who send the other 20%. You're still sending out 80% of the number of messages you receive before you ever account for replying to a legitimate email.

Re:Will Not Work

[Tony Hoyle]

It gets fun when two greylisting servers start talking to each other..

Server 1: Hello
Server 2: Go away for 5 minutes then come back to me
Server 1: Go away for 5 minutes then come back to me
Server 2: WTF? I said 5 minutes!
Server 2: WTF? I said 5 minutes!
etc.
etc.

I kept two servers talking for *3 days* like that..

Re:Will Not Work

[Ortega-Starfire]

Just say it.

(x) lists suck

Re:Will Not Work

[calmofthestorm]

False, there are multiple standards for signing email. Certificate signatures are only one method, GPG is also used.

A larger concern is that once someone's computer is infected with a spambot you get their key.

Re:Will Not Work

[AnyoneEB]

How do digital signatures allow easy harvesting of email addresses?

Certificates must be centrally stored or related to a trusted central authority. With this, you only have to break that central authority to get all the valid e-mail addresses. In addition, if all e-mail had to be signed, then people wouldn't be able to use throwaway e-mail addresses as easily, so every "give us your e-mail" would mean that a valid e-mail address was being harvested.

For certificates to be meaningful someone has to sign them and the person looking at the certificate needs to know and to some extent trust the public key of the signing authority. There is no reason that signing authority needs to know every e-mail address they have signed a certificate for. Especially in the case of a set of CAs (like the current HTTPS setup), the CA could possibly sign a signing key for an organization granting that key rights to sign any email address under a certain domain. Another method could involve WoT to sign keys in a distributed matter (which unfortunately does not work very well; see: the number of PGP encrypted emails in your inbox) or DNSSEC to provide a signing key or set of signing keys for a domain via DNS.

Your other concern implies that making a certificate is somehow difficult. An anonymous e-mail service (or you using your own personal domain/subdomain or with a signing key limited to signing any address like you+something@example.com) could easily just generate a brand new certificate for every e-mail it sends. If you generate throwaways they are probably only for receiving e-mail anyway, so they would not need their own keys unless you want that e-mail to be encrypted... and I thought this discussion was just about signing e-mails.

Because today there isn't a central certificate authority that is required to be used by everyone sending e-mail. This idea would make that a reality. That CA would have all the private keys for all the certs in a one-stop shop for the government. Encrypting wouldn't do any good, because it would be done using one of the same keys that was available in the CA.

The CA signs your public key. It should never need to see your private key. If their process is such that they generate the key for you, then it is broken. That is a problem with the CA, not the idea of signed certificates.

The CA does have the power to generate a new fake certificate for any e-mail address. This would only fool people who had not received e-mail from that address before, though. Although if it was used just after an old certificate expired, then the attack could be transparent, so government coercion is still a reason to be weary of a centralized system, but reading your e-mail becomes a coordinated active attack because they have to trick you and the people you communicate with into accepting different keys.

Re:Will Not Work

[registrar]

(x) Countermeasures must work if phased in gradually - Like IPV6 or changes to daylight savings time?

Well, IPV6 is hardly working and DST is hardly a countermeasure. GP has a point here.

There is a strong network effect working against deployment of co-operative spam countermeasures. Somebody needs to kick-start the process. The challenge is not to find something that will work straight away, but to find something that is (a) initially no worse than the existing system, (b) allows co-operation, (c) provides other benefits. Anything based on digital signatures is likely to fill those criteria. Such a process is not very commercially viable because companies don't make money by introducing things that are 'no worse' and encourage co-operation with the enemy. But it can work with help: For example:

(1) Sites like gmail can store and sign my emails. Initially it doesn't have to be terribly secure or verifiable, just easy for me to send an email that is cryptographically signed. It's initially no worse than unsigned email, and could be made better.

(2) Government agencies (the tax department) could agree to receive certain correspondence by email provided that they were cryptographically signed. If the correspondence has previously been "postal only" then it is initially no worse. Government doesn't have to worry too much about upsetting the "client", and any cost can be justified if there are other benefits to the Government.

(3) There are occasions when companies might not mind being hard to email. E.g. customer support... they could permit email contact where there is a digital signature "as a spam countermeasure and to help verify your account details".

Great post, btw, thanks. Spam is solvable, but only if the persistent naysayers get told off by the visionaries.

Re:Will Not Work

[Random Walk]

Greylisting is done during the SMTP session. No new message is created, the original message is just rejected during delivery.

The scenario you're talking about has nothing to do with greylisting, just with some fundamentally broken implementation of it.

Re:Will Not Work

[wild_berry]

Give the guy a break. Like most all people, he doesn't understand how to configure Postfix right.

Re:Will Not Work

[olman]

Nice ranty response, here's why it's wrong:

(x) Requires immediate total cooperation from everybody at once - Easy to obtain, in that we really only need the mail server admins to cooperate, then everyone (who wants to get their email) will play along pretty damned quick.

Yeah. Right. And after the said server admin gets laid off the "new guy" will move back to business as usual.

More damningly:

(x) Willingness of users to install OS patches received by email - No one can save those who lick plugged-in lamp cords.

(x) Armies of worm riddled broadband-connected Windows boxes - Can spam as much as they want, it will never get read.

(x) Outlook - To repeat, MS already proposed something similar.

This is where the logic really falls apart.

You need to have a signature to send email, right? So you would (if this came to pass) get signature when you "sign up" to email service. So far so good.

Now then, this signature gets integrated into thunderbird/outlook/whatever so it exists in the said windows box.

When that box is pwned, it will spew as many signed spam messages as the botnet wants to send. Hey presto, your inbox fills up with spam with genuine signatures!

Not so different of the present situation where perfectly legitimate email source sends a torrent (ha) of spam.

Even if you block the SMTP port (and many ISPs do AFAIK) it won't help - You still have a big known target of windows + outlook to do the job for you.. And thanks to integrating outlook to windows, everything can happen happily in the background without the user noticing anything. Yay.

Re:Will Not Work

[Moxon]

You seem to have greylisting and challenge/response filtering confused.

Postifx

[Anonymous Coward]

Sounds like a mail server I used to use.

Is it some rival to JavaFX?

V for Venema

[digitaldc]

I always said if you had poorly-written code or spam clogging up your inbox, you would need a Venema.

Re:V for Venema

[Anonymous Coward]

Alright, then bend over buddy. Its time for your 'Venema'!

I lost a lot of respect for Wietse Venema

[SuperBanana]

...once I started reading his replies on the postfix-user mailing list. He's extremely blunt. While many are VERY helpful and detailed, a number are a sentence or two long that, paraphrased, consist of "you're an idiot."

However, he's nothing compared to Victor Duchovni (who works for Morgan Stanley, and is a major poster on the postfix-users list). His signature, and I'm not making this up:

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

Yeah, you read that right. 11 lines long...and this asshole thinks he's so fucking important, he lectures you about how to thank him so he can delete your acknowledgment/thank you as quickly as possible. He's often more willing to insult than help, and on numerous occasions, comes to the wrong conclusion. Worse still, he often presents his solution with complete authority and confidence, putting the helpless user on a primrose path.

Re:I lost a lot of respect for Wietse Venema

[Camel Pilot]

and this asshole thinks he's so fucking important

errr maybe he is... I mean important. If someone has specific and in depth knowledge and spends time helping the less knowledgeable, being an asshole sometimes come with the territory.

Re:I lost a lot of respect for Wietse Venema

[shis-ka-bob]

Let me get this straight. Two men help strangers to use free software and you are calling one an ass because he wants to 1) share the results of a fix with future users of the group, and 2) avoid flowery 'thank you' follow ups because he has high pressure work to do (there is no other kind of work at Morgan Stanley). Is that the jist of it?

Re:I lost a lot of respect for Wietse Venema

[superskippy]

It's quite simple. If you write an MTA, you have to be an asshole. It's the law.

Re:I lost a lot of respect for Wietse Venema

[Free the Cowards]

For some reason many people prefer to have polite, useless help than have someone who directly solves their problem without a bunch of extra words on the side. It boggles the mind, and it's a large part of why I significantly curtailed the time I spend helping people work through their problems. For some reason, a whole lot of people with questions get angry with people who ask things like "what are you actually trying to do here?" or who tell them that their whole approach is wrong, but are perfectly fine with people who go along answering questions politely and wrongly for dozens of messages.

Re:I lost a lot of respect for Wietse Venema

[m50d]

No, it's the hypocrisy and self-righteousness involved in adding eleven lines onto every email you send to tell people you don't want them sending you redundant information in emails.

Re:I lost a lot of respect for Wietse Venema

[Rahga]

There's all sorts of low pressure work to do at Morgan Stanley... For example:

* Schmooze with Paulson and Bernanke.
* Knock the ball into the cup.
* Read twitters from your best buds at Goldman Sachs

It used to be busier before they had to remove a few items from that list like...
* Play with Credit Default Swaps
* Short Sell

Re:I lost a lot of respect for Wietse Venema

[Anonymous Coward]

AC so Victor & Wietse will still talk to me. :-)

As a very long time participant on the postfix-users mailing list, I can agree Victor is arrogant and abrasive; and Wietse is indeed blunt.... however both of these guys have been answering questions like "how do I turn my computer on" for almost 9 years.

Their tireless & selfless devotion to Postfix and it's users, and helping real people with real issues, is absolutely EXTRAORDINARY. Wietse in particular... not many lead developers, and people of otherise high caliber, remain so dedicated for so long to just simply helping people.

I have a huge amount of respect for these guys. And few people understand the vast & complex email ecosystem better.

Re:I lost a lot of respect for Wietse Venema

[gbjbaanb]

Hey I just thought I'd drop you a follow-up post to say, yeah thanks, for your post. cheers now, have a nice day.

Re:I lost a lot of respect for Wietse Venema

[nyctopterus]

Yeah, it's really mind boggling how people dislike being insulted and patronised. Idiots.

Re:I lost a lot of respect for Wietse Venema

[nyctopterus]

You know, even if you're doing something good, if you do it with ill temper and lack of grace, you're still being an arse.

Re:I lost a lot of respect for Wietse Venema

[glwtta]

He's extremely blunt.

Yeah, that's a terrible quality. Much better to be nice to people and not provide useful information.

While many are VERY helpful and detailed, a number are a sentence or two long that, paraphrased, consist of "you're an idiot."

There's a critical piece of data missing: were those posters, in fact, asking idiotic questions?

Oh, and Viktor makes an altogether non-arduous request of people he's already helped, albeit in a brusque manner - I'm burning with outrage.

Phone sex operators are there to be indiscriminately nice to people - it's not a very useful quality on support lists.

Re:I lost a lot of respect for Wietse Venema

[gnick]

Your post reinforces the polite-but-wrong genre of responses.

--
gnick

Disclaimer: A/C replies get ignored.

If this post helped you to understand why you missed the point, the best way to thank me is to not send an "it worked, thanks" response. If you must respond, please put "It worked, thanks" in the "Subject" and post as A/C so I can ignore them without inconvenience.

Re:I lost a lot of respect for Wietse Venema

[Anonymous Coward]

There's a new one now, brush up on your Japanese

Re:I lost a lot of respect for Wietse Venema

[X_Bones]

The problem with this guy browbeating others into not thanking him on the mailing list is that idiots like me, who don't subscribe to postfix-user, have no way to browse the archives a year later and be able to differentiate between questions that have working answers and questions that the asker just gave up on. That means I'm gonna ask him the same question again, and I'd bet a dollar that he'd remember answering it the first time and just tell me to go check the archives instead of wasting his time. And if he gives bad advice with anywhere near the frequency that your parent's post mentions, then I sure as hell would like to know when any of his posts actually lead to a useful and working solution to a problem so I can ignore the others.

It's not my job to make life easier for anyone's email filter. His need for an uncluttered inbox is trumped by the need for the community to archive useful information.

Re:I lost a lot of respect for Wietse Venema

[SuperBanana]

The "flowery" thank-you follow-ups you speak of are actually the norm, not vise-versa. On the Sun Managers list, it was EXPECTED that you post a follow-up to your question, explaining what responses you received, what was correct, what you learned, and who to acknowledge for responding and providing correct solutions. It's the de-facto standard on other lists I'm on, though not to as great a degree. It's a user community, not a help-desk queue.

Victor thinks he's so important that he can demand people not extend the courtesy of saying thank you in exactly the way he wants it, because it wastes his precision brainpower and precious seconds to have to read the message body to see whether to hit the "delete" key. If that's not unbridled arrogance, I don't know what is. I'd be willing to bet he doesn't even do that- I bet he's got a rule that deletes any message with "thank you" in the subject.

The funny thing is, I've seen a couple of Postfix-users posters specifically go out of their way to thank him, not put "thank you" in the subject line, AND cc the list. It's delicious.

Re:I lost a lot of respect for Wietse Venema

[dw604]

Try to get help on EFNET irc - it's the same thing. Volunteers make fun of you when you ask stupid questions, and there's nothing your sorry ass can do about it! :D

Re:I lost a lot of respect for Wietse Venema

[Atario]

If someone has specific and in depth knowledge and spends time helping the less knowledgeable, being an asshole sometimes come with the territory.

Slashdot itself is proof enough of that.

Re:I lost a lot of respect for Wietse Venema

[Free the Cowards]

The problem is that they take direct talk for insults, and direct answers for patronizing. People come onto these lists with the attitude that they're smarter than everyone, yet somehow they still need help. If your question is wrong then it's in everybody's best interests if someone points out that the question is wrong and guides you on how to approach things better, and it's in nobody's best interests to take it at face value and be happy and polite while leading you both into darkness.

Re:I lost a lot of respect for Wietse Venema

[mandelbr0t]

Actually, if someone deals with the unwashed masses regularly, it might be a good idea to learn some manners and/or diplomacy. There's no excuse for being an asshole, not even being ridiculously intelligent and having to deal with real idiots. Everyone has stress in their lives, and it's like geniuses can't be bothered to deal with it gracefully. Quietly ignoring the "it works, thanks" e-mail saves just as much time, without alienating the person with his first response.

Re:I lost a lot of respect for Wietse Venema

[Super_Z]

Personally, I've found both Venema and Duchovni to be both extremely helpful and clueful. IMHO their MTA more than outweights any haughtyness or abrasiveness from their side.

Re:I lost a lot of respect for Wietse Venema

[greg1104]

If you don't get the "it worked" follow-ups to the list, others looking through the list archives trying to resolve the same issue don't know whether a) the proposed solution really worked, or b) the person just gave up or resolved it another way. It's unfortunate that Viktor doesn't understand confirmed answers are therefore useful for reducing his long-term support workload.

Re:I lost a lot of respect for Wietse Venema

[kv9]

...and this asshole thinks he's so fucking important, he lectures you about how to thank him so he can delete your acknowledgment/thank you as quickly as possible.

he seems important enough if you decided to write a post on /. whining about his abrasive attitude (and probably lost some sleep over it -- how dare he, that asshole!). and he seems efficient too.

this sort of behavior is common in IT, no? I figured people got used to it by now. you should read up on nerd tact filters before you act all butthurt about it in public.

ps: fuck you.

Re:I lost a lot of respect for Wietse Venema

[kv9]

if my name was Viktor I'd look smugly down on you commoners aswell.

Re:I lost a lot of respect for Wietse Venema

[Fjan11]

He's extremely blunt.

In his defense: He's also Dutch and male. You could say he is double handicapped. (Most Dutchmen, like me, are not very politically correct. It's a cultural thing that tends to offend those not in the know)

Re:I lost a lot of respect for Wietse Venema

[tzot]

As the Greg House TV character once asked a patient: "Would you prefer a doctor that holds your hand while you die, or one that ignores you while you get better? Although I suppose it would particularly suck to have a doctor that ignores you while you die."

Re:I lost a lot of respect for Wietse Venema

[turbidostato]

"Victor thinks he's so important that he can demand people not extend the courtesy of saying thank you in exactly the way he wants it"

Viktor thinks nothing.

It has been the norm for ages not to waste both time and bandwith with a "just to say thank you" response that benefits to no one (the same goes with the plain "me too" message) and so he says on his signature. True, I think it's quite pretentious (and against another known-for-ages Internet tradition) to have such a long signature specially when just a link to the old (but still maintained) "How to ask questions the smart way" or the RFC 1855 would have been enough but I'm sure he would be glad with an answer *to the list* recouping the question, the failed attempts and the working solution (or lack of it) so it can be easily spoted on the archives or even making its way to the FAQ.

Maybe it's rude but it certainly it's common sense too.

Re:I lost a lot of respect for Wietse Venema

[Anonymous Coward]

why do you see politeness and usefulness as a dichotomy?

Is it so hard to be polite and helpful?
Is it so hard to always be polite?

Re:I lost a lot of respect for Wietse Venema

[aeoo]

"You know, even if you're doing something good, if you do it with ill temper and lack of grace, you're still being an arse."

What do you call someone who is technically incompetent, and yet is polite and socially well respected?

Do you call this type "a nice guy"?

Everyone wants respect these days instead of kindness. There is a difference. Kindness is social deference, often of a formulaic and protocol-like nature. Kindness is when something is aligned with your intention at the deepest level, while not necessarily flowery or concerned with protocol.

I wouldn't being being disrespected by someone who was kind to me. But I wouldn't want to be respected by someone who is not kind to me. But that's just me I guess. Most people want to be the ego sugar, even if there is no substance to it.

Re:I lost a lot of respect for Wietse Venema

[aeoo]

Ha... I made a mistake. I said "Kindness is social deference, often of a formulaic and protocol-like nature." What I meant was "Respect ... " instead of "Kindness ..." I think it is obvious, but thought I'd mention it.

Re:I lost a lot of respect for Wietse Venema

[Ice Station Zebra]

Subject: Re: PATCH: Zero-length MX records and reject_unknown_sender_domain
        From: wietse () porcupine ! org (Wietse Venema)
        Date: 2006-12-28 1:09:35
        Message-ID: 20061228010935.6731DBC0A9 () spike ! porcupine ! org

        Todd A. Green:
        > Wietse Venema wrote:
        > >> Perhaps surprisingly, the MX result syntax check happens only when
        > >> the DNS lookup client actually asks for a result.
        >
        > Thank you Wietse. After patching I'm getting:
        >
        > 550 4.1.8 : Sender address rejected: Domain not found
        >
        > Given we did get a record back, could the error message be "Domain does
        > not handle email" or "Domain has Null MX record" or anything that would
        > let us differentiate null MX records in our logs from those who don't
        > have A/MX records?

        Fsck off.

        Wietse

        > Thanks again for the quick fix,
        > Todd
        >
        >

Re:I lost a lot of respect for Wietse Venema

[Anonymous Coward]

oh, be fair: openbsd is much more than an MTA

Re:I lost a lot of respect for Wietse Venema

[Random Walk]

As someone else has already pointed out: an archived email discussion is worthless without an indication whether the proposed solution actually worked. "Thank you, it works" is important.. not for showing your politeness, but for providing a crucial piece of information.

Er...Spellcheck much?

[FishAdmin]

Not that Spelling Nazi's are prevalent here or anything, but shouldn't that title read "Postfix's Creator" and not "Postifix's Creator"?

Re:Er...Spellcheck much?

[halcyon1234]

Little typos like that don't matter. I mean, things work just fine whenever I sign into BankFoAmercia.com. Okay, sometimes the initially login fails, and I have to login at BankOfAmerica.com again, but after that things are fine.

Strange, though, I never can seem to make my paychecks last more than a day or two. Hrm.

Re:Er...Spellcheck much?

[Anonymous Coward]

A grammar Nazi would point out that 'Spellcheck much?' makes no grammatical sense. WTF are you trying to say there?

Re:Er...Spellcheck much?

[Anonymous Coward]

The god damn apostrophe goes after the s when it shows ownership god damn it !
For fuck sake will someone please fix the motherfucking spellcheckers that are getting this shit wrong ?

Re:Er...Spellcheck much?

[Anonymous Coward]

Sorry, but most of us here write in accepted American English here, and you're absolutely wrong.

The apostrophe goes before the 's' if the owner is singular. It goes after the 's' if the owners are plural.

Irregular plural nouns take an apostrophe and then an 's'.

An archaic usage is that if the owner's name ends in 's' or an 's' sound from an 'x', then only an apostrophe is added. This is still often acceptable but its usage is becoming increasingly rare.

What is the poster's name?
What are the posters' names?
The bosses' opinions often matter, even when they are clueless.
This is the child's bicycle.
That is the children's playground.
Don't get caught using the executives' names online or you'll get fired.
What is the best book about Ross's expedition?
What is the best book about Ross' expedition?

Re:Er...Spellcheck much?

[Atario]

I see why your paychecks never last long enough: you're logging on to BankOfAmerica.com. It's an obvious scam site, what with the random, hefty withdrawals for "fees" and whatnot.

Re:Er...Spellcheck much?

[kv9]

Not that Spelling Nazi's are prevalent here or anything, but...

Spelling Nazi's what?

Re:Er...Spellcheck much?

[Scaba]

Yes, and it should also be "Nazis," not "Nazi's."

Re:Er...Spellcheck much?

[FishAdmin]

Now, now; let's not confuse Grammar Nazis with Spelling Nazis! These are not the errors you're looking for; move along.

Hey, CmdrThicko is it ....

[DrSkwid]

Postifix, Postifx or Postfix? Make your mind up !

Just use gmail

[msevior]

I dunno what google do, but I get about 1 spam per 3 days on an account that receives about 50 messages a day.

My work MS exchange address with the latest anti-spam stuff gets about 10 spams per day with the same legitimate email rate.

Without anti-spam I get about 200 spams a day at work.

Re:Just use gmail

[rolfc]

Google have experienced mail-administrators, while your work has someone who knows how to point and click?

Re:Just use gmail

[rolfc]

Right ;) Flamebait!

Re:Just use gmail

[gmuslera]

Some of the techniques recommended in the article work for all servers. While i don't share the pull model (not all the world have plenty of outgoing bandwidth to make this work, at least at peak hours), the other suggestions are valid and widely used (including the many eyeballs one, the kind you do when you press "report spam" in gmail).

But "just using gmail" is a good simplification of it... just let someone else worry about your problem.

Re:Just use gmail

[quenda]

That works well, until you get "sorry your account has been disabled" from Google. Then run in circles with only auto-responders to your increasingly desperate emails begging for help.

Re:Just use gmail

[theCoder]

The pull model really isn't a good idea, because that is what spammers are already trying to get people to do. They want you to open the email and click the link. A pull model just makes that automatic. Not to mention all the marketing people (pseudo-spammers) that would just love to know which of their recipients actually look at their emails, and how long they look at them, etc. I already get mailings (alumni stuff, etc) that are just links to a web page where I can read the actual letter.

And of course, "just use gmail" isn't really a solution. It only works until someone figures out how to get through gmail's filters, or Google really sells out and starts allowing select "partners" to advertise to members directly. Though there is some irony in the idea that you can avoid email advertising by using a system that has ads in the email viewer. I'm not saying anything bad about Google or gmail, just pointing out the irony :)

Re:Just use gmail

[harp2812]

I dunno what google do, but I get about 1 spam per 3 days on an account that receives about 50 messages a day.

Postini, IIRC:
http://www.postini.com/goog/google.php

Re:Just use gmail

[Anonymous Coward]

Use GMail. Millions of spammers can't be worng.

Re:Just use gmail

[Anonymous Coward]

Blocking Google ads is much easier than blocking spam.

Satan writes?

[Anonymous Coward]

I didn't know Satan had an interest in the goings on of Postfix.

Not only that.

[khasim]

From TFA:

In my personal opinion, the reliability of email reached its maximum near 1998; it has gone down ever since as the result of increasingly aggressive anti-spam/virus measures. This observation has led me to conclude that the spammers aren't destroying the email infrastructure, it's the well-meaning people with their countermeasures.

I use Exim4 as a pre-processor for a GroupWise system.

This allows me to reject messages during the SMTP connection (no receive and then bounce back) and I have customized the rejection messages to include my phone number. As long as YOUR email admin handles error messages in any sane way, you'll get a phone number to call and talk to the guy who set up the system that rejected your email. I get a call about every other month now.

The real problem is not "aggressive anti-spam/virus measures".

It is that 80%+ of the inbound connections are spam-related. So just about ANY action taken will reduce the amount of spam. But the email admins still need to continually evaluate their processes.

Re:Not only that.

[theshowmecanuck]

I think he is talking about reliability in that every email sent gets to its destination. Right now, email can be blocked as spam. It doesn't matter whether you do the blocking at the SMTP level or not, it is still being blocked including some legitimate emails. If legitimate email is being blocked for any reason, it means the service is not reliable. Your caveat "As long as YOUR email admin handles error messages in any sane way" doesn't solve anything since the person sending the email is usually not responsible for how their email server is configured. Meaning that for them, the service is either reliable or it isn't. This ultimately means that if someone's legitimate email gets blocked by you/your server for some erroneous reason, that your email server is not reliable, and less so than in 1998. The article is saying our current anti spam counter measures are what is making email less reliable.

Re:Not only that.

[nabsltd]

It doesn't matter whether you do the blocking at the SMTP level or not, it is still being blocked including some legitimate emails. If legitimate email is being blocked for any reason, it means the service is not reliable.

You misunderstand the term "reliable" as it applies to networking.

TCP is called "reliable" and is said to "guarantee delivery". This is not true. TCP merely guarantees that if it was not delivered, the client program knows it was not delivered. SMTP behaves in exactly the same way.

So, the key is to block at the SMTP level, which guarantees that the real sending server gets an error message. If that sending server is legitimate (e.g., a GMail server), then the error message is passed along to the original sender, and they can do something about it. This is as "reliable" as e-mail ever can be, given the nature of the Internet.

So, what TFA was complaining about was anti-spam measures that break the SMTP "reliable" contract of either actually delivering the e-mail or else returning an error message to the sender. MS Exchange and qmail come to mind as two programs that make it very hard to notify the sender of an error without creating blowback spam.

Re:Not only that.

[calmofthestorm]

*ring*

Hello?

V1agr4 4 s4l3 confadintial cheep! U want big l0v3 r0d/br34s7s?

our w3bsi7e us3s secur3 256 bi7 ord3r proc3ss1ng sh0p w17h confidance!

Re:Not only that.

[turbidostato]

"As long as YOUR email admin handles error messages in any sane way, you'll get a phone number to call and talk to the guy who set up the system that rejected your email."

Well, I hope you will accept my reverted-payment phone call from Spain... in Spanish then.

Re:Not only that.

[turbidostato]

"You misunderstand the term "reliable" as it applies to networking."

You misunderstand in that you think we are talking about networking.

Not.

We are talking about the electronic mail service. And Venema is quite rigth about saying that today the electronic mail service is less reliable than in 1998.

"So, the key is to block at the SMTP level, which guarantees that the real sending server gets an error message."

So what?
1) My [Big ISP] is not going to tell me some e-mail has been rejected 550 because [whatever]. It might even put it on /dev/null.
2) Even if my [Big ISP] returns me its SMTP chat regarding something about inverse resolution jabba-dabba I won't understand it unless I'm considerably knowledgeable about the SMTP. Good luck trying to get a working solution from the [Big ISP] helpdesk service. ...Oh, but it happens *I* am knowledgeable about the SMTP. So indeed I can manage my own server.
3) But then, I "just" have to make an international phone call in the middle of the night to try to talk to the f* postmaster that decided that I'm a spammer because my IP address belonged some years ago to my provider's residential DHCP pool just to learn that...
3.1) Is their [another Big ISP] policy not to accept mail from my IP and there's nothing they can do or...
3.2) They are using [spamhouse du jour] and they are very glad using it since it in fact reduces there spam volume, the hell with whatever legit mail they are thrasing out unless hard pressed by the high C*Os, which is not the case.

The fact is that if you were talking about snail mail instead of email you would say "the heck with your damn lame excuses regarding international post agencies, delayed deliveries due to revolution at Toofarynstan or internal comdex regurgitation over the ultraviolet scam filter in Dunkerk: I put the letter on the postal box and it reliably ends up in the hands of its intended receptor or it doesn't". That's the only working meaning for reliability.

Re:Not only that.

[Random Walk]

I think your comparison with the post office just illustrates what's wrong with the way you look at the issue. It's perfectly possible that the recipient refuses delivery (in particular if it's certified mail or a packet, and he'd need to sign). It's also possible that the recipient does not live there anymore (moved, deceased).

The point is, you cant reasonable expect your letter to be delivered. You only can expect that the post office will inform you of problems. That's exactly the same with snail mail as with email. Reliability means 'tell me if there was a failure'.

If your [Big ISP] fails at this, complain with them, or take them to court for willful suppression of communication (yeah.. in civilized countries there are laws which forbid that).

Re:Not only that.

[turbidostato]

"It's perfectly possible that the recipient refuses delivery"

Yes, of course.

But the problem is that:
1) The recipient is *NOT* recieving the delivery to begin with, but someone else. "Hey, Mike, did you read my e-mail? No, I didn't recieve an email from you".
2) The recipient is *NOT* refusing my delivery, someone else is doing it on his behalf.

Again, were it be snail mail, it is not as if the recipient rejects the delivery but as if the chief postal officer from Wellington, Alabama said:
-So this for John Doe, Wellington, Alabama? But it comes from Dunkerk, North Carolina, *I* won't accept mail from Dunkerk, North Carolina.
-But John Doe is expecting this letter; won't you pass it to him so *he* is the one that accepts or rejects it?
-No, I won't. You may phone him instead.

"The point is, you cant reasonable expect your letter to be delivered."

I think this resumes your whole point. And I'd better won't tell what I think about the sensibleness of your point.

Re:Not only that.

[Random Walk]

And I'd better won't tell what I think about the sensibleness of your point.

It's perfectly sensible.. because it's realistic, rather than dreamland. Shit happens. All the time.

Also, in any civilized region of this planet, it is the recipient of your email who is rejecting it. Whether he sorts the mail personally, or has given that job to someone else is his own decision.. if the chief postal officer refuses the mail, the recipient has ordered him to take care of sorting his mail. Tough luck.

And yes, I am the chief postal officer over here. And I happily redirect spam to /dev/null if - and only if - a user tells me to do that. It's his decision and his risk. You may phone him instead.

If programming was a million times more difficult

[Anonymous Coward]

We wouldn't have fewer people interested in it, we would just have a million times more bugs or one millionth the number of programs available.

Just because it is more difficult doesn't mean the people attempting it are going to do a better job at it. Flying men into outer space is difficult, just because flying men to Jupiter is a million times more difficult doesn't mean the approach we create will be more successful at it.

If anything, programming needs to be easier, so more people would do it then we could have more solutions to choose from. A parallel brute force approach with selection can produce better solutions for everybody.

If you enjoyed this post...

[Anonymous Coward]

...why not send your thanks to Victor.Duchovni@MorganStanley.com and perhaps tell him a bit about your day.

My Solution to Spam?

[Archangel Michael]

Spam Assassin.

No, not the program of the same name, an actual assassin that kills spammers, CEOs of companies that use SPAM etc.

And if he has some extra time, assassinate some of the Wall Street Pirates responsible for the mess we're in.

I suggest 1 Trillion Dollars as a bounty, since the Government is handing money out like candy.

Re:My Solution to Spam?

[Anonymous Coward]

Excellent; now all I need to do is to find your email address so I can joe-job a few people I don't like and save myself the trouble and risk....

Still waiting for a "one solution" email product

[bogaboga]

I wish the Linux platform had a "one solution" product for these services. The services pegged to Postfix that I am talking about include: -

Mailman/Mailing Lists, Autoresponders, Greylisting, POP3/IMAP, unlimited domains, Sender Policy Framework (SPF), per-user filtering, per-domain policy rules, ClamAV virus, Filtering and Spamassassin Spam Filtering.

Getting these to flawlessly get set-up from scratch is a feat in itself. Why don't we have such a product? I am no coder so I cannot do much except reporting problems.

I imagine a single script a user can run then have all those services running within parameters to be supplied. Linux folks are capable of a lot more so this should not be that difficult.

Just forgot! Add a calendering application [of your choice] to the above line up.

My solution

[Anonymous Coward]

My solution: Nuke China.

Re:My solution

[Chrisq]

I think to eliminate spam you'd have to nuke the entire world.

Re:My solution

[Anonymous Coward]

It's the only way to be sure.

Re:My solution

[UnknowingFool]

Just make sure you do it from orbit. It's the only way to be sure or so I've been told.

Re:My solution

[Anonymous Coward]

The internet agrees.

Is it me ...

[rainhill]

...or title should spell Postfix?

Greylisting

[CaptSaltyJack]

greylistd is an option, though I haven't tested it thoroughly. For those not familiar with it, greylistd works alongside your MTA and rejects ALL incoming e-mails on their first attempt. On the second attempt after some time has passed*, it accepts the email and whitelists that IP/sender for a user-specified amount of time (defaults to 60 days I believe?).

The idea is that spambots do not attempt to redeliver rejected emails, whereas regular "legit" mail servers do. When an email is greylisted, the MTA sends back a special response similar to a rejection, though it does indicate that it's a greylist response. I can see that spambots will eventually get around this by attempting redelivery, I would think. So I don't see greylisting as a long-term solution, but I'd welcome any comments on this.

By the way, if anyone knows a sure-fire way to get spam mail sent to a particular email address, please reply to this comment and let me know. I need a real-world test.

*I noticed most servers attempt to send again within 15-20 minutes; that is also rejected as I suppose the greylist server thinks that's too soon...?

Re:Greylisting

[rfunk]

There are lots of different greylist implementations, with different configuration abilities and defaults.

It should be possible to configure the greylist server to have only a five-minute (or even one-minute) waiting period, which is much more reasonable than 15 minutes.

I have noticed that greylisting isn't quite as effective as it once was, because lots of spammers are actually using real queueing mail senders now. But it still has a lot of effectiveness.

See http://greylisting.org/ for more details on the concept. I personally use postgrey.

Re:Greylisting

[zolltron]

The idea is that spambots do not attempt to redeliver rejected emails, whereas regular "legit" mail servers do.

One problem with this system is that not all "legit" servers do. My old university adopted grey listing, and once that happened nothing from another university ever arrived. This is pretty bad when you have two research university actively engaged in many cooperative projects and all of a sudden they can't talk to one another.

It eventually got sorted out, but after a few angry phone calls from friends, colleagues, etc.

Re:Greylisting

[The Moof]

Spammers are getting better with the greylisting. They resubmit after 'x' minutes of a temporary rejection.

However, clients seem to think e-mail is instantaneous, which causes some support calls with greylisting. I get a few complaints now and them when an e-mail message takes 10-15 minutes to come through. Usually by the same people who e-mail you soemthing, then call you right away to tell you about it. While I can educate as many people as I can about the reliability of e-mail (not guarenteed to go through, sometimes takes a while), it goes right in one ear and out the other with non-techy folks.

Not to mention greylisting depends on the oother admins to do things correctly. Sometimes the message delays for an hour. One time, I even saw the other MTA sit on a greylisted message for 7 days (as in 1 week).

I had a similar situation with SPF when the other mail sever was misconfigured and we were rejecting the mail (as instructed by their server). We did inform the other mail admin, but to date, it's still misconfigured. The end result of it I was I was instructed to remove SPF from our mail servers since it was "blocking business e-mail."

Re:Greylisting

[nabsltd]

I have noticed that greylisting isn't quite as effective as it once was, because lots of spammers are actually using real queueing mail senders now.

I haven't noticed a lot of reduction in effectiveness in my homebrew greylisting implementation, but I also temporarily blacklist IPs that send spam or viruses using a exponential scoring system (e.g., send one spam and you get blocked for an hour, send 20 and you get blocked for 16 days, while 100 gets you blocked for a year and half).

Re:Greylisting

[mr_mischief]

Because bouncing spam to the supposed sender who didn't really send it is the optimal solution?

Re:Greylisting

[mhazen]

Then you should fire the ass of whichever mail administrator decided that configuring a mail server to not follow the RFC's was a good idea.

This is one very good reason mail is such a problem child: there's a high number of clueless admins who assume just because they don't understand how it works, breaking the rules is fair game.

Re:Greylisting

[WuphonsReach]

I had a similar situation with SPF when the other mail sever was misconfigured and we were rejecting the mail (as instructed by their server). We did inform the other mail admin, but to date, it's still misconfigured. The end result of it I was I was instructed to remove SPF from our mail servers since it was "blocking business e-mail."

Before implementing any sort of "block" at the mail server level, always make sure that you have a way to whitelist senders to get around the block.

As exceptions occur, you can then quickly whitelist senders without having to strip out the entire blocking action.

Re:Greylisting

[The Moof]

There was whitelisting in place, but the problem was they didn't even want the first message to be blocked. So, they had me remove the SPF all together to stop the first message blocking. They were afraid the message would be blocked and discarded without anyone calling to say "My e-mail bounced."

Re:Greylisting

[Sancho]

"Rules"

I generally think that adhering to the RFCs is a good idea. That said, RFCs are not set in stone, nor is there any "rule" that says that they must be followed. Greylisting uses side-effects of RFC adherence to function, but the general rule of accepting things as liberally as possible should still apply. With greylisting, it doesn't.

Also, keep in mind that the older SMTP RFC didn't specify that clients must retry when temporary errors are returned. These servers may be adhering to the older RFC. Who's to say that they must adhere to the newer one?

Re:Greylisting

[Chapter80]

By the way, if anyone knows a sure-fire way to get spam mail sent to a particular email address, please reply to this comment and let me know. I need a real-world test.

http://spamyourenemies.com/

haven't tested it myself, so I guess I don't know if it's surefire.

If my response solves your problem, the best way to thank me is to not send an "it worked, thanks" follow-up. If you must respond, please put "It worked, thanks" in the "Subject" so I can delete these quickly.

/sarcasm

Re:Greylisting

[kayditty]

Uh, if you use a 4xx reject code, you aren't bouncing anything.

Re:Obligatory checklist

[agbinfo]

Well how about this solution: http://slashdot.org/~agbinfo/journal/208701

wietse.cx

[Anonymous Coward]

the giver

SnailMail

[cryptodan]

You can throw your mail in the trash or send it back.

Re:SnailMail

[AndrewNeo]

Or put some change in the return envelope, and when they call you stating they got some amount of change, demand that they send it back to you. (I'd link to the quote but bash.org is down)

Re:Still waiting for a "one solution" email produc

[hey]

Because that's not the Linux/Unix way.
And because everything eventually touches mail so that means everything would have to be integrated into "one solution".

Do people really still have problems with spam?!?

[JoeDuncan]

Seriously, do they?

I find it hard to believe that people still have serious problems with spam.

There is a perfectly workable spam solution that my grandmother wouldn't have a problem implementing.

I have a GMail account to which all my other e-mail accounts are forwarded. I access this account through IMAP with Thunderbird. I use Thunderbird's built in learning spam filter.

When I first signed up for GMail/started using Tunderbird, I had almost more spam than e-mails and I get at least 20-30 real e-mails a day. So I just flagged all the spam I got as spam in both Thunderbird and GMail.

Now, I get maybe, and I stress *maybe*, a single spam message that shows up in my Thunderbird inbox per week.

I go through my Thunderbird Junk folder and my GMail spam folder about once a month to look for false positives, but they are few and far between, less than one a month.

Seeing as how this solution is simple, automatic, easy and pretty much ubiquitous (who doesn't have access to Thunderbird and GMail?) I don't see why anyone needs to suffer from spam at all anymore. I sure don't.

Other than ideological reasons (i.e. problems with either Google or Mozilla) I see no reason not to use this solution.

Re:Still waiting for a "one solution" email produc

[tomz16]

Getting these to flawlessly get set-up from scratch is a feat in itself. Why don't we have such a product? I am no coder so I cannot do much except reporting problems.

I imagine a single script a user can run then have all those services running within parameters to be supplied. Linux folks are capable of a lot more so this should not be that difficult.

... because mail IS complicated, and each of these products has its own quirks and gotchas.

Someone who cannot be bothered to read the teeny fraction of relevant documentation necessary to properly set up this software probably has no business administering it (especially on a production network). Since a poorly configured mail server really has the potential to piss thousands of people off around the planet, I'm actually content with the current state of affairs...

P.S. you are looking for a product called Microsoft Exchange. It has nice big buttons you can point and click on. Luckily the costs involved and the presence of an official certification program serve as an effective barrier to entry for most amateur admins.

Re:Obligatory checklist

[Anonymous Coward]

You can't have the system automatically reply to all spam like that. Go ahead and try it to see what happens. Your mail server will be buried with undeliverable forged headers *and* you'll be sending out envelope-reversed spam.

To Sum up TFA

[Anonymous Coward]

He says currently most of the cost of email is on the recipients. He thinks the structure of email should change from push to pull so the cost is more on the sender side - e.g. instead of checking your or your ISP's mail server for messages, you should poll the sender's server... You would know to poll these servers based still on small pushed email - though each with digital certificates and signatures you trust, notifying you of awaiting mail on their system. Sounds interesting... He believes that integrating Spam/AV into the MTA is not as good as separating this - and compares it with an analogy to the benefits of human specialization... sounds ok.

Also talks about Postfix internals which was kind of interesting.

Not at all

[bogaboga]

...And because everything eventually touches mail so that means everything would have to be integrated into "one solution"...

Who said that? OK...why not provide such a product for those that want it? Why? Not every body does things on Unix/Linux "the Unix/Linux way." Do you still install your Linux software from source? Gentoo and Slackware had one of their feet on this route. Have you heard of Gobo Linux? They do their stuff another way...and it's exciting.

Ok, let me ask: What would be the problem with a product being put together like I suggested, with this product working exactly as advertised?

Please provide me an answer...What exactly is "the Linux way?"

I have a feeling that it's folks like you that have made Linux not that popular among the Joe Six Packs of today, because you insist on Linux doing things "the Linux way", even when this way does not produce results that one can be proud of in terms of penetration and wide appeal.

It's sad indeed.

Re:Not at all

[pohl]

Who said that? OK...why not provide such a product for those that want it? Why?

The sad reality is that "the unix/linux way" is entirely up to those who contribute code. On the bright side, anyone who wants to redefine "the way" has a clear course of action.

Re:Not at all

[mr_mischief]

Who said people who want to keep the culture and methods of the culture are the same people who keep pushing for universal acceptance?

Keep using Windows, and I'll be working while you're rebooting. That's not a problem for me. It's a problem for you.

Re:Not at all

[JK_the_Slacker]

The Unix Way? One app, one function, zero fuss. When you try to integrate everything into one app, you introduce a complexity that tends to break things. I for one hate all-in-one apps of any kind. I find a much lower ratio of bugs when I have one app to do one thing. This is true in the Windows as well as the Linux world.

Do I think this is keeping Linux from being popular? Nope. I think it's this philosophy thhat makes Linux as popular as it is... THINGS WORK.

Re:Not at all

[Handover Phist]

OK you two settle down, there's no need to go off all half cocked. The Linux Way is about simplicity. Creating a program that does one thing very well and in such a way that both input and output can be easily piped from or to another program. Using this Way it is possible to create a solution as proposed by Bogaboga (sorry if I inadvertently capitalized when I shouldn't). Fact is, it's a pretty damned big project, and nobody is currently working on it.

Thanks for volunteering though!
 
/Standard Linux asshole DIY response ;)

Re:Not at all

[AnyoneEB]

As a supporter of the Unix philosophy of "one app, one function", I think the proper solution here would be an app that does basic setup of the configuration files. Maybe it would come with a few templates and ask some questions with explanations to fill in the blanks. That way it can be separate and the people who know what they are doing can just not install it, but simple use cases can be covered with limited knowledge.

Piece on what makes Postfix & Wrapper so good

[Anonymous Coward]

There's a nice little article all about what makes some software secure, where other application disintegrate under exposure to the "internet elements" over at Three Sixty Information Security. It features TCP Wrappers, Postfix, and several others, and asks the question "what makes this software so uncommonly good?".

AG

Re:Still waiting for a "one solution" email produc

[bogaboga]

At my company, Microsoft Exchange is quite stable. It processes about 1100 emails per hour. Our admin says that it is admins who do not know what they are doing that give Exchange a bad name. I am sure this applies to the Linux crowd too.

Re:Do people really still have problems with spam?

[Anonymous Coward]

Cool!

Re:Still waiting for a "one solution" email produc

[Hatta]

I believe the author of Sendmail once said "Sendmail is complex because the world is complex." Now, it probably doesn't have to be as complex as Sendmail, but there will never be a one size fits all solution.

Re:Still waiting for a "one solution" email produc

[gujo-odori]

You don't need to be a coder to set those things up, but you do need some level of competence in general systems administration and mail administration in particular, and there's not really any way around that. Sure, some Linux distros make setting those things up relatively easy (Debian and its derivatives are perhaps the best for that), but you still need some idea of what you're doing?

Why? Because email is really complex. So complex that the "Simple" in SMTP could be taken as some kind of inside joke, although it was actually relatively simple back when SMTP was born. Email routing and filtering is many respects the most complex thing done on the Internet. A single script to get all that stuff set up and working would be quite complex and almost certain to not work for everyone. Moreover, getting it set up wouldn't remove the need for ongoing competent administration.

I work for an email security company, and one of the reasons there is so much money in this is because it takes a lot of specialization to be really good at it, and for many businesses it makes the most sense to outsource it to specialists, even when they have top-notch mail admins (many of our customers are Global 2000 companies, and they have really good sysadmins in those kinds of places), email security is something we can do better than they can in most cases, and always more economically.

I don't see how the pull model helps

[kwerle]

I've seen a few folks advocate the pull model for email and say that the burden then rests more on the sender than the receiver. I just don't see it.

I'm a spammer sending as much email to as many folks as possible. What would I rather do: send the message itself (let's say it's 2K), or send tiny receipts for a message (let's say 1/2K or less)? Then when the receivers pull their message I send the 2K message. And if I start to get flooded I dynamically reduce the size to 1K or even less? And if I'm slow, I increase the size to 5K or more (pretty pictures, etc).

I don't have to store the content - I can just generate it dynamically. And I can even send a bunch of receipts and change the spam content over time depending on who is paying me and how effective some spam solution is at any given time.

So, seriously, how does the pull method help? It seems to me that it's worse than push.

Re:I don't see how the pull model helps

[mr_mischief]

Actually, by generating it dynamically you're just taxing CPU cycles instead of disk storage. Disks are cheap.

What you could do, though, is program a mail server that lets every user pull the same single message no matter whether they were even sent the notification it was waiting. Then, just spam the notifications out.

You've effectively just saved the spammer all the bandwidth for customers who don't click on the email.

Also, the pull model means I can spam out notifications for people to check your email pull server and bury it under the unexpected load. Certificates for servers help limit this, but it's still a possibility.

Re:I don't see how the pull model helps

[Anonymous Coward]

Not only that but now the spammer accurately knows which e-mail addresses work and which mails people are looking at.

It's a marketer's wet dream and a bad idea.

Re:I don't see how the pull model helps

[seek31337]

Not if you add a domain lookup into the process. So I get a message from 'xyz.com'. Much like SPF, I look up a specific TXT record on xyz.com for where I pull my message from. Deny URLS that are IPs on the server level. Make the MTA handle delivery inside your network or to normal mailboxes, and the fetching doesn't need to happen on the client.

Vola, you now have have (1) the cost of spammers registering a domain. (2) the cost of setting up servers. (3) the inability for people to send email from arbitraty IPs. (4) RBL lists are now merely a list of domains to reject, instead of a whole slew of IPs and domains. Once a domain is on an RBL list, they can no longer just change IPs and go at it, they need to pony up for a new domain. Botnets, as they are now, would be nearly impossible to get working.

Re:I don't see how the pull model helps

[Just Some Guy]

I've seen a few folks advocate the pull model for email and say that the burden then rests more on the sender than the receiver. I just don't see it.

I have a friend who sends out monthly newsletters to people who've signed up through his site to read them. These are biggish PDFs, and with his bandwidth it takes him several hours to send them all out. This is no problem; he starts the batch late at night and everyone has their newsletter by the next morning.

Contrast with the pull model. My friend sends out the same number of notification messages. The next morning at 8:00AM Eastern time when everyone is checking their email, he has 3,000 people trying to download their newsletter simultaneously. The server catches on fire. Repeat at 8:00AM Chicago and 8:00AM Los Angeles.

That's why the burden is so much higher than with regular email. Instead of planning for averages, you have to plan for peaks.

Re:I don't see how the pull model helps

[Sancho]

I would argue that such a newsletter doesn't belong on the e-mail medium in the first place. The newsletter should be hosted on the web somewhere where people can download it.

Re:I don't see how the pull model helps

[Anonymous Coward]

I would call the servers catching on fire a burden on the sender.

Re:I don't see how the pull model helps

[kwerle]

Just as the current push system has a message from the receiver:
"Try again in X minutes - I'm busy",
I imagine the sender in the pull system will have the same message. Problem solved.

And as I outlined before, you could have N messages for the same receipt: light traffic, medium traffic, heavy traffic.

Re:I don't see how the pull model helps

[Just Some Guy]

I would argue that such a newsletter doesn't belong on the e-mail medium in the first place. The newsletter should be hosted on the web somewhere where people can download it.

That makes sense to you and me, but he has quite a few tens of thousands of customers who want their newsletter. The business in question is a successful brick-and-mortar and the mailer is filled with reviews and articles, more like a magazine than a page (or set of pages). The point is that he has a demand to fill and this system works for him.

Re:I don't see how the pull model helps

[Just Some Guy]

Just as the current push system has a message from the receiver: "Try again in X minutes - I'm busy", I imagine the sender in the pull system will have the same message.

Customers are notoriously impatient. Do you like waiting for a web page to load, or sitting on hold to talk to an operator? Imagine how much people would dislike seeing the current system replaced with one where they have to queue up. Imagine how unwilling companies will be to do this to their customers.

Re:I don't see how the pull model helps

[mr_mischief]

If you think the outrageous expense of $200 for a cheap desktop box running Linux, $10 a year for a domain, and $50 a month for internet access is going to break the spammers, you seriously underestimate the profitability.

If I'm a spammer, I can send you and a fifty million other people a small notice that there's mail waiting for you on the domain I actually have registered. You won't fetch that mail, and neither will anyone else with a clue. The other 47 million people will.

Re:I don't see how the pull model helps

[kayditty]

That doesn't necessarily work if you rely on a cookie inserted by the sender, unless SMTP is changed. I assume that would go in the DATA section under such a system. But the envelope exchange happens before that. The envelope originating address would have to be saved and sent to a checking function after parsing the message, which would be tedious and limiting for infrastractural purposes. You can't reject the notification outright, but you might be able to prevent pulling the real message, though that would inhibit your ability to do processing or forwarding to other processes.

Another hard sell on the pullp method, though I think it mostly makes sense myself, is that it would be quite difficult to implement restrictions on message size and all sorts of other things when you don't control the session. So, yet again, SMTP would have to be wildly extended for that purpose, and that's something that obviously isn't going to happen. Of course, the whole pull system in general is a new idea which would require an unprecedented amount of cooperation to come into existence.

Re:I don't see how the pull model helps

[kayditty]

I also forgot to mention that a lot of people seem to interpret the pull system as a POP-level thing. In that case, this wouldn't work at all without additional headers and MUA software.

Re:I don't see how the pull model helps

[totally bogus dude]

Well, the main benefit is in centralising the locations of spam. As it is now, a lot comes from zombie PCs with dynamic IPs. Send and forget works in their favour, because it doesn't matter if the machine moves around or even if it gets cleaned if it's already sent your message out.

By making it a requirement that they keep their system online in order for people to receive the spam message proper, theoretically you degrade their mobility a bit. This makes community blacklists more useful, because once you've spammed a bunch of people using a particular domain name everyone will start blocking or discarding messages that refer to that domain. So now that $10/year for a domain name becomes $10 per X thousand messages. It also may make it easier to track down spammers.

In reality, I don't think it'd make much difference. It's no magic bullet, and there's pretty obvious "solutions" to the problems this would cause spammers.

Re:I don't see how the pull model helps

[mr_mischief]

Centralizing the system that holds the actual messages would be a benefit. You'd have to do it by DNS name though, because one spammer can't be allowed to shut down a range of IPs as a spam source. That's part of our issue with fighting spam now, and any new solution needs to address it.

Re:I don't see how the pull model helps

[kwerle]

I never imagined the MUA would see that message. I figured the MTA would deal with all that, and the MUA would continue talking to your server. Then it's up to the server to pull the messages. Maybe that's where I'm not getting it.

Of course that whole system would only work for directly connected machines (24x7).

I never considered that the pull system would be web-esque. Maybe that's where my disconnect is. But I don't think so...

Re:I don't see how the pull model helps

[seek31337]

$10/year * changing it ever day
$50/month * changing ISPs every day for violating abuse pocicies
$200 for a cheap linux box will not be enough power to handle your load.

Tanke your $200 linux box and toss 3000 requests/second at it on your $50/mo internet connection. As a test... see how well it works for ya'

Re:I don't see how the pull model helps

[seek31337]

Absolutely. As I said, if it's a TXT entry, ala SPF, for a given domain, you can shut it down at the domain/sub-domain level. RBL's become lists of domain names you should ignore. If the protocol is something like:

domain xxy.com for you@place.com id 3133768

place.com looks up the TXT record for xxy.com
place.com requests message from the server specified in xxy.com (e.g. http://smesgs.xxy.com/?to=you@place.com&id=3133768)
place.com validates message as an actual email, handles errors, queueing
place.com delivers message locally.

You could also have multiple IDs, etc. Could be POST, could be a custom protocol.

disallow IP based deliveries. Make it domain only.

Ridiculous

[Anonymous Coward]

In my personal opinion, the reliability of email reached its maximum near 1998; it has gone down ever since as the result of increasingly aggressive anti-spam/virus measures. This observation has led me to conclude that the spammers aren't destroying the email infrastructure, it's the well-meaning people with their countermeasures.

I recently abandoned a domain I've had for years because I was getting a spam email every few seconds. I could go to bed with an empty inbox, and wake up to have dozens of spam emails, even after filtering.

Without spam filtering, email would be useless to me, I would give up on it entirely. Spam filtering is not destroying the email infrastructure, it is an absolute necessity, forced upon us by spammers, that unfortunately causes a modest amount of collateral damage. To point at that collateral damage and say that it must be the filtering causing the problems is... well... totally out of touch.

And for that matter, why is the Postfix website in such shambles? Go find me the changelog. Is it in the announcements section? No. Is it in the umpteen unorganised links in the documentation section? No. You have to select "Download", pick a mirror, and then it gives you the changelog. Why is it so impossible to find anything on the Postfix website?

Re:Obligatory checklist

[agbinfo]

You can't have the system automatically reply to all spam like that. Go ahead and try it to see what happens. Your mail server will be buried with undeliverable forged headers

If you get enough SPAM that replying to them is that big a problem, shouldn't that be a bigger incentive to reducing the SPAM issue once and for all. Undeliverable is not an issue since it doesn't end up in someone's email box; Just delete it. A mechanism to stop circular replies would be needed though.

*and* you'll be sending out envelope-reversed spam.

The envelope-reversed spam may be an issue. But this is the type of issue that has other technical solutions. An authentication scheme could easily be implemented to reduce this. For example, email coming from example.com could be signed with a hash. Simply connecting to the an example.com SMTP server could provide enough information to validate the hash. On the other hand, validation replies would be relatively easy to detect so mail server that are receiving those could try to match the recipients with sent email addresses. This would make it possible to simply delete inappropriate requests.

I admit I haven't thought of every possible issue but I think it could be a good starting point.

The real problem

[geekmansworld]

This man is a God-Damned genius:

"...The technical arms race will continue unless politicians and law enforcement join the battle with effective measures that work across national borders.

This observation has led me to conclude that the spammers aren't destroying the email infrastructure, it's the well-meaning people with their countermeasures."

Yes! Yes! Yes!

As a system administrator, I can't tell you how many times a failure to receive a customer's e-mail was due to a poorly-configured junk scanner on the customer's network.

And fighting spam is indeed a two-pronged approach. Sysadmins AND politicians need to be proactive about fighting spam. Spam is an issue that affects communications, especially business communications, with unacceptable severity. It's time for politicians to do their fair share.

Re:The real problem

[burning-toast]

As a system administrator, I can't tell you how many times a failure to receive a customer's e-mail was due to a poorly-configured mail server on the sender's network.

Fixed that for you. I think the number of mis-configured mail servers and DNS records far exceeds the number of mis-configured spam filters.

Reason #1 that spam filters tend to be ineffective: Sysadmins do not fill out the suggested (or even required) information in DNS, FQDN identification strings, etc. Because Admin's tend to get ahead of themselves and do not test for strict compatibility with the RFC standards. A lot of false-positive flagging by spam filters is because messages are coming from unverified sources because of missing PTR records, no SPF / DomainKeys information, Server HELO string containing garbage, etc.

If you setup your mail system with proper forward and reverse lookup addresses (stop using PTR records for your MX address!), proper message routing and anti-splashback, sane retry and throttling settings, SPF / Domainkeys, and HELO identification strings, you will likely have 0 problem sending mail to just about anyone. With the exception of Yahoo.com and BellSouth.net of course as they have drunk squirrels running their filters...)

I always operate off of the rule that my mail servers will comply with the RFC requirements to the letter while sending messages. While accepting messages I have to be a bit more lenient though because the other administrators on the net aren't quite as attentive (or even competent)...

Note: The filters and systems I administer process 30,000 messages daily for multiple domain names at multiple locations. This has been our biggest cause of marking messages as junk incorrectly when we are receiving them. Though less savvy sender system administrators like to blame us for having misconfigured junk mail rules.

Re:The real problem

[burning-toast]

I meant to say stop using CNAME records for MX resolution, not PTR records. Doh!

Re:The real problem

[Onymous Coward]

The technical arms race will continue unless politicians and law enforcement join the battle with effective measures that work across national borders.

I'd say instead "The technical arms race will continue regardless of political efforts." How likely is it that you'll get harmony enough on the legislation in conjunction with capable enforcement the world over? The fact that virtually nothing has been done so far is something of an indication of how effective governance-based anti-spam efforts will be.

Spamhaus paints another picture, though, of an organization with some degree of ability to identify spammers (and thus hold them accountable via DNSBLs), but turn that sort of operation into a government entity and it will instantly grow far too slugglish to be effective. As it is, the Spamhaus SBL gets such relatively few hits that it's negligible. (Props to Spamhaus just the same.)

If there's a problem with email breakdown, more often than not it is a result of failure to comply with RFCs. Interestingly, RFCs can be looked at as yet another political way to address the problem. So far RFCs have been a little more egalitarian and meritocratic (at the same time) than what we normally consider governance, but in essence it is still collectively decided rules. (And RFCs haven't always been the right way to do things.)

The network is too porous to be something you can lock down by laws and policing. I take issue with using "genius" to describe a failure to see that.

No, the answer is nowhere near governmental enforcement. The answer is grassroots, and it is technical. Anyway, that's my opinion.

Re:The real problem

[geekmansworld]

Yes, but what about when I go through MY mail server's logs and find that our servers never even communicated with the senders' server(s)? What do you suggest the problem is then?

I think you presume too much competence on the part of our customers. Often *ahem* SENDERS which exhibit the problem I have just described also send e-mail which has been marked as "Junk" or "Spam" by the SENDING server. That means that their mail servers' filters are scanning incoming AND outgoing e-mails.

Why? Perhaps laziness in ensuring security on a workstation-level. Perhaps their ISP is their outgoing MTA. Beats me.

Re:The real problem

[geekmansworld]

This is the most common argument used by people who say that legislation can not combat spam.

To that I would say that governments also need to use diplomacy as part of their efforts. If spammers move to [generic African nation], then western governments simply use a carrot or stick approach to ensure that country's co-operation in fighting spam.

No, you can't lock up every spammer. But you can prosecute enough of them that most potential spammers will think twice about what they do, and thus reduce the overall amount of spam to an acceptable level.

Re:The real problem

[Onymous Coward]

Outlawing something doesn't stop it.

To prosecute, you have to catch. A cooperative government isn't necessarily a capable one.

And botnets are only one route.

Re:The real problem

[kayditty]

I always operate off of the rule that my mail servers will comply with the RFC requirements to the letter while sending messages. While accepting messages I have to be a bit more lenient though because the other administrators on the net aren't quite as attentive (or even competent)...

Be liberal in what you accept, and conservative in what you send.

It's quite true what you say. RFCs weren't meant as law books, because the internet isn't a closed system. Strict regulations on MTA behavior and DNS configuration are completely unenforceable, but if many people can standardize on one particular protocol and follow it to its letter, then that's about one of the only ways things can work. False positives are simply a consequence that some are willing to take; that's a philosophical standpoint, and it's more one that affects them than you, unless you work for a company which relies heavily upon sending e-mail. I don't, and I don't rely heavily on receiving it either, so I check HELO, FQDN, reverse records, et al here at home, but I don't think I'd do that if I were running a critical mail system.

Re:The real problem

[illtud]

This man is a God-Damned genius:

I'm horrified that although this story is pretty old by now, yours is the first post that I've seen that acknowleges Wietse Venema in any way (OK, I'm reading at +3). His contribution to unix security and just his general cluefulness over the years has been exemplary. Anybody kicking his opinion had better have a massive gob of network security history behind them or they're going to look like a draft-dodger critizing a 4-tour vietnam vet.

mentions a few things, but no outline

[m0llusk]

FTA

A lot of things went into the Postfix mail system. Some were already discussed in this interview. It would take a lot of time and space to discuss everything, so I will just mention a few.

Freenet uses a similar technique against spam

[FreenetFan]

Freenet uses similar techniques against spam on the Freenet Messaging System (FMS).

Two things are mentioned in the article: many eyeballs, and moving to a pull technology from a push one.

FMS uses a web of trust, similar to PGP's to rate the trustworthiness of users, and this makes it simple to do collaborative filtering of spammmers (many eyeballs).

It also uses a pull technology, where each user has their own message queue, and you poll the queues of people you trust. There are tricks to make this scale up, so you don't have to be polling millions of people all of the time.

Initial entry to the web of trust is done mainly through a captcha system, although it can be done through any out-of-band method. Even if the captchas are defeated, which they will be regularly since this is an arms race, the first two steps should mitigate the damage done, by rapidly spreading bad trust values for the spammer to other users before they get to downloading their messages.

It works well in practice on a small scale, but obviously there are neither the numbers nor the dedicated spammers to test it out properly.

If anyone wants a challenge, please come on Freenet and try to spam the Freenet Messaging System!

Re:Still waiting for a "one solution" email produc

[Luyseyal]

One email every 3 seconds is not a difficult task, unless you work for a lawfirm that likes to email around PDF attachments running in excess of 100 MB. Then we'll talk.

-l

Sorry, won't work

[Anonymous Coward]

What stops a spammer from grabbing the signing keys and using them to their heart's content? Remember, most spam comes from botnets and they're not exactly technological idiots.

Re:Sorry, won't work

[calmofthestorm]

Web of trust similar to FMS on Freenet? I claim this will reduce spam dramatically, not eliminate it.

We need a pay-per-email protocol

[spectro]

Spam will be a problem as long as it is so cheap for them to send you their crap. Maybe with a new email protocol parallel to SMTP where you have to pay a fee for each email you send (this fee could be set on a sender by sender basis).

What I am talking here is a new email protocol with new server software and mailboxes. The big ones (google, hotmail, yahoo, aol, etc) could allow you to sign-up for these new mailboxes where you could even GET PAID a percentage of the fee for each email you receive.

This would not stop spam but how does it sound getting $0.02 for each spam you receive?

Re:We need a pay-per-email protocol

[techno-vampire]

This would not stop spam but how does it sound getting $0.02 for each spam you receive?

As long as it's used parallel to SMTP, as you suggested, it will be worthless because spammers will simply refuse to use it. And, blocking all mail sent via SMTP and only accepting this new protocol won't help either, because not everybody's going to have access to it.

Re:Obligatory checklist

[mr_mischief]

Most mail servers (all the good ones) already detect mail loops at least as a configurable option. That's not the problem.

If you're hash-validating mail to fix the problems of envelope-reversed spam, why not just hash-validate mail in the first place?

If I want to spam you, I can just send spam to someone using your proposed envelope reversal as you. The way to stop that is to require validated senders, but the server admin on the sending server gets to decide who's a valid sender. A spammer can afford a Linux box running an MTA, so unless you can force others to swear in court or something that their authenticated and authorized senders aren't sending spam, then you're still SOL.

Pull instead of Push?

[Anonymous Coward]

Simple, Email as it stands is flawed.

A botnet can just take over random computers and they can throw out emails everywhere with whatever message they want and make it look like it is from wherever. Change email to be pull instead of push. Now the email client has to connect to the sever to download any messages mean for it. Unless each botnet client has that URL pointing to it it is now impossible or at least very hard for them to work as a spam device. You can then white or blacklist servers rather than email addresses (Bit harder to setup servers for spamming purposes unless the server itself is botted) and you can use encryption and security certificates to verify the server's identity.

Of course actually changing something that has become so ingrained is nearly immpossible.

The war on spam

[cpghost]

Just make it clear to politicians that terrorists can hide their communications in the spam flow (to defeat traffic analysis), and you'll wonder how fast governments will scramble to not only outlaw spam, but also to target and prosecute spammers. It could be much more effective than any other technical solution. But is the benefit of catching the top-100 ROKSO spammers and sending them to Guantanamo worth the increased surveillance and governments' grip on the Internet?

Easy solution to spam

[Anonymous Coward]

Attack the demand, not the supply.

1. Send bogus spam
2. When someone responds, cut their Internet access for a week
3. Profitable spam responses decline and the spammers give up.

Re:Easy solution to spam

[Onymous Coward]

Actually, this is pretty clever.

If not actually feasible.

Re:If programming was a million times more difficu

[Skapare]

Certainly where programming is more difficult, those doing the programming can make more mistakes. But this rise in mistakes is not in proportion to increase in difficulty for the top programmers that would remain. The real serious impact of increased difficulty is that less programming would get done. The difficulty referred to is more about the entry barrier to programming, rather than the work itself. But anything that would slow down the programmer and make them think about what they were doing is a good thing in general.

We might well have a lot fewer programs. But that would be a good thing considering some of the junk out there.

I did some contract work for a company over the Y2k transition evaluating the bugs that happened leading up to, and following, the big year changeover. Only one such bug was found in a program in C, and it was found months ahead in some testing. Of about 50 bugs found, the vast majority were in "quicky" programs done in languages like Perl and shell scripts. Programmers used shortcuts that were easy to do in these languages (concatenating "19" to converted values of years since 1900, more often getting "19100" instead of "1900").

Good programmers were doing things the right way with respect to Y2k years before the transition. Of those bugs I could determine when they were coded, I ranked them by date and found that the median was in late 1998. That means half of the bugs were coded within less than 2 years of 2000. Programmers were simply not paying attention to what they were doing and how they were doing it.

That would be "outbox" for most of us...

[Anonymous Coward]

although if you want to call yours an inbox....

The "pull" model for email

[Skapare]

The best theoretic solution is to change the email distribution model, but this may never happen. Right now, email is a "push" technology where the sender has most of the control, and where the receiver bears most of the cost. The alternative is to use a "pull" model, where the sender keeps the email message on their own server until the receiver downloads it. For example, when my bank wants to send me email, they would send a short message with an URL to view their mail, and my email software would download the message for me. This assumes of course that my email software recognizes my bank's email digital signature and their Web site's SSL certificate, otherwise we would have a phishing problem. Legacy mail software would tell the user that they have email at their bank, and leave it up to the user to download their email.

The "pull" model would change the economics of email. It would move the bulk of the cost from the receivers where it is now, to the senders where it belongs. No-one would read email if its sender doesn't provide a service where recipients can download it from.

We could go ahead and establish a standard for this pull model. We don't have to suddenly change everything over to the pull model all at once since the asynchronous notification of a message being available would be sent via email. But with such a standard in place, this allows more legitimate senders to start using it, as well as mail agent/client to recognize it. It can be a gradual migration. The notifications would just look like an enclosed URL to an email agent/client that doesn't implement the detection of the notification.

Of course, spammers will use this method. But it forces them to have a server running somewhere to accomplish this. This issue would be addressed by performing certain validations on the notification that would normally not be doable on just any URL included in a message. Among the validations is that the URL must have a relationship to the server sending the notification (if it's the very same machine, that's a quick positive, but at the very least it might need to be the same domain name). Mere IP addresses as URL hosts would be rejected. And a specific port number might be specified by the standard for the pull server (notifications with a different port can be rejected). These validations would, among other things, make sure that botnet machines are not accessed to pull messages.

My point is, we can start doing this "now" (as soon as a standard is established). The transition can be a gradual process. And it can be one where we verify the correctness and make changes to the standard if necessary on a smaller scale.

Still, I have some concerns about the pull model. It does give the sender more information about the recipient (for example, what time of day that read mail). Some of that can be avoided with auto-pulling. And spammers can still cut their costs by not actually having duplicate messages on the pull servers, even though the URLs to access them would all be different (to avoid notifications being detected as duplications).

Re:The "pull" model for email

[John Bayko]

We could go ahead and establish a standard for this pull model.

I vote for RSS or Atom, modified for email support. Already working, already deployed, a good start.

Public awareness is required to fight spam

[ftekkie]

Quite rightly the importance of legal measures regarding the spam are most important. But that will not happen unless public awareness steps the stage. Both legislation-wise and privacy-wise. By privacy I mean the fact that most people don't care about their private data being left on discussion boards, lists, etc.

One simple step to prevent spam is to handle your data carefully and in cases in which you need to display your data, at least obfuscate it. There are many tools to do it, e.g. obfuscatr, a Dashboard widget for Mac OS X users. It provides JavaScript or just plain hexadecimal encoding of your email. See the details at obfuscatr 1.1.0 release announcement.

Of course, there are loads of online tools that perform similarly.

Re:Still waiting for a "one solution" email produc

[Anonymous Coward]

You could use something like Zimbra which takes care of most of your list of requirements... it's a little lacking in the mailing lists side of things (e.g. self administrating lists: list admins) but it should be easy enough to link to Sympa via SOAP. Otherwise it does mailing lists fine. SPF is flawed IMHO, instead look at BATV.

Ideally you'd run your own custom rolled Postfix MTA with greylisting, BATV and whatever else you please, then a Zimbra mailhost behind that. This way you retain all the standard control at the edge, and the Zimbra gooey goodness is available internally. The alternative is to run Zimbra standalone, though you give up some flexibility - or you have to get hacking to get it to do particular things. The trade offs/balance is up to you

Re:Obligatory checklist

[agbinfo]

If you're hash-validating mail to fix the problems of envelope-reversed spam, why not just hash-validate mail in the first place?

Because I don't want to impose any additional technical requirements. I want the SMTP protocol to work as is.

If I get a spammer that sends me email, his spam won't make it to my inbox. If it's a mailing list I didn't subscribe to, I don't care about it. If it's a legitimate unsolicited email and the sender hasn't been white listed yet then he probably won't mind having to respond to one time CAPTCHA.

On the other, if my mail server starts receiving a lot of requests for authentication then I can provide a way to easily filter spam. Since this type of email always comes from servers that provide authentication, I can safely let the mail through.

If I want to spam you, I can just send spam to someone using your proposed envelope reversal as you.

You can't spam me. You could spam someone else if the mail server of the forged "FROM" address doesn't check for the signature but what would be the point. The only message you'd spam with is a message with some text such as "I believe I have received an email from you. You address has not been white listed. To prove that this is not spam, please click here..."

The way to stop that is to require validated senders, but the server admin on the sending server gets to decide who's a valid sender. A spammer can afford a Linux box running an MTA, so unless you can force others to swear in court or something that their authenticated and authorized senders aren't sending spam, then you're still SOL.

If an authenticated server is sending spam, then that server will get a lot of reverse mail to ask for mail validation. That authentication only says that some mail came from that server. That mail is not trusted. It's considered spam until the sender is approved.

Here's a scenario:

• Spammer sends email with forged from.
• My mail server looks at to address, and it's not one of the personalized email addresses so it won't automatically accept.
• My mail server checks from address and compares to my white list; It's not there so it accepts email and puts it in possible spam folder then sends a request for self authorization;
• Since the FROM address was faked, there are a few possibilities.
  1. The from server I sent the request to ignores my request and treats it as spam so the spam email eventually gets deleted.
  2. The from server authenticates itself. My server sees that this is invalid and stops deleting the spam message.
  3. The from server doesn't authenticate itself but the inbox my server attempts to send to email to doesn't exist. It eventually receives a bounce. My server might try again depending on configuration.
  4. The from server doesn't authenticate itself and the inbox my server attempts to send to exists. The human recipient deletes the email and marks it as spam.

The worst that can happen is that some human recipient that got spammed is white listed because he chose to respond to the CAPTCHA or my mail server gets black listed on that particular server. If Google Mail or any other big source of email starts using this system, this would be very unlikely.

Proof of Work

[Agripa]

I am inclined white list and then require a Proof of Work to bring any message not on the white list to my attention without error prone automated spam checking. When possible, reject at the smtp level of course to avoid relying on the easily forged headers and provide immediate feedback.

Unfortunately, no Proof of Work authentication systems are available yet.

Re:Still waiting for a "one solution" email produc

[KlaymenDK]

In spite of your tone, I'm drawn to this discussion.

Personally, I would *love* to run my own mail server, but I *know* I'm bound to make a lousy job of it because, as you say, it's complicated as all-getout and only knowledgeable folk should be allowed to operate such machinery. Let Joe User do it and he'll flood the Internet with yet more spam.

The thing is though, this used to be the case, too, for media streaming file servers, setting up X on a BSD box, and so on -- but eventually, solutions cropped up that met people's needs in an easy way. Samba, SWAT, I know you know what I mean.

The gods know that I've read and read and read documentation, but frankly, I want to have better things to look back upon when my life eventually flashes before my eyes. Right now, I'm Joe, so it would be futile to try. Which leaves me without a mail server.

This being *nix, and mail applications being composed of many tiny pieces, it *should* (in nice, convenient theory) be possible to give people an easy way to install and configure a mail system piecemeal.

I guess the reason the situation is still as it is, is because traditionally the task of handling mail has been deferred to ISPs and big players; and only lately are people getting smart enough to question whether that really is in their best interest.

E.g. Postgrey

[KlaymenDK]

"Postgrey is a program which implements greylisting and is
designed to work with the Postfix MTA."

WWW: http://postgrey.schweikert.ch/

Available for FreeBSD at freshports ... see, it's not that weird a thought? We just need to make it a more common one.

Re:Do people really still have problems with spam?

[KlaymenDK]

Yes, people really still have problems with spam.

Yes, some people even don't want to use GMail, now isn't that crazy? It's a very, very nice AJAX email 'client', and it really does do wonders for (that is, against) spam. All you have to put up with is to let a huge and insanely powerful foreign corporation read your email, but that can't be so bad, now, can it?

Yes, it can.

Re:If programming was a million times more difficu

[statemachine]

If anything, programming needs to be easier, so more people would do it then we could have more solutions to choose from. A parallel brute force approach with selection can produce better solutions for everybody.

We already have that in a wide range -- VB, shell, etc., on up to C.

If you want easy, it's there. But then you just shift the bugs and "features" out of your code into someone else's. I'm not saying one is necessarily better, since requirements differ. But I don't think the number of bugs has been reduced. You just can't get any easier than some of the "languages" that are offered, so that's not the problem.

People being prone to error is the problem.

Same solution as Daniel J. Bernstein

[rg3]

It seems despite past tensions between Wietse Venema and Daniel J. Bernstein (the author of qmail), they agree on the approach to solving spam. Bernstein proposed a similar system called Internet Mail 2000 (Wikipedia entry).

Any "pull" projects out there?

[CleverDan]

From the article:

The alternative is to use a "pull" model, where the sender keeps the email message on their own server until the receiver downloads it.

Anyone know of any projects that make this easier to do?

Re:Do people really still have problems with spam?

[caluml]

Firefox + Gmail + http://getfiregpg.org/ = An excellent, private email service.

Yeah, I think you got it right off.

[Anonymous Coward]

Apparently, being helpful is not appreciated if you aren't also willing to stroke the egos of the people you're trying to help.

One of my favorite people on the net is Howard Chu. Howard frequently answers people's questions either with a link and no explanatory text or with a specific reference to a page in a book.

Sometimes people jump salty "How dare you tell me to RTFM you inconsiderate rude bastiche!" and get roundly flamed (usually not by Howard, incidentally, but by people like me who appreciate the clarity and brevity of his replies).

Pull model

[John Bayko]

It depends on how the pull model is implemented. For example, an RSS feed can't get spammed, because it's polling based - you know where it's coming from and you decide when to get something. It could be extended to add email functionality. For example, getting email from someone you don't know could be supported by allowing a notification message to be propagated through an intermediary known by both, or several intermediaries, all of who are essentially "vouching" for the trustworthiness of the sender.

Spam wouldn't be absolutely impossible in a system like that, but each instance would get shut down so fast (e.g. if someone's machine is infected, you can stop polling their feed, yet you can still send email to them telling them their machine is a bot now because they could still poll you), and the spread would be so limited, that spam would probably not be worthwhile anymore.

Err, no.

[KlaymenDK]

Err, no. Google's still got your mail.

Re:Err, no.

[vic-traill]

Err, no. Google's still got your mail.

I dunno. They've got a bunch of encrypted documents attached to your gmail address. Whether that is equivalent to having your e-mail is a topic for healthy debate, I'd say.

Re:Still waiting for a "one solution" email produc

[dbIII]

It processes about 1100 emails per hour

That used to be 60MHz Pentium territory with just about any mail transfer program you can think of. Of course MS Exchange does a lot of other stuff because it is rather dismal as an email program. Throw enough hardware at it (never in just a single machine unless you want trouble) and it works however - and with recent versions backups are now possible!

The very bad name came from the early versions. Stupid mistakes like being configured as an open relay by default and having to shut the entire steaming mass down to do backups gave MS Exchange the reputation it has today. I had the misfortune of dealing with three MS Exchange Version 5.5 production machines and a fourth used for disaster recovery drills - the mail kept coming through but with a vast amount of stuffing about. IMHO it wasn't good enough to be released as version one - it was really a hobby system you pay a huge amount for supported by an ecosystem of third party software just to keep it alive. Give me configuration files instead of registry hacks any day.

Greylisting no longer stops spammers

[dbIII]

The spammers know about it now so it no longer works very well. "Eventually" was about three weeks. The primary role of greylisting now is to provoke people into making phone calls to sysadmins when the urgent email they were promised fifteen minutes ago has not arrived.

Email is now assumed to be close to instant communication. Greylisting breaks this badly and the spammers moved on long ago.

You got it

[bogaboga]

That's what I meant. You got it, thanks. What we need is a script to grab Postfix and all the necessary addons to create a robust solution.

But I tell you...to get a Linux server running with all pertinent services in today's complex world is a feat in itself.

When one of the components gets an upgrade, chaos can reign. This should not be the case. What I was suggesting would take care of all this and many other issues.

Re:Still waiting for a "one solution" email produc

[oheso]

The certification program is not a barrier to entry -- no certificate is required to become an Exchange admin.

Blue frog

[Anonymous Coward]

I think the BlueSecurity people did it right; the only problem was that their infrastructure couldn't handle the war they created.

Okopippi has died.

Oh vell...

Re:Do people really still have problems with spam?

[JoeDuncan]

It's a very, very nice AJAX email 'client', and it really does do wonders for (that is, against) spam.

Sure, but you are not required to use their client. As I pointed out, you can use your own client (like Thunderbird).

All you have to put up with is to let a huge and insanely powerful foreign corporation read your email

You see now, that just sounds paranoid to me. I highly doubt that Google has the people power to actually have a human read every piece of mail that comes through their servers. Now having a computer scan my e-mail, and create content hashes etc... for the purposes of providing a better service and blocking 20-30 spam messages a day, I have no problem with, and am perfectly happy to consent to. Who cares if a computer inputs your e-mail content into an algorithm somewhere? It all just gets turned into a series of innocuous numbers, there's no actual person snooping through your mail.

Of course, Google has access to the stored content and could provide said data were they subpoenaed, but this is no different than any other ISP/e-mail provider. If you have e-mail, the people you pay for the service most likely have copies of all your e-mail on disk somewhere. Big deal. Google is no different in this respect than any other e-mail service provider. In this day an age, it is highly unlikely that you will find much effective difference between the terms of service of one e-mail provider to another. They're all pretty much the same (yes I am sure there must be some "high-end" premium services available that cost much more for more user-friendly terms of service - but this is a digression, I'm talking about e-mail service for regular users).

So what? Who cares if Google has access to my e-mail content? E-mail has never been guaranteed to be secure in the first place, and a lot of it is flying around the internet in a perfectly readable form to anyone capable of intercepting the data packets. You have to assume your e-mail is publicly readable by default anyway! If I do have something sensitive to say to someone, that I don't want anyone else to be able to read ever, I will encrypt it. It's hardly the job of my e-mail provider to guarantee my e-mail privacy when messages are being packet-switched across a public network.