Slashdot Mirror
Ask Slashdot: How Do You Handle SPF For Spam Filtering?
An anonymous reader writes "Our organization had had a decent SPF record of our own for a long time. Recently, we decided to try using SPF for filtering inbound mail. On the up side, a lot of bad mail was being caught. On the down side, it seems like there is always a 'very important' message being caught in the filter because the sender has failed to consider all mail sources in writing their record. At first, I tried to assist sending parties with correcting their records out of hope that it was isolated. This quickly started to consume far too much time. I'm learning that many have set up inaccurate but syntactically valid SPF records and forgotten about them, which is probably the worst outcome for SPF as a standard. Are you using SPF? How are you handling false positives caused by inaccurate SPF records?"
We're experiencing the identical problem, but keep on doing what you're doing. It is scary to see 'valid' mail from Fortune 500 companies who can't keep track of their own SPF records being dropped by Postini, but it is happening on a regular basis. We're doing what we can to help out those who are affected and eventually we will help reduce spam and get everyone into the 21st century with their outbound mail and SPF records.
Or anything else, for that matter--mark it as "spf failed" and score it as you feel appropriate for your filtering setup.
Spamassassin handles SPF, reasonably intelligently, that is, not trusting it completely, not giving it more weight than it deserves.
Hanging your spam fighting hat on any single hook is problematic. and SA uses a wealth of tools with constantly updating itself via
scripts. Its been largely trouble free, and we have it set up so that it will learn false positives and false negatives when users
move these to the corresponding folders.
I've been well served by Spamassassin for some time now, it runs quietly
on our mail server. SA does not block mail. It flags it. Our mail server will evaluate these flags and trash outright the most
egregious spam, but we have the limits set low enough such that we will allow the questionable things through.
We error on the side of caution, but we still dump a lot of mail right after SA flags it. (Our business can do that, your business
may not be able to do that.)
Sig Battery depleted. Reverting to safe mode.
Too many users are still careless with it. This is because it was proposed as a DNS standard, but was poisoned by Microsoft cluttering it with the entirely distinct "DomainKeys" project, then deliberately mislabeled the SPF version that they use. (See the history at spf.pobix.bom)
The result is that SPF has been less useful than expected. Also, SPF does not work well over mail relays unless they're configured to to indexing and re-writing of the "From " field, and pass the bounces through the relay server on its way back to the original sender. But it has been enormously effective to add to spam *scores*, to give anything with a bad SPF result a much lower score as a potentially valid email message.
The best mail filtering is based on geographic region of the envelope IP address. Most spam comes from countries without any proper anti-spam legislation, so simply junk them all until they get their act together.
Use SPF as part of a whitelist/blacklist scheme.
For sources that have their shit together, trust their SPF records as an absolute metric.
SPF does work if set up correctly.
We had the same problem.
Our solution: if the message makes it past all our spam filters and the only problem is that the sender's client isn't allowed by their SPF record, we send a friendly, plain-english informational message back to the sender, then deliver their message to our users.
"Please tell your company's IT staff that your mail server isn't set up correctly. This may cause your messages to be rejected or classified as spam. Please forward the following information to your system administrator: [SPF record] [sender's IP]. Thank you!"
Usually the issue is that the sender has set up GMail to send "from" their company email without sending through the company SMTP servers, usually without authorization.
We do the same thing with broken DKIM signatures.
This has worked well for us, since senders get tired of the message and seem to get things dealt with. We considered adding a "threat" saying that if the problem wasn't fixed, we'd block the sender, but that was rightfully shot down as pretty mean.
We actually mark on SPF checks on the outside but accept them. They then pass through a spam quarantine system which if they are SPF-FAIL or SPF-SOFTFAIL they are quarantined. Users can release them if they want to but they have to check their "Junk" reports to look for them or go to the portal itself. They have up to 15 days to release them or they are gone. However since these are a policy based rule, they can't white list senders that are messed up. Doesn't take long for a major vendor to learn they need to do email right.
Biggest problem is people with SPF records ending with ~ALL which basically means here is where we think our mail is coming from but if it is from somewhere else don't block it. Kind of stupid to not know where your email is being sent from. Especially annoying when a known company is setup like that that gets used as a phishing attack which is why we now quarantine SPF-SOFTFAIL now.
Thanks, Google!
I can't be bothered to run my own damn SMTP/IMAP/etc any more.
Here is how I increased my SPF using one weird trick!
Silence is a state of mime.
That works fine until the CEO misses an email from a prospective client.
That's what DMARC is for. It let's companies specify exactly how to handle their SPF (and DKIM) rules based on how thoroughly they have covered their bases. The company I work for deals with a ton of phishing against our user base and implemented SPF, DKIM, and DMARC with great success.
Google has excellent documentation on the protocol.
"Don't teach a man to fish, feed yourself. He's a grown man. Fishing's not that hard." - Ron Swanson
We just get Google to worry about spam for us.
We use Postini for archiving
We have low TTL MX records and a back up provider in case we have to worry about Google (again)
We have an account with Socket labs, in case Google screws up sending mail.
We have too few legitimate origins for maintaining SPF to be a problem
This works, is robust, piss easy to manage and dirt cheap. 150 - 200 employee company.
You CAN'T reject based on SPF, as you have learned. You'll lose too much valid mail.
So the best you can do is use it as part of a spam-scoring system, like SpamAssassin. Unfortunately, anti-spam systems that try to assign a "score", or try to analyze mail content, are ALSO worthless. Spammers have *completely* figured out how to get past any anti-spam system that analyzes content.
The sad fact is that the only way to effectively stop spam is to pay for an anti-spam service like Postini (which is going away, I hear), or buy something like an IronPort.
Basically, professionally-maintained, commercial blacklists are the ONLY really effective anti-spam system. If you are doing anything else, you aren't doing enough.
If they are too stupid to set it up correctly, then they aren't the fools whom you are supposed to part with their money?!
"Too stupid" is exactly the kind of person I'm looking for!
It's the smart people I don't want to hear from. You know the people I'm talking about: the ones who are so smart that they don't have to work, they just program their botnet to send viagra spam and sit back and collect the money. I admire them, but they're useless to me.
"Believe me!" -- Donald Trump
Add DMARC to your processing (http://www.dmarc.org/). It's a 'yes, I really meant it' notification for senders to communicate to receivers.
Other than that, the other suggestions already posted are about as good as it gets: use SPF as one element in scoring a message. Mark the message if your e-mail system allows it (e.g. label:authenticated in green, or label:authfail in yellow).
Let us live so that when we come to die, even the undertaker will be sorry -- Mark Twain
We had the same problem.
Our solution: if the message makes it past all our spam filters and the only problem is that the sender's client isn't allowed by their SPF record, we send a friendly, plain-english informational message back to the sender, then deliver their message to our users.
"Please tell your company's IT staff that your mail server isn't set up correctly. This may cause your messages to be rejected or classified as spam. Please forward the following information to your system administrator: [SPF record] [sender's IP]. Thank you!"
That's nice if you don't rely on email for communication with your customers. Generally, customers (and potential customers) don't respond well to nagging -- especially if whatever you're asking is outside of their control. They may be too small to have anyone to complain to (and no idea how to fix it themselves), or they may complain to the corporate helpdesk who says "Nope, our server is set up the way we want it".
If I got an email like that from a vendor I was trying to reach, I wouldn't be doing service with that vendor.
One vendor that we were paying support to was trapping our entire domain's email in a spam filter - none of our emails could go through, we ended up faxing problem reports to them. This went on for 2 months before they finally admitted that there was a problem in their (outsourced) email spam filter, they ended up crediting us 2 months of our monthly support contract because their SLA promised 24 hour response to emailed support requests, and we didn't get any response at all.
Because of course SMTP administration competence of the company's (possibly hosted) email is directly proportional to competence in the field the company works in.
Pull your head out of your arse - in the real world, businesses need to communicate.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Hells yes. I use postifx and reject connections that fail SPF with a 500 error. For example..
"550 5.7.1 : Sender address rejected: Please see http://www.openspf.org/Why?id=dassergey%40tadka.com&ip=103.22.182.230&receiver=mail.server.com%20:%20Reason:%20mechanism"
I have found that most senders just forward the bounce message to their administrator who (you would hope) have the skills to rectify the problem.
I also use DKIM http://www.dkim.org/ and DMARC http://www.dmarc.org/. They're worth checking out too.
We reject mails which fail the SPF check immediately within the mail session. That is the only safe way, because then the sender will receive a bounce message from his own mail server.
We never received complaints regarding SPF rejects, but maybe because we do not have large incoming mail traffic.
Even if there were false positives, it would not hurt anybody, because the sender is guaranteed to be immediately notified that his message had not reached its recipient. He could contact us using a different method, not mail - in addition to complaining to his (so called) system administrators.
Of limited usefulness. MD walks in "i'm expecting this email for a potential tender, the client tells me we are rejecting their email, and submissions close today!".
The MD receiving the odd spam is not a career limiting prospect. Missing a multi-million dollar contract due to non-delivery of business correspondence (or that reason being used as a scapegoat by your company's tendering department) potentially is.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
And meanwhile in the real world where nailing some important email because the sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA, your boss is right now handing you your walking papers.
The only sane way to use SPF is to drop a spam score of an email. Outright filtering on bad or missing SPF records is just a recipe for a large number of false positives.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Of course SPF filtering is not perfect; no one filter method ever is, thanks to all the badly configured mail servers out there. So, try not to build mail servers that reject incoming mail based on any one test, like one for SPF, or else you'll probably end up maintaining a lengthy whilelist, which will not only make you unpopular with the users, but is also a waste of time.
Instead, I built a mail system, using Exim (an extremely flexible MTA), that keeps track of a total number of 'strikes', as I call them, for positive scores for a number of filter categories, e.g. broken reverse lookup, bad HELO response, bad sender domain, blacklisted domain, no callout response, etc. Most of my filters are applied during the recipient phase of the delivery, before the MTA agrees to receive the message body (the data phase). If a message gets at least a certain number of strikes (in my case 3), it's rejected. If only 1 or 2, it ends up in the spambox of the intended recipient. These filter tests are separate from the ClamAV and spamassassin tests, both of which can still lead to an outright rejection.
In my experience, this method of counting strikes and using spamboxes to route messages to if they have only failed one or two filter categories has proved to be highly reliable and virtually maintenance free.
In other words you're generating soft bounces and making your server vulnerable to being used as an attack vector.
The world's burning. Moped Jesus spotted on I50. Details at 11.
The idea is good, but in practice it's horrible. Here's what we do:
We never give a bonus to anything scoring SPF "pass" unless its from one of a few large domains that we know have good SPF records.
We add a small score (1 SpamAssassin point) for SPF softfail.
We add a moderate score in most cases for SPF fail.
For specific, often-phished domains like paypal.com and ebay.com, we add lots of points for SPF fail and softfail.
Because of course SMTP administration competence of the company's (possibly hosted) email is directly proportional to competence in the field the company works in.
Yup, pretty much. If you walk around - alone - wearing an "i'm with stupid" t-shirt, I don't care if you make Stephen Hawking look like Forest Gump, people will steer clear of you.
Pull your head out of your arse - in the real world, businesses need to communicate.
Yes. Yes, they very much do. And if they don't take that function seriously enough to make sure their audience can hear them, do you really want to do business with them?
They also need to make pay their bills - Do you also overlook your customers just "forgetting" to pay you because they have their AP system set up poorly?
Unless, of course, your core business depends on a steady stream of "bigger idiots", in which case, just reverse the polarity of the SPFion flux.
We use Postini (now Google message security), and we turned on their SPF checks. Unfortunately, there are no knobs for this, so SPF pass is "good", SPF softfail is "bad", and SFP fail is "always quarantine, even if the address is whitelisted".
Despite some high-profile mail getting snagged by this, we educated our users and let them know that while we're more stringent with SPF than most places, it's not "our fault" that other people can't configure their domains correctly. After an initial spate of 10 or so messages where we had to call other IT departments to set them straight, it's more or less died down now and our users know how the problems get resolved.
We also created this page to send to the e-mail originator, so we didn't have to craft well-thought-out e-mail responses each time:
http://web.suffieldacademy.org/~jhealy/spf.html
All of this makes for a little more work for us, but it has cut down on spam. Also, it makes the internet a little better of a place as we slowly drag other domains into compliance by catching their mistakes. ;-)
For the past year, one of our business domains has been a patsy for a continual bombardment of spam against the internet. Typically our catch-all account would receive at least 200 bounce back failures a day. Since then, we enabled SPF on our domain and it drastically increased the amount of bounce-backs considerably... which is good, as it means less people are seeing our domain in a malicious light as they're immediately denying the spam as it's caught by an SPF filter before it goes through other filters.
I encourage all mail server admins to enable SPF to hamper the ability of spammers to hit their targets.
Please don't do this. One of the problems SPF solves is that spammers pick some random domain then spoof emails from that domain to send to millions of people. If you happen to be one of the unlucky people whose domain is targeted, you get a million bounces in your inbox.
The whole point of SPF is that if an email fails an SPF check, the email may not have come from the purported sender, and you should not treat it as genuine. You're completely undermining what SPF is for by doing this.
Bogtha Bogtha Bogtha
The way I do is is that I send an NDR on rejected mail caused by bad SPF records (with some anti-flood limits). So far, as far as I know, bounced emails always eventually reached us. What happens is the person gets a NDR mail ("We have rejected your email due to bad SPF records"). The person getting the bounce forwards it to their IT dept, where it's usually taken care of rather quickly. If it's a small business without a dedicated IT staff, I'm sometimes asked to explain how this works, but usually, such companies do not have SPF records at all, which means the mail won't bounce.
Religion is the best example of mass psychosis
This. SPF is fine as something you can use to tag messages and increase their filter score. If they lack a valid SPF there's a slightly higher risk it's spam. But to block based on SPF records is just goofy. It's a good idea, but nowhere near universally adopted and there's plenty of valid reasons why mail would go through a different source.
On my own mail server, I add 1 to the scoring, with tag at 3.5 and block at 5.5. Those have worked pretty well (I use Kerio Connect, which has a Spamassassin-based system).
-- Josh Turiel
"2. Do not eat iPod Shuffle."
SPF's great benefit is protecting your users against phishing attacks. Common phishing targets such as Paypal, eBay and Gmail, as well as banks and other finance organisations have very well managed SPF records. The onus for ensuring SPF records are correct falls on the sender, not the recipient. If someone inputs an SPF record for their server, then sends an e-mail via a different server you're doing exactly as requested by blocking the message. Problems of misconfigured SPF records will ease once more organizations implement it aggressively so that administrators will not be able to assume their mail is flowing correctly when delivery is initially successful to a few lax servers.
Relying on any form of email for multi-million dollar tender delivery is the career limiting move here.
Patent litigation: A doctrine of Mutually Assured Destruction... in which everyone seems willing to push the button
If you walk around - alone - wearing an "i'm with stupid" t-shirt, I don't care if you make Stephen Hawking look like Forest Gump, ...
Oh come on! Forest Gump would run circles around Stephen Hawking, any day of the week!
I havent seen it in a while, but for a while we kept having issues with some implementations of Exchange refusing email based on our perfectly valid SPF records. The problem only occurred if we had both SPF AND SenderID (M$' so called "SPF 2.0") records for our domain. If we deleted our SPF records and only left the SenderID record, the mail would be delivered.
Way to go Microsoft in taking a perfectly good standard and deciding its not QUITE good enough and muddying the waters with your own crap.
You can use SPF as a hint. A bad SPF record could trigger longer greylisting delay, therefore lowering the ability of a spammer to reach destination before been known by blacklists. milter-greylist allows such setup.
SPF means Sender Policy Framework. It is a method to specify and check the precise list of servers which are allowed to send mails in the name of a domain.
Receiving servers use it to check the validity of the reverse-path of the mail. Reverse-path is the address to where bounce messages should be sent.
It is nothing more, nothing less. It causes great confusion because many thinks that it is intended to be a final SPAM prevention method, when it is clearly not.
Alone, it makes sending SPAM only a bit more difficult. On the other hand it does make easier to catch spams with other methods. As more and more domains and receiving servers start to use it, it becomes more and more effective in this role.
The simplest benefit of setting up SPF for a domain that your domain name cannot be forged as the reverse-path of spam mails. So you will get less back-scatter spam and angry mails. SPF almost completely eliminated back-scatter spam for us, which was once quite an annoying issue.
Sure. But shit rolls down-hill, and it's not the MD who's going to get fired.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Put it this way - in an ideal world, I agree with your premise. But the reality is that people expect email to work, and don't care WHY it failed. The fact that you bounced their legitimate correspondence is unacceptable to the business side of the company. The one that brings IN money.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
In other words you're generating soft bounces and making your server vulnerable to being used as an attack vector.
Nope, we do limits - one message per week per sender. So no effective attack vector.
The whole point of SPF is that if an email fails an SPF check, the email may not have come from the purported sender, and you should not treat it as genuine. You're completely undermining what SPF is for by doing this.
Yeah, if everyone did this it would be bad - zillions of polite informational messages. You make a good point.
On one hand we're a small shop with a low-traffic email server. We've had good success getting sending domains to fix their record. On the other hand, perhaps we should go back to passive mode. I'll mention it to the system administrators.
If I got an email like that from a vendor I was trying to reach, I wouldn't be doing service with that vendor.
One vendor that we were paying support to was trapping our entire domain's email in a spam filter - none of our emails could go through...
I see what you're saying. Just wanted to emphasize that email messages are never dropped in our system, they are always delivered. The SPF check is still part of the spam checks, but by the time we get to the "send informational message" stage we've already decided the message isn't spam. We also check the sending IP against the Backscatterer RBL, which hopefully keeps us from sending to domains that have been hijacked.
The informational message is only sent once per week to any given sender. We do have an "don't email me about this again" link in the email. Finally, we don't send the message to anyone sending from a rather large commercially-maintained list of known webmail address and ISP mail servers, which exempts a large portion of senders from ever seeing the message (these users can't do anything to fix a broken spf record).
Yes, people would steer clear of someone who looks superficially stupid, that is a given. That isn't a defense of why you should steer clear of such a person though, especially if their skills and services are of particular use, or if they are a potentially big customer. By rejecting providers, you are going to tend to pay more for services because now you only look at a subset of potential providers, or by rejecting customers, you are rejecting sources of income. Either way, there is a cost associated with that. If it turns out it only costs you a couple hundred dollars in the long run, then maybe it is worth the saved effort by your IT department. If it is going to cost you much more, is it really worth it? How many thousands of dollars in extra costs or lost revanue is acceptable because you don't feel like dealing with such people?
It reminds me of when a coworker once started up a simple online store in his free time and offered me a bit of money to look over the front and backends while he still learned web development. I found out his website redirected IE users to a page telling them to get Firefox and didn't let them get to the actual store. As this was around 2005 or 2006, the logs showed that 90% of his traffic was being redirected to that page, including cases of people trying multiple times to get back to the product list. His response, "I don't want to deal with people too stupid to use IE, and I would have to waste time to make my site work on IE too." My response, "First, if I copy-paste your pages, they completely functional in IE as is. Second... you sell hot sauce, what do you care what browser people use?" He just brushed it off, and continued to insist that he didn't want to bother with such people. Considering he was able to get a couple hundred dollars a month after a bit of local promotion, and assuming there isn't some massive correlation between browser use and hot sauce purchasing, he chose not to turn that into couple thousand dollars a month over such superficial, trival BS.
So, exactly how much money would you give up because you want to tell a customer, "Sorry, we don't want to accept your email," even if your business dealt with products that has nothing to do with email otherwise?
LOL Modpoints, tha sarcastic humor is killing me!
Huh? SPF implementations do not filter on a missing SPF record. A missing SPF records just means that, the sending mail servers of the domain are not specified. It never meant that the domain must not send any mails from any servers.
Quote from openspf.org :
"The domain does not have an SPF record or the SPF record does not evaluate to a result. Intended action: accept"
I don't think you know what SPF is or how it works.
SPF specifies, by leveraging the DNS, which servers are allowed to send you mail from that domain. The problem it addresses is "spoofing" - I send an email to your servers claiming to be from you. If you have a SPF record, you can say servers x,y,z,a-c are allowed to send email claiming to be from foo.com.
There's three possibilities:
1) No SPF record - process normally
2) SPF record, sent from valid host - generally downweight spam score, since verifiably the server that sent the email was in that organization
3) SPF record, sent from invalid host - the subject of this Ask Slashdot. It should be "always bad" but Incompetent admins set up SPF, then get it wrong or don't update it and mail bounces. Hence the problems discussed here.
"sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA" - no problem if the SPF record is correct. Or if you can't be bothered to do that, just get rid of it.
Filtering on missing SPF records is stupid. Filtering on bad (i.e., sending host was marked as disallowed) SPF records should be 0% false positives, were it not for idiot admins - again, the subject of this AS
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
And meanwhile in the real world where nailing some important email because the sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA, your boss is right now handing you your walking papers.
The only sane way to use SPF is to drop a spam score of an email. Outright filtering on bad or missing SPF records is just a recipe for a large number of false positives.
You don't know how SPF works, right? - Because that would excuse your statement...
Basically, SPF is a fail-open system:
A) No SPF: Allow the mail (fail-open)
B) SPF present and the mail fails the check: Refuse the mail
C) SPF present and the mail passes the check: Allow the mail
Option B has an exception for 'soft-fail' if the SPF uses the ~all. It will allow all mail through but tags those that fail SPF.
There's absolutely no reason not to refuse mail if the SPF check fails.
Remember, an incorrect SPF will result in a lot of mail lost, and it won't take many minutes before this is noticed. The fastest way to fix it is not to contact every single communication peer and have them bypass their SPF check; it is of course to just fix the SPF record. A 10-second dialogue with a competent postmaster at just one of the failed recipients should yield the reason for the fail, like the missing mail source. All that's left is to fix the SPF by adding the relevant "a:xxxxx.xxx" or "ip4:xx.xx.xx.xx" to the record.
"For every complex problem, there is a solution that is simple, neat, and wrong." -- H.L. Mencken (1880-1956) --
It is better to reject it at SMTP level and let the sender know about the fact immediately, instead of scoring the message low and lost in the spam folder, and the sender has no idea if it reached the recipient.
So you get hit with a 10.000 address spam run.
Are you still contending that you are not a nuisance?
Shut. that. system. down.
"I know I will be modded down for this": where's the option '-1, Asking for it'?
Except that email is commonly used for such. I have negotiated contracts worth hundreds of thousands of dollars by email. (not yet multi-million, but well on my way, nonetheless)
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Antispam measures are effective only when methods are combined, with no single method being more important than everything else.
Also, graylisting in front of your mail server is one of most effective methods to defend. Legitimate mail infrastructure will rarely (if ever) use different IP address for every delivery tried.
Graylisting, combined with RBL, combined with spamassassin (with SPF and most other methods used through it)... And you have decent solution. What remains is to define your policies and your spam problems are terminated, for looong time.
http://opencm3.net, http://www.nongnu.org/gm2/
Not only I use SPF, but I also maintain the tumgreyspf package in Debian, which does both SPF and greylisting.
And for those who have bad SPF records, and send email from the wrong IP? Well too bad, they wont be able to write to me. But as well to so many other persons (gmail, hotmail, etc.) that it's not really a huge concern, and that I don't feel like I should be the one having to spend the time explaining how to configure things.
There's truth in that, sadly. The only rejected mail I've sent were spam reports via SpamCop to a notorious company, who rejected it because of SPF errors.
Is the sender IP whitelisted? Allow.
Is the sender IP not presenting a valid reverse DNS name? Block.
Is the sender IP listed on SpamHaus? Block.
Does the sender IP not match the domain SPF record (even soft-fail is a fail)? Block.
Is the sender IP not retrying after being told to try again in five minutes (greylisting)? Block.
And then you can do things to do with the actual message itself, like DKIM signing, spam-detection, etc.
Instantly cut out 99.9% of your spam, and at the same time anyone who fails those tests isn't running a mail server worthy of the name in this day and age. If your ISP even ALLOWS you to send email from an address without a valid reverse-DNS, for example, I'd question what the hell they are doing with their systems or why they expect customers to be able to email at all without being blacklisted by someone.
Anyone who can't pass those tests isn't going to be able to send email to anyone else anyway. It won't just be your company, but almost everyone they communicate with (Google, etc. pretty much do the same run of tests on your name, if not more strict!). And if they aren't bothering to keep their SPF records up to date, I'm surprised that they get mail sent at all.
But don't, under any circumstances, accept the mail for later delivery. Reject it there and then. Don't faff about wasting resources, providing "proof" that the email was received to the sender, or storing things up that you then aren't sure you should deliver.
In the small businesses I've set up with email, this literally cuts 99.9% of all spam, and the number of false positives is effectively zero (and usually, as you say, due to idiots not setting / updating their email / domain properly). And when one of those botnets gets hold of your domain and tries to spam you with hundreds of connections a second, you can work through them easily without having to worry about them ending up in mailboxes.
This is the default setup for things like Postfix on most Linux distros, for example. I fail to believe that those defaults are changed by everyone who sets up a mailserver, or that setups that DON'T confirm to those tests actually send much email to anyone without having problems.
And, if you really, really, really are a small business that has problems that you can't get around by just using your ISP's recommended method, sticking Google Apps on a domain, or renting a £10/month VPS that does it all for you - WITH THOSE DEFAULTS, and more - is literally chicken-feed for communicating with customers and suppliers.
If they are idiots, that's their problem. If you want to accept and deliver email that's quite obviously spoofed because you're afraid of catching up their emails by mistake, that's your problem. Only one of those types of problems can be fixed.
If it comes to it, and you hear complaints from those companies, just whitelist their domains if their business is that important to you. But if they can't even be bothered to work out why, I would guess, at least 10-50% of their email just bounces or never gets to the other end, there's nothing you can do about that.
If you have correct SPF, you will most likely get more spam. Why? Someone spams mailservers with your domainname, then it bounces because of failed SPF, and many mailservers are configured wrong, so you will get backscatter spam.
The irony of this is, that SPF says "it wasn't me" and the wrong configured mailservers then send YOU an answer saying "hey, we're not accepting your mail, because your SPF says it is not your mail".
you are destroying the sense of SPF.
consider someone spams you from a faked domain (the thing spf should prevent). Then you send backscatter-spam to the real domain.
fail.
Here's the problem, and it's a fundamental problem with the SPF framework, as a business process. The consequences of error should reside with the entity that has the most direct control over what causes or prevents that error. What is happening here is that Entity A has their SPF set up wrong, but potentially all the impact could land on Entity B. SPF isn't rocket science, but it's the kind of thing that needs to be checked periodically and tied tightly into change control processes...in other words, a perfect example of a system that will have issues, in modern IT environments. I don't really have a suggestion, other than a standardized process for detecting events like this and reporting them to the sending organization. But it's hard to notify a company to let them know that their website has been hacked; I imagine it must be horrendously hard to find whomever is in charge of their mail infrastructure to point out to him that he's doing SPF wrong.
For your security, this post has been encrypted with ROT-13, twice.
If they are too stupid to set it up correctly then they don't deserve to work with you.
100% right but still missing the point.
There are people running mail servers who should not be, they don't know what they are doing or they don't spend enough time getting it right. As a company you can't reject these mails because they could be worth real money. There are jokers in the world and us hard working people have to make allowances for them.
Just use SPF as part of a spam scoring system, but never to outright block mails.
That works fine until the CEO misses an email from a prospective client.
Unless you plan to profit from stupidity, that prospective client is worthless if they can't even set up a functional SPF record. Either you're too stupid to know about SPF or you do it right. Everything else is dumb beyond reason.
Lots of people are dumb as a brick when it comes to IT. Some of these people manage mail servers and DNS servers. Some of them want to buy stuff from your company. This makes their stupid misconfiguration your problem. Much though I'd enjoy burning these morons at the stake you can't burn your customers alive and expect them to keep buying from you. ( Unless you are Microsoft. )
Don't block on SPF - Use it as part of a spam scoring system.
And meanwhile in the real world where nailing some important email because the sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA, your boss is right now handing you your walking papers.
Or you are handing your boss walking papers because that business you just missed out on was the difference between profit and going bust.
The only sane way to use SPF is to drop a spam score of an email. Outright filtering on bad or missing SPF records is just a recipe for a large number of false positives.
That's what I've said twice already in this thread. I don't think its a large number of false positives but there certainly are some.
There's absolutely no reason not to refuse mail if the SPF check fails.
Yes there is. As other people have pointed out bad admins set it incorrectly every once in a while. Companies change their ISP and forget to update DNS, or alter their internal setup so the mail hits the internet from a different IP.
SPF means the sender's domain admin EXPLICITLY configured their domain to advertise a "drop messages that didn't come through X MTA".
User of that domain should be informed of this, and if important emails are lost due to bad SPF records, then it's the admin (or part of the sending organization's) fault.
Please, please don't do this. You take SPF, designed as a defensive shield for you, turn it into an offensive weapon and unleash it on an innocent victim every time you send out a "friendly" message like this. Please turn this off now and then Google "joe job" to learn how you are making the problem worse, rather than addressing it in any meaningful way.
Ignore SPF if you like, but don't use it as an method to select targets to attack with "friendly fire."
you are destroying the sense of SPF.
consider someone spams you from a faked domain (the thing spf should prevent). Then you send backscatter-spam to the real domain.
fail.
Your argument applies to any spam filter, not just SPF. If you do the rejection during the delivery, the sending mailserver (the spammer) gets it. If you accept the email, then later send the NDR based on the (possibly forged) details in the email, you may be contributing to backscatter. This has nothing to do with SPF, and everything to do with how your mail system accepts and rejects messages.
No, because you send the NDR to a adress, which was detected by SPF as NOT being the source of the message. So you reach the wrong person. So instead you should just send nothing, if you filter based on SPF, because the whole point in spf is, when you filter it, you do it because you do not know the source (and the pretended source-adress is wrong)
definately a +1
nosig today
No, because you send the NDR to a adress, which was detected by SPF as NOT being the source of the message. So you reach the wrong person. So instead you should just send nothing, if you filter based on SPF, because the whole point in spf is, when you filter it, you do it because you do not know the source (and the pretended source-adress is wrong)
So you're saying there is no solution?
That works fine until the CEO misses an email from a prospective client.
Unless you plan to profit from stupidity, that prospective client is worthless if they can't even set up a functional SPF record. Either you're too stupid to know about SPF or you do it right. Everything else is dumb beyond reason.
The logic behind what you're saying is good in nature, but bad in execution; I can with 100% surety say that I have lost business from people with tons of money to blow because someone who doesn't know how to configure their IT environment properly got them to blow tons of money on them.
Translation: I lose money when I don't get email from 'the ignorant'.
...Much though I'd enjoy burning these morons at the stake you can't burn your customers alive and expect them to keep buying from you. ( Unless you are Microsoft. )...
Ooooooh Burn! And Bravo! :)
No, if you reject during the SMTP session, you don't send an NDR at all, you send a 5xx SMTP response code. (It doesn't matter if the reason for rejecting is SPF-related or not). It is then up to the sending server to inform the sender with an NDR, and the sending server should only be sending on behalf of authorized/authenticated users, or be a permitted relay for such a server.
Anything detected as junk after the SMTP session is over should be marked as junk in some way so it can be stored in a junk folder. If an NDR is written then, it could be sending backscatter. If it's simply discarded, neither the sender or receiver will know when false positives are occurring.
Agreed. Any decent outbound MTA should handle the 5xx error code properly and send their own Non Delivery Report to the original sender.
Many mail servers are returning 5xx error codes due to forward/reverse DNS not matching, unknown recipient, content spam filtering, etc. which are being handled just fine. Bouncing due to SPF failure is preferred.
The short answer is: If you tell us that an e-mail from you is not to be trusted, we will honor that request. But if our system falsely catches an e-mail from you, say because of a bad SPF record, we will whitelist that sender.
We have a custom system I built and haven't yet had time to polish for a open source project (and it needs it before it could be publicly released). It has an awesome feature though...
Messages are rejected at the SMTP level, for things like SPF, greylisting, and SpamAssassin. The bounce message has a URL that the sender can visit to release the message. I was anticipating this might get abused and have to have a captcha on it, but so far it has not. Actually, it's worse than that, 99+% of people who get this message don't read any further than the subject, which is generated by their mail server, so they usually contact us some other way saying "Your mail server is broken".
BUT, the killer feature is that our users can go to a website and see the messages sent to them, and "release" them. So if you are looking for an important message no problem! You go to that page, type Control-F and the sender name or subject you are expecting, and click a button, and you have the message. It's kind of like a quarantine, but it's controlled by the sender AND the recipient.
Now, as far as what we do about senders that have broken SPF... We add them to a whitelist and tell them "You've been whitelisted, but your domain is publishing a notification that says this e-mail is invalid, you probably want your mail server admin to fix this because other places are honoring this as well."
No big deal, not ideological stands, we just deal with it and report it on.
Because that's the crucial thing: Your domain is telling me that this e-mail is not to be trusted. The CEO of our company understands that yelling "You can never have a false positive" means that they are going to have to deal with an inbox full of sewage -- they understand what false negatives are.
Sean
There's absolutely no reason not to refuse mail if the SPF check fails.
How about mailing lists?
yeah, and the problem is, there are servers, which send NDR because of SPF.
And even when you are a server in the chain, which accepted the message, you SHOULD NOT send a NDR, because SPF is saying "hey, the sender-info is wrong". So a NDR is pointless, in ANY case, even when you accepted the message for delivery.
No you fail. Backscatter-spam does not exist if everyone does SPF. Since you can not fake a domain.
The "We have rejected your email due to bad SPF records" will surely get back to the domain whose SPF records are bad, regardless of if the email causing it was fake or not the bad SPF records still need to be fixed and are the responsibility of domain owner to manage.
Correctly setup SPF records on a victims domain will cause fake emails to be rejected (with no "We have rejected ..." message generated).
No backscatter-spam issue.
nope, you fail hard.
How do you detect a bad configured SPF? You are getting a mail, from a server listed as "cannot send mail". Now this can happen, when you have perfectly setup SPF and a spammer just spams from his own host ignoring that you're using SPF. Now the target system bounces the spam to your system, saying "hey, spammer-system cannot send mail for you".
You set up SPF, because you WANTED that the spammer-system cannot send mail for you. But you certainly did not want other systems to bounce the rejected mails to you.