Slashdot Mirror
Ask Slashdot: How Do You Handle SPF For Spam Filtering?
An anonymous reader writes "Our organization had had a decent SPF record of our own for a long time. Recently, we decided to try using SPF for filtering inbound mail. On the up side, a lot of bad mail was being caught. On the down side, it seems like there is always a 'very important' message being caught in the filter because the sender has failed to consider all mail sources in writing their record. At first, I tried to assist sending parties with correcting their records out of hope that it was isolated. This quickly started to consume far too much time. I'm learning that many have set up inaccurate but syntactically valid SPF records and forgotten about them, which is probably the worst outcome for SPF as a standard. Are you using SPF? How are you handling false positives caused by inaccurate SPF records?"
Or anything else, for that matter--mark it as "spf failed" and score it as you feel appropriate for your filtering setup.
Some would say you are doing a disservice to your customers by continuing a practice that is hurting their business in an effort to promote a technology standard that is not working.
Spamassassin handles SPF, reasonably intelligently, that is, not trusting it completely, not giving it more weight than it deserves.
Hanging your spam fighting hat on any single hook is problematic. and SA uses a wealth of tools with constantly updating itself via
scripts. Its been largely trouble free, and we have it set up so that it will learn false positives and false negatives when users
move these to the corresponding folders.
I've been well served by Spamassassin for some time now, it runs quietly
on our mail server. SA does not block mail. It flags it. Our mail server will evaluate these flags and trash outright the most
egregious spam, but we have the limits set low enough such that we will allow the questionable things through.
We error on the side of caution, but we still dump a lot of mail right after SA flags it. (Our business can do that, your business
may not be able to do that.)
Sig Battery depleted. Reverting to safe mode.
Too many users are still careless with it. This is because it was proposed as a DNS standard, but was poisoned by Microsoft cluttering it with the entirely distinct "DomainKeys" project, then deliberately mislabeled the SPF version that they use. (See the history at spf.pobix.bom)
The result is that SPF has been less useful than expected. Also, SPF does not work well over mail relays unless they're configured to to indexing and re-writing of the "From " field, and pass the bounces through the relay server on its way back to the original sender. But it has been enormously effective to add to spam *scores*, to give anything with a bad SPF result a much lower score as a potentially valid email message.
Use SPF as part of a whitelist/blacklist scheme.
For sources that have their shit together, trust their SPF records as an absolute metric.
SPF does work if set up correctly.
We had the same problem.
Our solution: if the message makes it past all our spam filters and the only problem is that the sender's client isn't allowed by their SPF record, we send a friendly, plain-english informational message back to the sender, then deliver their message to our users.
"Please tell your company's IT staff that your mail server isn't set up correctly. This may cause your messages to be rejected or classified as spam. Please forward the following information to your system administrator: [SPF record] [sender's IP]. Thank you!"
Usually the issue is that the sender has set up GMail to send "from" their company email without sending through the company SMTP servers, usually without authorization.
We do the same thing with broken DKIM signatures.
This has worked well for us, since senders get tired of the message and seem to get things dealt with. We considered adding a "threat" saying that if the problem wasn't fixed, we'd block the sender, but that was rightfully shot down as pretty mean.
That works fine until the CEO misses an email from a prospective client.
That's what DMARC is for. It let's companies specify exactly how to handle their SPF (and DKIM) rules based on how thoroughly they have covered their bases. The company I work for deals with a ton of phishing against our user base and implemented SPF, DKIM, and DMARC with great success.
Google has excellent documentation on the protocol.
"Don't teach a man to fish, feed yourself. He's a grown man. Fishing's not that hard." - Ron Swanson
If they are too stupid to set it up correctly, then they aren't the fools whom you are supposed to part with their money?!
"Too stupid" is exactly the kind of person I'm looking for!
It's the smart people I don't want to hear from. You know the people I'm talking about: the ones who are so smart that they don't have to work, they just program their botnet to send viagra spam and sit back and collect the money. I admire them, but they're useless to me.
"Believe me!" -- Donald Trump
Because of course SMTP administration competence of the company's (possibly hosted) email is directly proportional to competence in the field the company works in.
Pull your head out of your arse - in the real world, businesses need to communicate.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
We reject mails which fail the SPF check immediately within the mail session. That is the only safe way, because then the sender will receive a bounce message from his own mail server.
We never received complaints regarding SPF rejects, but maybe because we do not have large incoming mail traffic.
Even if there were false positives, it would not hurt anybody, because the sender is guaranteed to be immediately notified that his message had not reached its recipient. He could contact us using a different method, not mail - in addition to complaining to his (so called) system administrators.
See megaupload. If you're a business, using cloud a service for email storage is just WAY too legally grey right now. IMHO.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
And meanwhile in the real world where nailing some important email because the sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA, your boss is right now handing you your walking papers.
The only sane way to use SPF is to drop a spam score of an email. Outright filtering on bad or missing SPF records is just a recipe for a large number of false positives.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Anyone who understands SMTP and spam knew from the very moment that SPF and its cousins/descendants were proposed that it was a hopeless measure. That, after ten years, guys like me are still having to explain "setting your SMTP server to reject because of SPF" tells you just how badly SPF failed.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Please don't do this. One of the problems SPF solves is that spammers pick some random domain then spoof emails from that domain to send to millions of people. If you happen to be one of the unlucky people whose domain is targeted, you get a million bounces in your inbox.
The whole point of SPF is that if an email fails an SPF check, the email may not have come from the purported sender, and you should not treat it as genuine. You're completely undermining what SPF is for by doing this.
Bogtha Bogtha Bogtha
The way I do is is that I send an NDR on rejected mail caused by bad SPF records (with some anti-flood limits). So far, as far as I know, bounced emails always eventually reached us. What happens is the person gets a NDR mail ("We have rejected your email due to bad SPF records"). The person getting the bounce forwards it to their IT dept, where it's usually taken care of rather quickly. If it's a small business without a dedicated IT staff, I'm sometimes asked to explain how this works, but usually, such companies do not have SPF records at all, which means the mail won't bounce.
Religion is the best example of mass psychosis
This. SPF is fine as something you can use to tag messages and increase their filter score. If they lack a valid SPF there's a slightly higher risk it's spam. But to block based on SPF records is just goofy. It's a good idea, but nowhere near universally adopted and there's plenty of valid reasons why mail would go through a different source.
On my own mail server, I add 1 to the scoring, with tag at 3.5 and block at 5.5. Those have worked pretty well (I use Kerio Connect, which has a Spamassassin-based system).
-- Josh Turiel
"2. Do not eat iPod Shuffle."
Relying on any form of email for multi-million dollar tender delivery is the career limiting move here.
Patent litigation: A doctrine of Mutually Assured Destruction... in which everyone seems willing to push the button
If you walk around - alone - wearing an "i'm with stupid" t-shirt, I don't care if you make Stephen Hawking look like Forest Gump, ...
Oh come on! Forest Gump would run circles around Stephen Hawking, any day of the week!
Yes, people would steer clear of someone who looks superficially stupid, that is a given. That isn't a defense of why you should steer clear of such a person though, especially if their skills and services are of particular use, or if they are a potentially big customer. By rejecting providers, you are going to tend to pay more for services because now you only look at a subset of potential providers, or by rejecting customers, you are rejecting sources of income. Either way, there is a cost associated with that. If it turns out it only costs you a couple hundred dollars in the long run, then maybe it is worth the saved effort by your IT department. If it is going to cost you much more, is it really worth it? How many thousands of dollars in extra costs or lost revanue is acceptable because you don't feel like dealing with such people?
It reminds me of when a coworker once started up a simple online store in his free time and offered me a bit of money to look over the front and backends while he still learned web development. I found out his website redirected IE users to a page telling them to get Firefox and didn't let them get to the actual store. As this was around 2005 or 2006, the logs showed that 90% of his traffic was being redirected to that page, including cases of people trying multiple times to get back to the product list. His response, "I don't want to deal with people too stupid to use IE, and I would have to waste time to make my site work on IE too." My response, "First, if I copy-paste your pages, they completely functional in IE as is. Second... you sell hot sauce, what do you care what browser people use?" He just brushed it off, and continued to insist that he didn't want to bother with such people. Considering he was able to get a couple hundred dollars a month after a bit of local promotion, and assuming there isn't some massive correlation between browser use and hot sauce purchasing, he chose not to turn that into couple thousand dollars a month over such superficial, trival BS.
So, exactly how much money would you give up because you want to tell a customer, "Sorry, we don't want to accept your email," even if your business dealt with products that has nothing to do with email otherwise?
Huh? SPF implementations do not filter on a missing SPF record. A missing SPF records just means that, the sending mail servers of the domain are not specified. It never meant that the domain must not send any mails from any servers.
Quote from openspf.org :
"The domain does not have an SPF record or the SPF record does not evaluate to a result. Intended action: accept"
I don't think you know what SPF is or how it works.
SPF specifies, by leveraging the DNS, which servers are allowed to send you mail from that domain. The problem it addresses is "spoofing" - I send an email to your servers claiming to be from you. If you have a SPF record, you can say servers x,y,z,a-c are allowed to send email claiming to be from foo.com.
There's three possibilities:
1) No SPF record - process normally
2) SPF record, sent from valid host - generally downweight spam score, since verifiably the server that sent the email was in that organization
3) SPF record, sent from invalid host - the subject of this Ask Slashdot. It should be "always bad" but Incompetent admins set up SPF, then get it wrong or don't update it and mail bounces. Hence the problems discussed here.
"sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA" - no problem if the SPF record is correct. Or if you can't be bothered to do that, just get rid of it.
Filtering on missing SPF records is stupid. Filtering on bad (i.e., sending host was marked as disallowed) SPF records should be 0% false positives, were it not for idiot admins - again, the subject of this AS
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
And meanwhile in the real world where nailing some important email because the sender was sending all his email through a local MTA because his ISP doesn't have an externally accessible MTA, your boss is right now handing you your walking papers.
The only sane way to use SPF is to drop a spam score of an email. Outright filtering on bad or missing SPF records is just a recipe for a large number of false positives.
You don't know how SPF works, right? - Because that would excuse your statement...
Basically, SPF is a fail-open system:
A) No SPF: Allow the mail (fail-open)
B) SPF present and the mail fails the check: Refuse the mail
C) SPF present and the mail passes the check: Allow the mail
Option B has an exception for 'soft-fail' if the SPF uses the ~all. It will allow all mail through but tags those that fail SPF.
There's absolutely no reason not to refuse mail if the SPF check fails.
Remember, an incorrect SPF will result in a lot of mail lost, and it won't take many minutes before this is noticed. The fastest way to fix it is not to contact every single communication peer and have them bypass their SPF check; it is of course to just fix the SPF record. A 10-second dialogue with a competent postmaster at just one of the failed recipients should yield the reason for the fail, like the missing mail source. All that's left is to fix the SPF by adding the relevant "a:xxxxx.xxx" or "ip4:xx.xx.xx.xx" to the record.
"For every complex problem, there is a solution that is simple, neat, and wrong." -- H.L. Mencken (1880-1956) --
It's called the submission in SMTP, your company has to setup a MTA to accept authenticated TLS connections on port 587 (or any other than 25) to relay "its email domains" to the outside world via a SPF allowed host. That is a one time setup for you corporate email profile/MUA.
This. If an admin like the GP is so high and mighty about DNS records meeting RFC compliance (You do listen for DNS on both UDP and TCP right? And you've signed your domain with DNSSEC?), you can at least do your SMTP services correctly too. Asking for an authed SMTP submission session for each domain is now the correct best practice. Unauthed SMTP relays are a dying breed.
If you have correct SPF, you will most likely get more spam. Why? Someone spams mailservers with your domainname, then it bounces because of failed SPF, and many mailservers are configured wrong, so you will get backscatter spam.
The irony of this is, that SPF says "it wasn't me" and the wrong configured mailservers then send YOU an answer saying "hey, we're not accepting your mail, because your SPF says it is not your mail".
you are destroying the sense of SPF.
consider someone spams you from a faked domain (the thing spf should prevent). Then you send backscatter-spam to the real domain.
fail.
That works fine until the CEO misses an email from a prospective client.
Unless you plan to profit from stupidity, that prospective client is worthless if they can't even set up a functional SPF record. Either you're too stupid to know about SPF or you do it right. Everything else is dumb beyond reason.
Lots of people are dumb as a brick when it comes to IT. Some of these people manage mail servers and DNS servers. Some of them want to buy stuff from your company. This makes their stupid misconfiguration your problem. Much though I'd enjoy burning these morons at the stake you can't burn your customers alive and expect them to keep buying from you. ( Unless you are Microsoft. )
Don't block on SPF - Use it as part of a spam scoring system.
SPF means the sender's domain admin EXPLICITLY configured their domain to advertise a "drop messages that didn't come through X MTA".
User of that domain should be informed of this, and if important emails are lost due to bad SPF records, then it's the admin (or part of the sending organization's) fault.
Please, please don't do this. You take SPF, designed as a defensive shield for you, turn it into an offensive weapon and unleash it on an innocent victim every time you send out a "friendly" message like this. Please turn this off now and then Google "joe job" to learn how you are making the problem worse, rather than addressing it in any meaningful way.
Ignore SPF if you like, but don't use it as an method to select targets to attack with "friendly fire."