Slashdot Mirror
Gmail, SPF, and Broken Email Forwarding?
alek writes "I recently stopped getting Email from a friend ... which turns out to be related to his use of SPF records and my forwarding to gmail. This 'lost Email problem' may get worse with
Google implementing Domain Keys." Alek is looking for a non-complicated solution to this non-trivial problem; read on below for more details.
"Background: Like many people, I have me@mydomain.com as my public facing Email address. When Email comes into my server, I forward it to me@gmail.com. But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for friend@frienddomain.com, gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server. Please note that this is exactly what SPF is designed to prevent — spammers from sending Emails with your address — but it breaks forwarding and has other problems.
What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from friend@frienddomain.com comes in, and is forwarded to gmail ... with an "OK" as the response — i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears — it's not even in my gmail spam filter ... so there is no trace of it at all. If my friend sends directly to me@gmail.com, it shows up ... since his domain sends directly and the SPF test is passed. Note that on my gmail account, I associate me@mydomain.com with my me@gmail.com account ... so perhaps there should be a recipient test applied before SPF is tested on the sender ... although this arguably defeats the purpose of SPF.
The logical solution is to configure sendmail on my server to do Sender Rewriting — anyone have an easy FAQ to do this? But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"
What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from friend@frienddomain.com comes in, and is forwarded to gmail ... with an "OK" as the response — i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears — it's not even in my gmail spam filter ... so there is no trace of it at all. If my friend sends directly to me@gmail.com, it shows up ... since his domain sends directly and the SPF test is passed. Note that on my gmail account, I associate me@mydomain.com with my me@gmail.com account ... so perhaps there should be a recipient test applied before SPF is tested on the sender ... although this arguably defeats the purpose of SPF.
The logical solution is to configure sendmail on my server to do Sender Rewriting — anyone have an easy FAQ to do this? But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"
I prefer SPF 60. It allows me to keep the pasty white, computer nerd complexion that drives the women wild.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
The Internet Assigned Numbers Authority (IANA) also currently has the following second level domain names reserved which can be used as examples.
Shutting down free speech with violence isn't fighting fascism. It IS fascism!
Yes, of course. Have all your email sent to Google in the first place! You don't have to switch everything over to the Google app tool, you can just set MX records for your domain pointing to them, and collect it all (or forward it inside or outside Google.) It's free (with a paid version available.) Check it out here http://www.google.com/a/help/intl/en/index.html
Effective spam filtering for forwarded email is pretty much impossible, as you lose vital information in the forwarding. Either get rid of your forwarding address, or have it hosted at Google as well. Probably the largest single reduction in spam I've ever made was the week that I got rid of years-old forwarding addresses. If the forwarding address is more important, just get it hosted at Google directly, or tell people to stop using it!
Matthew @ Bytemark Hosting
What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from friend@frienddomain.com comes in, and is forwarded to gmail ... with an "OK" as the response -- i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears -- it's not even in my gmail spam filter ... so there is no trace of it at all.
While the RFCs specify that an MTA that is dropping should notify the sender in various ways, modern MTAs often violate these parts of the spec, pretending to accept and then dropping the mail and/or failing to send bounce notifications.
This is deliberate. Not sending bounce messages reduces the load on the servers and net (now that most mail traffic bounces). Pretending to accept mail which is actually dropped is a defense against guessing email addresses and probing filters to see what gets past them.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Doesn't GMail offer the ability to fetch your email from POP accounts now? It would probably not be the ideal solution, but perhaps you should stop forwarding and instead start POPping.
Editor Emeritus and Senior Writer, TeleRead.org
Domain Keys authenticates that the message was generated by a server with access to the DK private key. Forwarding the message does not affect the originator of the message, so the Domain Key authentication still checks out.
SPF and DKs solve similar issues, but in a much different manner.
Sign up for Google Apps, and then you can have all mail sent to me@mydomain.com be handled by GMail. All you have to do is sign up at http://www.google.com/a/ and link your domain. Then point your domain's MX records to aspmx.l.google.com.
In the future, all you have to do in order to get your mail is to go to http://mail.google.com/a/mydomain.com/ instead of http://www.gmail.com (and you can even set it up so that http://mail.mydomain.com CNAMES to your email login page)
There's actually a fairly simple procmail fix right on the spf site: http://www.openspf.org/FAQ/Forwarding
This is also known as, "The Problem With SPF." SPF breaks forwarding. This is well known. People who use SPF need to be aware of the ramifications.
The SPF people have created SRS, as you are aware, to work around this problem. It is a complicated and unappealing workaround. I certainly won't do it.
You have three options as I see it:
1) Stop forwarding. It's really a terrible idea. Install webmail on your mailserver. Check out RoundCube, for instance.
2) Wait for people to figure out that strict SPF policies break SMTP too badly for most users.
3) Implement SRS. (this would probably be easier if you were using a modern MTA)
I guess you were hoping for an easy fix, but there simply isn't one.
I put SPF on my domain not because I think that it'll solve the world's spam problem, but because it helps reduce the (large) number of bogus returns that come back to my domain (the more recipients that have SPF checking on, and realize that some sender in China isn't a legitimate source for emails from my domain, eats and discards the message rather than bouncing back some wasteful return spam to me).
SPF is great. It isn't a total solution, and there are negatives, but it certainly is better than the anyone is anyone free for all.
That's outstandingly unhelpful. How about attaching a link to a decent SRS implementation? Or sending them to OpenSPF?
Randomly throwing down on people legitimately asking for some technical help is a big problem in the OSS community. Whether or not /. is the appropriate place to ask this question is debatable, but since it made the front page and there is no helpful SRS faq on this site, might as well direct them somewhere.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
Another satisfied google hosted apps customer chiming in. I have a reseller webhosting account that I keep about 10-15 domains on for myself/friends/family which does acceptable e-mail, but I advise everyone to just shove their e-mail over to gmail/a instead.
You get your own hosted mail/webmail service with (currently) 7gb of storage per/account, no preset account limit, POP and IMAP, as well as great spam-filtering.
All free.
And for $50/acct/year you can have 25gb/acct storage, API access to customize it for single-signon and/or gateways, a full Postini implementation, and 99.9% uptime guarantee.
Hate to sound like a shill, but it's a fantastic service and I don't mind pimping it.
"You worthless post!"
-Shakespeare, 2 Gentlemen of Verona, 1. 1. 147
Comment removed based on user account deletion
At first I was wondering why they hell someone that had a working email server would shuttle it through Gmail, but then I read about using the spam filters, etc.
While that sounds good on the surface, is anyone out there not a little apprehensive about having all your email, particularly if you're a business, going through and being stored on their servers? I mean, someday Google will bend completely for govt. wanting to search all emails for 'terrorists' activities, and God knows who else will too.
I guess I'd want a bit more privacy on my emails, especially if they contained sensitive or proprietary information. I know...they're in plain text and could be intercepted if not encrypted, but, this is altogether different. It is stored on google's servers and there for easy data mining.
I'm getting ready to dig out my old email server post Katrina...can you not use procmail and spamassassin to filter spam as effectively as Gmail does?
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
For God's sake. It's just text! RFC 2606 doesn't specify what you're allowed to write in a text message.
If you're actually going to do some testing then it might matter. What matters here is can the reader understand the question. I can. Can you?
I think you're looking for whoosh.
If you are worried that your "sensitive" email could be stored and eventually used against you:
1) stop using email altogether.
2) you need to get to a drug rehab center... cocaine is a hell of a drug
My e-mail goes through my domain, forwarded to Gmail, and then is downloaded to my computer via POP. Gmail is my offsite back-up (that is accessible from anywhere) and home is where I do most of my mail viewing/sending. All of those GB of space, local copies in case Gmail fails, remote copies in case my computer fails. And assuming Google is "not evil", then I should be ok.
Layne
SPF will validate the Return-path header if there is one instead of the From: address.
Unfortunately, I don't know how to make either sendmail or postfix insert a return path when they forward an e-mail, but the easy work around is to install mail list software as your forwarder. You can create a mailing list as your incoming e-mail, with only 1 mail list member, (which is your g-mail account). Mail list software will automagically insert the appropriate return-path header that is needed in this case.
DKIM and DomainKeys work in a fundamentally different way. The message is SIGNED. Hosts are not indicated one way or the other. So any DKIM signed mail can transit any number of hosts provided they don't modify the signed sections.
SPF has no such luxury unless implemented in a much more advanced manner in terms of the senders publishing. And it's not GMail's fault for following the SPF records as published, they should do a better job of rejecting early rather than just /dev/null-ing the email though.
People over-generalize terms quite often, and "forwarding" has different meanings in different situations. Generally the difference boils down to if you're talking about a "server" implementation or a "mail client" implementation.
In this case, the SPF folks are addressing server admins, so by "forwarding" they mean sending the message to a new recipient without altering the headers. This use problably originates back to the old ".forward" files on unix machines, but may go back further. Most server-side implementations use this meaning for "forward".
However, forwarding by hitting the "forward" button most mail clients does something different. That creates a new message with new headers and preserves the old body text. sending with the same headers is called "redirect" in most mail clients.
Isn't it great how mail clients and mail servers use different meanings for the same word?
Even the client/server pair that go together from the same company have this problem. For example, Microsoft - exchange server has forwarding contacts, which forward without header changes, while Outlook clients do change the headers when you hit the "forward" button.
-Matt
For this example, I'm assuming that your email is joe@example.com and your gmail address is joe-example@gmail.com.
Create an alias (/etc/mail/aliases) for the address that get's forwarded to gmail.
joe: joe-example@gmail.com
Also create an alias for <foo>-owner:
joe-owner: joe
Sendmail will look for this special <foo>-owner alias whenever sending mail to the <foo> alias, and use it as the envelope sender on the outgoing mail. So any mail that is sent to joe@example.com will be resent by sendmail with a sender address of joe@example.com. The header addresses will remain unchanged, so hitting reply will still go to the right person.
Is this the solution to all SPF forwarding brokeness? Of course not, but it's a surpisingly simple solution to a number of common forwarding situation. Note that you better be careful about spam filtering on your machine, or your mail server (your sender's address) will appear to Google as a source of spam, and might get filtered.
Short answer is, no. Google's large amount of incoming email, their patented algorithms, and the huge data mine they're sitting on give them a unique ability to provide very through and high-quality spam filtering.
Of course, that isn't to say that one can't do a half decent job with spamassassin, it just won't be as good as Google's filter.
Regards,
Joe Example
Since you are running your own SMTP server, you signed on to be a sysadmin. I am replying to you as a fellow sysadmin and I'll give sysadmin-style answers. Please don't take my response to be negative in any way, as I'm trying to help.
The logical solution is to configure sendmail on my server to do Sender Rewriting -- anyone have an easy FAQ to do this?
If you follow the link that you just gave for Sender Rewriting, it answers your question. "Implementation" links to modules, source, and configurations.
But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"
I say that you don't know how many people are implementing SRS, nor do you know how many forward e-mail to Gmail. Let's stick to the basics before giving up so readily. I take it that you absolutely do not want to give up carte blanche forwarding from your own SMTP server to Gmail; so I'll tailor my reply to that.
But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for friend@frienddomain.com, gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server.
Your friend has published an SPF record because he doesn't want people forging his domain in the envelope-sender field. This is a common spam tactic that ruins the reputation of someone's domain, either through spammer apathy or sometimes pure malice. Your e-mail forwarding (especially since you run your own SMTP) to Gmail is out of pure convenience to you and is unnecessary, so don't ask your friend to drop his SPF record.
There are two ways to solve this:
1) Have your friend add your SMTP server to his SPF record.
2) Implement SRS if you want to solve it once and for all. If you follow your own links, there are explanations, examples, and actual code. You haven't said which SMTP server you're running, so you've limited the responses people can give you for your situation.
I publish SPF records for my domains. There isn't anything "broken" about wanting to protect my domains' reputations from forgery. Very few people have a problem with forwarding that they didn't create themselves. This exception I'm talking about is people who have old university accounts (or similar) which only allow e-mail checking through a shell account and forwarding purely through a ".forward" file (or similar), with no POP, IMAP, or administrative access. This is not you. But for anyone who this describes, because of the draconian service policies, they shouldn't be giving out that e-mail address to new contacts, publish on papers, etc.
My SMTP server checks SPF, but not DK. With SPF, the forged domains are instantly rejected, requiring minimal overhead. DK requires reception of the entire message (because the headers are in the DATA phase) in order to validate the message, on every message -- this uses unnecessary network bandwidth, and it places an extra load on my system since it would have to calculate and verify signatures for every single message. Maybe that's not an issue for you if you only receive a handful a day, but I receive thousands. Spammers know that including fake DK info in a message and then sending millions of these is effectively a Denial of Service attack on the servers that indiscriminately check DK signatures.
I also use backup relays. For the relays that are not under my control and don't implement SRS, I simply bypass SPF checks from those IP addresses.
About Google silently dropping your e-mail: Keep in mind that with your carte blanche forwarding, you're also forwarding spam. You are essentially spamming Gmail, even though it is you simply forwarding e-mail to your own account. It is difficult for Google to know this without human intervention or implementing some co
One very good reason not to have your email address @gmail.com, if you are using it for your business, is that a LOT of businesses, wholesale vendors, even the federal government will not accept an @gmail.com address because of the large number of frauds associated with free email accounts (not just gmail, but also hotmail, yahoo mail, etc.) For example, this last tax season the federal govenment would not accept a gmail account for notification of your tax return status when filing electronically.
It is much better from a business standpoint to have your own domain and email sent to your domain. If your MX points at gmail, that's okay. Just don't make your email address me@gmail.com if you want to be taken seriously.
Of course it won't stop spam. It wasn't designed to. Its purpose is to stop joe jobs.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
Have your friend look up the SPF records for a bunch of big domains. He'll notice that most of them use "~all" - a SoftFail - which is accepted by Gmail. He's probably using "-all," which makes the message just drop. The only examples I've seen of SPF hardfails in the wild are from banks. However, loads of domains are using softfail - Facebook, Google, Microsoft, eBay, MIT, UC Berkeley - to name a few.
I have a gmail account, I get a handful of spam a week slipping through. I don't ever advertise my gmail account, however it's a common enough username with no numbers so dictionary attacks would hit it.
I have a private email server, with clamav running spamassassin and postfix tuned to prevent spam (Simple settings really), I get even less spam than my gmail. This address has been published for years on multiple websites, I use for just about everything, in cleartext on websites that are spidered.
In my experience you can do just as well or better than gmail without any headaches and a simple setup. Expect a few hours initial setup, and maybe an hour every 6 months to check if you're missing something the auto-updates can't update. It's been like this for a few years so far.
The reasons I still use them are that I think the quality and utility outweigh the risk, and because my much-smaller web hosting company is more likely to do something bird-brained than Google is.
That's actually a foolish remark. Use google to search for things like "gmail outage" or "gmail issue". My favorite is "gmail security issue" with over 100k results.
I've heard stories personally about people logging into gmail and ending up in someone else's Inbox. Yes, that's right, full access to someone else's email. Or how about another goodie: mass deletes of random emails.
I don't understand why people have the idea that Google is better then competent system administrators - it's just plain foolish and naive.
Regards,
Website Hosting