Slashdot Mirror
Classed as Spam by Large-Scale Free Email Servers?
bartle asks: "I run my own personal domain that serves all of my email needs of myself and a few friends. In general this has worked out pretty well but there's a fairly significant limitation: if I send an email to a Hotmail or Yahoo account that I've never contacted before it tends to get filed as spam. This means that if I'm writing someone out of the blue I need to send an email from a free service which kind of defeats the purpose of running ones own email server. My domain has a SPF record, the IP resolves, and it doesn't appear to be on any blacklists. I can not find any documentation on what hoops I need to jump through before Hotmail and Yahoo will consider my mail legitimate. I understand that there's a general paranoia about publishing information that could assist spammers but this attitude seems to be leaving do-it-yourselfers out in the cold. Does anybody have any ideas? Are there guidelines or protocols I can follow to make my email non-spam?"
You're on a dynamic IP address, aren't you?
:\
:P
Most major mail provider bl dynamic IP's. The way around this is to smarthost against a known, static-IP mail server. In short, smarthost outbound mail. Inbound mail is fine, and you'll be all fixed up.
Not flaming you, but any good smtp faq or mailing list would have told you this, and Sendmail's FAQ answered it for me 3 years ago.
Just wondering how we get to an ask slashdot from a simple mail administration question. Google really *would* have answered this.
I know, I know, I'm new here. Next meme.
Karma: Chameleon (mostly due to the fact that you come and go).
Most likely your IP is listed at those sites as belonging to a range given out to ISP's for reistribution to their customers. You are probably rejected for that reason, because 'normal' domestic users don't have mail servers, or so these parties seem to wrongfully think. You can configure your mail server to send out the mail through our ISP's smtp server (smarthost).
I got free gmail invites if you want. :-P
..sacrifice a goat to the Dark Lord of Remond. Offer a virgin to the Yahoo people. In no time your mail will be flowing once more and as a bonus, nice men from Nigeria will send you fantastic offers! Enjoy.
If your server is located at a hosting facility, then it is also possible that your ip resolves to a host name that is different than the one in your SMTP headers. You can ask the owners of your IP address to add your host name to rDNS.
Someone stole my old sig.
Likely it's because when the other end does a reverse DNS record lookup, and your hostname and the PTR record don't match up. Usually this ends up resulting in receiving fine but problems sending.
Try setting up your ISP's SMTP server as your outgoing mail relay. In other words, when you send mail to your SMTP server, instead of looking up the remote host, doing an MX record lookup, etc., just send to your ISP's SMTP server. They should be configured to accept anything from your IP (you are their customer after all), and it only requires one extra hop for your email on the way out the door.
Instead of:
Email client -> Your SMTP -> MX record lookup -> Destination
it becomes
Email client -> Your SMTP -> ISP SMTP -> MX record lookup -> Destination
After doing this, from your point of view, nothing will have changed, and you can learn to sysadmin on a small scale to your heart's content.
A lot easier than getting an ISP to change the PTR record to your hostname.
- I don't need to go outside, my CRT tan'll do me just fine.
Have you thought of spelling viagra as V.1@gr4?
Rather than connecting directly to Yahoo's or Hotmail gateways, use your ISP's mail relay. That's what it's there for.
http://www.justfuckinggoogleit.com/
Perhaps you have looked through your message for spaminsh words, perhaps not.
When I was tuning spamassassin on a daily basis, I got lots of surpizes related to what triggered the filters at first.
Many people put little ads in their sigs for example
We had a person who put their address in the sig. the problem was that they lived on 888 48th street (or similar address.) this looked like an 800 number to spamassassin.
If you really want to see a dumb message, have someone send you a cell phone picture mail. The did not get through our filters at all. When I white listed it, it had so much advertising for the company it addition to the bit map it was shocking.
Please post a copy of your standard email so we can see it. (#### out the to address of course)
Good luck
Some time ago, I found that my company's mail was rejected by a large provider. Similar to your issue, our mails were also being rejected without any notice. I had good luck with calling the ISP. Once I got the right person on the line, we were able to diagnose the problem with ease. In my case, it was an incorrect reverse DNS lookup record that I had overlooked, which it was easily resolved. Your domain appears to have the same problem.
You may want to see if your ISP will update your reverse lookup record to match your DNS info (unlikely if your DSL is residential). If they won't do this for you, consider upgrading to a business class service that allows you to run a mail server and gives you some control over your DNS record(s). If this turns out not to be the issue, your best bet is to contact the recipient's service provider(s). Once you get the right person on the line, you may be pleasantly surprised.
-Turkey
if I send an email to a Hotmail or Yahoo account that I've never contacted before it tends to get filed as spam.
:)
You need contacts/friends/customers with better email service. When they ask why didn't you contact me, tell them it's because of their crappy email service. Even AOL (shudder) tells senders why their email is classified as spam.
If too much legit mail gets classified as spam, Hotmail & yahoo's customers will complain, or take their business elsewhere.
Since most yahoo & hotmail users don't pay for email, it's unlikely yahoo & hotmail would pay much attention
I used to work the help desk at a hosting company, and the most frustrating part of my job was helping people whose email was being unfairly classified as spam. I was supposed to hand off those cases to the abuse desk, but that department seemed better at antagonizing people than resolving the issue. So a couple times I tried to sort out the issue on my own. (Which probably helped get me fired.) What I discovered was that network admins are a law onto themselves, and often block email for all sorts of strange reasons. The most common reason is that they place too much faith in blacklists, but it's also pretty common for them to look at traffic patterns and tell themselves: "That block is hosting a lot of spammers!" It doesn't occur to them that there are legitimate newsletter and mailing list services that have good ant-spam policies, but that send out as much email as any spammer.
Here's your basic problem: Yahoo and Hotmail have no obligation to classify your email fairly, or even accept it at all. Nor are they required to document how they decide what's spam and what isn't. (And a big incentive not to: such documentation would be very useful to spammers!) But one thing they do is respond to complaints.
So fire off some email to abuse@hotmail.com and abuse@yahoo.com and ask them what's going on. I can't guarantee that they'll give you useful help, but they might -- provided you avoid antagonizing them. A confrontational attitude is not helpful in dealing with support people. They take a lot of shit for policies they don't determine, and won't take any more than they have to.
You could also talk to the support people at the ISP that leases you that IP number. The problem probably has nothing to do with them, but you never know. Besides, you might end up talking to somebody who's smart enough to figure this out, or who has seen this problem before.
My domain has a SPF record
Maybe that's your problem?
the IP resolves, and it doesn't appear to be on any blacklists
I have my own domain, and never have an issue - the only difference between me and you is that I don't use SPF.
Try getting rid of your SPF record and see if that helps.
I htink that you are saying that the name as given in the SMTP HELO statement does not match reverse DNS.
The HELO name usually defaults to the hostname.
This implies if the guy can get his box to HELO with what the IP resolves to he will be OK.
I'm just clarifying what I think you are saying to make the next step obvious.
Sam
blog.sam.liddicott.com
Rather than connecting directly to Yahoo's or Hotmail gateways, use your ISP's mail relay. That's what it's there for.
Problems:
1. Many ISPs won't relay mail unless the "From:" is in the ISP's domain. This prevents forgeries by zombies that try to relay through with random from addresses. More importantly for the ISP, they get to use viruses and spam as an excuse to force you to use their e-mail address, making it harder for you to switch ISPs.
1a. Yes, I know about "Reply-To:." Many brain-dead mail servers, list servers,
and even e-mail clients apparently don't.
2. ISPs often have limitations on attachment size. If I want to e-mail a 9MB file to a client or family member that can't deal with passworded FTP, I don't need my ISP's mail server rejecting the e-mail.
3. ISPs often disallow attachments which are executable. Again, not a hassle when dealing with computer-savvy recipients, but not all recipients are that sharp.
4. If the ISP ends up on a blacklist, your e-mail doesn't go through to mail servers that use that blacklist. I have a much better ability to control spam going through my server than to control spam going out through my ISP's mail server.
5. You're at the mercy of the ISP. It their mail server goes down or experiences other problems, your outgoing e-mail is either lost or delayed.
6. If there are e-mail delivery problems, your server won't have useful logs (since the actual delivery was attempted by your ISP's mail server. You won't be able to tell how many times a message was retried, whether something timed out in the protocol, etc.
7. I'd rather not have my ISP retaining copies of my e-mail, auto-scanning it with who-knows-what software, passing it on to the FBI for warrantless PATRIOT Act fishing expeditions, etc. While I know that they could do that with a port 25 snoop, chances are that they wouldn't routinely do that.
Yes, I know that there are inconvenient workarounds for some of the problems listed above, but, all in all, it's far preferable to use your own server.
I know that AOL blocks most of my emails, usually direct replies to emails from customers who I then have to contact using AIM. The good part is that they usually bounce back an email letting me know when I've been blocked.
The best way to solve the problem is for end users to not rely on unreliable email providers. One false positive when spam blocking is one too many especially if they don't bounce.
Maybe it's your overly permissive SPF record? I don't pretend to understand such things, but I've locked mine down to IP addresses which I actually send mail from and have no problems.
You have "-mx all", I think what you want is a "-a a:chrisbartle.com" and then make sure that all of your mail goes outbound through 216.17.137.189.
He's trying to use what appears to be a home DSL connection to run his email server.
... even if you find a way to make it work TODAY, the spammers will pick up on it and flood those services and then the mail admins there will find a way to break it TOMORROW and you'll be right back where you are now.
DUH! That's what most of the spam zombies do.
If you don't want to be rejected because you look like them, then you must change how you look.
Either get a business account or see if you can convince your ISP to change the reverse DNS lookup on your IP address (lots of luck on that).
Remember, it is all about how you APPEAR to the receiving machine.
If you APPEAR to be a legitimate server with the correct DNS entries, HELO, etc, then your mail will most likely be accepted.
There is NOTHING you can do with your home DSL connection/server that spammers cannot also do with a zombie on such a connection. So
If this project isn't worth the additional cost of a business account, is it really worth the headaches?
chrisbartle.com resolves to 216.17.137.189
but
216.17.137.189 resolves to bartle189.dsl.frii.net
So it doesn't resolve correctly. You might think you have a "static" IP address, but it appears the same as any other, dynamic, home DSL address.
Unless you can get frii.net to change their DNS entry for you, you'll continue to have your mail rejected.
Original submitter here with a general followup response for anyone who cares or perhaps reads this later in the archives.
It looks like their are two general suggestions: route all outgoing mail through my ISP's mailserver and change my DNS records to something more reputable.
I really don't want to follow the first suggestion for a variety of reasons - it mucks up my SPF record, I can't diagnose email problems, and I can't send stupidly large emails. I suppose if all else fails I might try it; if it doesn't work I can always switch back.
The second suggestion is good and one I can probably implement in short order. I think I can talk my ISP into letting me manage my tiny block of IPs or at least changing what they resolve to. Someone pointed out that Hotmail spam script might just be looking for the term "dsl" and blocking based on that. It sounds stupid but in a very plausible way.
Thanks are extended to most everyone who responded. There is certainly a lot of material out there on running one's own mail server but very little about getting past spam filters. I hope that I'm not the only one who found this little discussion useful.
How to report a Yahoo bug
h tml
to Yahoo.
http://docs.yahoo.com/info/support/contacts/bugs.
The problem is that there are a lot of spam fighters that believe it's perfectly acceptable to block email that originates from a DSL connection.
Now you can reduce the number of sites that are able to tell you are connecting via DSL,
(Many for example only check your rDNS so if you can get that set to your domain then they'll accept your email) but there will always be some that use block lists that include your IP.
To get around this, you really need to send email through a relay.
I don't know of any free relays that aren't also considered untrustworth (that doesn't mean there aren't any), but there are plenty of pay services that will forward your email for under $5 a month.
Most ISPs provide the service as part of the general connection package, though presumably you've already considered and rejected the idea of using your ISPs mail relay for some reason.
-- should you trust authority without question?
Have your ISP alter your reverse DNS information to reflect your domain. If they won't, change over to one that will.
This sig no verb.
I manage internet for a medium sized company. They have two mail servers, one is colocated, and the other is on their T1 in the office.
Neither one has a reverse DNS that matches forward DNS. They never have any trouble sending mail to Yahoo, Hotmail or AOL.
FYI they do not have SPF records, so that is another datapoint, but I wouldn't draw any conclusions from that.
MANY companies have non-matching forward and reverse DNS, so it would be stupid to block on that criteria. Maybe some diehard slashdot anti-spammers are doing it for the their personal email, but nobody real is (or they are only counting it as a small part of a spam score).
So I don't think your problem is due to a mismatched reverse DNS. More likely it is because it is a DSL address and people have been known to block on those.
I think that you and I are in a similiar situation. I run a company from home, and ideally I would have a local server. In my case I found my connection too unreliable so I plunked down money for a 1U colo in a nearby data center, and I run all my mail from that, no problem (again, forward and reverse DNS do not match).
I think that's probably your best bet if you can afford it.
BTW the amount of failed relaying attempts we get from DSL/Cable botnets is absolutely STAGGERING. Some of the bots are quite stupid and don't bother checking failure codes, so even rejecting those emails at connection time doesn't stop them from trying. I really don't blame the free providers for trashing emails from DSL/Cable addresses.
Guys, he's got a TXT record for SPF. If he starts relaying through his ISP's mail servers, he's either got to give up SPF, or add the appropriate include: modifier. Neither is particularly thrilling.
:)
Chris, I'm assuming you want to send out only from mail.~, since you have no PTR in the SPF, right? Still might be useful to add "A" to the SPF, since mail.~ is a CNAME. Or maybe not.
And if it helps any, you seem to be clear off all the RBLs I was able to check.
you failed to note that the parent also provided a thoughtful solution (i.e. hand the mail off to your ISP's SMTP server and stop trying to deliver it directly).
Problem is that some ISPs rewrite the From: header to match the username provided in the SMTP authentication sequence. For instance, Dan is the owner of example.com or an employee thereof. Dan is supposed to send e-mail as From: dan@example.com, but when he does so through his ISP's mail server, the server rewrites it to From: dan16623@isp.net / Reply-To: dan@example.com. This would get the mail delivered, but many MUAs will add the address specified in From: (not Reply-To:) to the address book, causing things to break when the sender moves house or otherwise switches ISPs.
> ... some ISPs rewrite the From: header ...
An ISP that does this is not a provider of internet mail. So anyone that wants to use internet mail should not subscribe to such ISP. If your contract specifies that email service is included you should ask for the money back. Internet mail has quite precise definitions in RFC 2821 and RFC 2822, and rewriting the "From" header is not part of this. It actually contradicts RFC2822 that says the "From:" identifies the author (and not the link from the author's PC to the mail server).
The real reason why "ask slashdot" is better than "search google" is that we are also discussing current pracices and contemplating what's right and what's wrong.
Rewriting headers is wrong, unless the user specifically asks for this service. And if an ISP wants to reflect the userid of the sending user for some reason they can use one of the "resent-..." headers fields (though I doubt that this would be correct RFC-wise, and morraly its wrong. If they need to put a stamp on outgoing email to help their abuse team locate offending users they should put it in an encrypted form in a non standard header, like the "X-sasl-enc:" header used by FastMail.FM on all outgoing name. Every single piece of outgoing mail has unique identification without compromising the sender's privacy).
I can tell you what's Hotmail reply would be: the same form every one gets: "Delete your Temporary Internet files". Doesn't matter what you ask... Then later they might answer to the point, but I doubt that they would whitelist an IP address the resolves to a broadband connection.
There might be very good reasons to run your own mail server on your own PC, but I don't see any advantage on using this approach for direct delivery of outgoing email. Relaying through your ISP's mail server would almost certainly be more reliable, and if not it is a good indication that you should switch ISPs. Probably using an ISP's server for relaying incoming email would also be more reliable (it would cache your email uwhen your PC is not available on broadband) but it might be trickier to set up. (as in your ISP's mail server would have to be the MX for your domain, and would have to be configured to accept all email for your domain and redirect it to your server, which would not allow you to do SMTP session rejects. There are no such complications I can think of for relaying your outgoing email through your ISP if they are following standards and not using dirty tricks like header rewriting.)
> Maybe it's your overly permissive SPF record? ... You have "-mx all" ...
SPF specifically says that the result of the SPF test is "pass" in this case. The same for SenderID when comparing to the "purported whatever(TM)". The problem with SPF is that people want to use the record but not follow the standard. Instead of checking if the email comes from an IP address authorised by the domain owner, people want to limit what domain owners can authorise.
This is a very bad side effect of SPF, and eventually it would mean that all email users that are not limited to a single or a few sending IP addresses would have to relay their email using authenticated SMTP to one of a few servers of a service provider that would be listed in the SPF record for their domain. SPF claims to not require huge changes in infrastructure, but its side effects do require a change in infrastructure that collectively would cost a lot. Perhaps it's not bad, but it's also not cheap... (at least not as cheap as running everything on your home PC).
So anyone that wants to use internet mail should not subscribe to such ISP.
In other words, in some geographic areas, "anyone that wants to use internet mail should not subscribe to" high-speed Internet access at all because the only broadband provider (the cable company, or the DSL company that does all it can to interfere with Speakeasy's attempts to provide service) does not provide conforming internet mail service to residential customers. How much does SMTP AUTH + TLS smarthosting cost per month?
Really, well, at least I thought it was. How do you know your not helping out a spammer?
/sarcasm
How do you know your not helping out a TERRORIST
eek.
If you can't laugh at spam, what can you laugh @?
Besides, there are practical reasons to have your own email server. Like being able to reconfigure the thing as you need to. For example, you might have noticed that he gave his email address as slashdot@chrisbartle.com. That's obviously one that he created for the occasion and will discard as soon as the conversation is over. Can't do stuff like that with most ISPs.
One thing Chris might consider doing is moving his email server to colocated or virtual host. A suggestion which will probably not please him, since that's something he has to pay for. But it's probably the only way to get off the email providers' shitlists. Of course, he should make sure that the hosting provider leases him an IP address that isn't on any blacklists...
Just get sendmail to relay via your ISP's mail server - no loss of DIY functionality, but the ISPs server will (should!) be whitelisted. George
Most of this discussion has focussed on the envelope, but maybe its the content. My spam filter (popfile) tends to classify html mail as spam, for example. When you send plain text do you get classified as spam?
Intron: the portion of DNA which expresses nothing useful.
In case anyone is interested in what I tried...
I changed out my IP names so they now resolve to something sensible. My email was still seen by Hotmail as spam and on a lark I tried something different: I sent an email from Pine instead of Outlook Express and it went through.
I've tried this a second time. I sent an identical piece of mail from Outlook Express and from Pine, same server, same everything else, and the email from Pine makes it through while the one from Outlook Express is marked as spam. Presumably there's some fuzzy filtering going on because sending from OE at work to Hotmail does through just fine.
I'm still not happy with this situation. I know how to get my email through Hotmail, for now, but there's no guarantee that it will work that way tomorrow or the day after.