Hotmail & Yahoo Mail Using Secret Domain Blacklist

Frequent contributor Bennett Haselton writes: "Hotmail and Yahoo Mail are apparently sharing a secret blacklist of domain names such that any mention of these domains will cause a message to be bounced back to the sender as spam. I found out about this because — surprise! — some of my new proxy site domains ended up on the blacklist. Hotmail and Yahoo are stonewalling, but here's what I've dug up so far — and why you should care." Read on for much more on how Bennett figured out what's going on, and why it's a hard problem to solve.

On December 7th I sent out a normal batch of emails to the Circumventor mailing list, where I send out new proxy sites for getting around Internet filters. I registered seven new domains and sent each domain to one seventh of the list; the list contains about 420,000 addresses, so each one went to about 60,000 people. (Each new site is only sent to a random subset of the list, so that a blocking company can't just subscribe one address to the list and block all new sites as soon as they're mailed out.)

The list is also comprised of 100%-verified-opt-in addresses, meaning that a new subscriber has to reply to a confirmation message in order to be added to the list. That's considered the gold standard for responsible mailing, but major email providers keep finding new ways to block the emails as "spam," which sometimes provide interesting insights into how the filters work behind the scenes.

After the last mailing, for example, all of my newly registered domains got disabled by the registrar because two of the domains had been incorrectly blacklisted by the Spamhaus Domain Block List. It took two days to discover the problem and then several hours to trace the problem to Spamhaus, although once I found Spamhaus's automated form I was able to get the domains un-blacklisted immediately. So the registrar re-enabled the domains a few hours later, although the traffic to the domains never returned to its previous levels. Spamhaus, meanwhile, continues to claim the DBL is a "zero false-positive" list, and has yet to acknowledge the error or contact me to help get to the bottom of how it happened. Well, they know how to reach me.

At least this time around, my domains didn't get disabled. Instead, the messages rolled out for a few hours with no problem (replies from users indicated that at least some hotmail.com and yahoo.com users were receiving them), until bounces abruptly started coming in from hotmail.com and yahoo.com addresses saying:

----- Transcript of session follows -----
... while talking to mta5.am0.yahoodns.net.:
>>> DATA
<<< 550 Message Contains SPAM Content
554 5.0.0 Service unavailable

After pummeling my address with bounce messages (to the point where my own Gmail account started bouncing because it was getting hammered with so many bounce messages from Hotmail and Yahoo), when the dust finally settled, I tried reproducing the error by sending test messages from my server's IP address to a test Hotmail account. It turns out that out of the seven different URLs that I had been mailing to our users, four of the domains in those URLs would generate a "550 Message Contains SPAM Content" error when sent from my IP to a Hotmail address, and the other three did not. The message didn't have to contain the banned domain in the From: address; the message would get blocked if it even mentioned the domain anywhere in the message body. (This only happened when sending from my own IP address at peacefire.org. It didn't happen if I tried sending a message from my Gmail account to a Hotmail address, even if the message contained one of the four banned domain names, so the issue probably won't reproduce if you try sending a test message yourself.)

But interestingly, Yahoo Mail started bouncing my messages at about the same time — out of the seven domain names, the same four domain names were being bounced by Yahoo Mail as by Hotmail, also with the error "550 Message Contains SPAM Content." That's far too unlikely to be a coincidence, so it looks as if Hotmail and Yahoo Mail are using a common secret blacklist of domain names that cause a message to be blocked as spam. (As it happens, the other three domains were also being bounced by Yahoo Mail with the error "Message Contains SUSPECT Content" — as opposed to "SPAM Content" — while those three domains were not blocked by Hotmail at all. That of course is aggravating, but the real clue lies in the fact that both Yahoo Mail and Hotmail were giving "SPAM Content" errors to the exact same subset of domains.)

I don't want to publish the list of all seven domain names here, so as not to make it too easy for censorware companies to block them all, but one of the four blacklisted domains was 'golflanding.com.' (All of the new domains I register are nonsensical two-word combinations, since those are the only .com domains that are likely to be (1) still available and (2) easy to remember.) As soon as it seemed like Hotmail and Yahoo Mail were working off of a common blacklist, I checked to see if Spamhaus had screwed up again and listed our domains, but none of the seven domains were on Spamhaus's lists.

I looked up golflanding.com on the blacklistalert.org service, which checks against all major spam blacklists, but no hits were listed there either (except for on some defunct services which haven't been updated in years).

So if Hotmail and Yahoo Mail are both using the domain blacklist, perhaps it's a list compiled by one company and then licensed to the other, or perhaps it's a third-party list not widely known to the public. (Hotmail uses their own SmartScreen filter, but I've found nothing online about Yahoo using it as well.) It's conceivable that one or more of the domains might have gotten blacklisted as a result of Hotmail or Yahoo users clicking their "This is spam" button. However, Hotmail allows newsletter publishers to view data about what percent of their messages to Hotmail users are being flagged by users as "spam," and when I looked up the stats for our IP, they showed a "complaint rate" of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list). Assuming that the complaint rates are similar for Yahoo Mail, it's unlikely that the domains got blacklisted as a result of user complaints, unless the blacklist trigger has a ridiculously low complaint threshold.

Neither the Hotmail postmaster site nor the Yahoo postmaster site mention anything about a list of domain names that could cause a message to be blocked for mentioning the domains in the message body. Yahoo Mail does provide a support form for newsletter publishers to send inquiries about why their mail is being blocked; I submitted that on Saturday and started a thread with email "support," although so far their response has just been to copy and paste articles from the Postmaster site, with tips like "Send email only to those that want it." Each time, I reply saying, No, this is not the problem, the problem is that the domains in the messages are getting incorrectly blacklisted, and each time, support cheerfully sends me another article. If I'm not literally talking to a bot, I might as well be.

I opened a similar ticket with Hotmail, and they sent me a form letter saying that the emails were being blocked because of SmartScreen, and that as a matter of policy, they would refuse to fix any errors being made by the SmartScreen filter. Waiting to see if I get a reply from a human next.

So why should you care? Well, for one thing, if you care about users in China and Iran being able to receive proxies to get around their Internet blockers, right now Hotmail and Yahoo are thwarting these proxies more effectively than those countries' own censors are. Yes, these are real people who really do write back to me after a mailing goes out, telling me about how they were able to use the proxies to receive banned political information, and sometimes how long the proxy lasted before the censors blocked it. This week, they had to do without.

But more importantly, this is an example of a general problem: That there are certain types of issues, like blocking of legitimate mail by spam filters, where the "free market" does not deliver the best experience to consumers, and the costs get passed on to everybody. Sometimes the problems could be solved with some effort, but the effort does not get made, because people believe that the free market will solve the problem, or that it already has.

In theory, if consumers have enough information about different companies and their services, the companies can compete to provide the best product to users. The problem is that if one type of information is systematically hidden from users — in this case, the fact that their mail provider is blocking mails from reaching them — then the "theory" falls apart. Since spam getting into your inbox is a visible problem, but missed email messages are an invisible problem, Hotmail's incentive is not to give the user the best experience, but rather to err on the side of blocking legitimate messages — even if the user might prefer to get slightly more spam, than to miss one important email that they were waiting for.

This means we're not just talking about a few messages getting caught in filters, which could happen even in an efficient marketplace. We're talking about a permanent equilibrium where the user gets a sub-par experience by default — a trade-off that causes them to miss more messages than they want to — and senders have to pay the cost of overcoming the marketplace inefficiencies. (Which means if the sender is a business you buy from or a charity you support, the costs get passed on to you.)

Pretty much the entire financial cost of sending email, is attributable to the failure of the "free market" to motivate email providers to deliver non-spam emails into their user's inboxes. If a company or organization uses an email list hosting company like AWeber or Constant Contact to email their users, they pay a fee of about $1 per month for every 100 users on their list (which would run me about $4,000 per month). That fee doesn't go towards bandwidth — even a 1-million-subscriber list, emailed once a month, would use less than 3 GB per month of bandwidth, which is what GeoCities was was giving away for free 10 years ago. What you're paying for is the fact that AWeber and Constant Contact have friends in the right places at Hotmail, Yahoo, and Gmail, so if your mails are getting blocked, they know the people to call to fix the problem. If you run your own list instead of paying a hosting fee to AWeber or Constant Contact, you'll end up paying other costs indirectly, through loss of income when your messages don't reach recipients, or in time and money spent trying to fix the issue. (I have to take this option anyway, since I send different URLs to different random subsets of my list, which is not supported by AWeber or Constant Contact.)

On the other hand, if the market actually "worked" — if email providers did reliably deliver non-spam messages to their users — a company or charity could run their own list for virtually zero cost, and would be able to keep all of that money. (I incur no up-front fees for running my own list; all of the costs are the time spent trying to get Yahoo, Gmail, and Hotmail to stop blocking it.) So every time you donate to a charity or buy from an online retailer, a little bit of that money goes towards the cost of that organization having to fight past marketplace failures in order to get their email to you.

I don't think there's an easy algorithmic solution, like crowdsourcing Facebook complaints or using random-sample voting on Digg. Generally, I just think we need more awareness of the fact that, under certain conditions (including those surrounding email deliverability), the "free market" is virtually guaranteed to arrive at a non-optimal solution. One manifestation of that awareness would be if Hotmail, Yahoo Mail, and Gmail created public points of contact where legitimate email publishers could find out why their emails were blocked, and had real humans responding to the messages and fixing the problems. By default, the imperfect information in the marketplace leads toward an equilibrium that errs on the side of blocking too much legitimate email, so anything that pushes the equilibrium back towards more legitimate messages getting delivered will improve the experience for users and lower costs for senders.

Besides, there's a more basic ethical issue here. If you're Hotmail and you tell your users that you're providing them with "email accounts," then those users expect those accounts to work — including having the ability to receive mails from mailing lists that they've signed up for. Helping legitimate emails get through to users is not just a matter of addressing a marketplace inefficiency, it's a matter of honesty.

Larry Lessig's book "Code is Law" describes how default choices built into the architecture of the Internet and other environments — the "code" — can steer our behavior in ways that we might not choose otherwise. I'm making essentially the same point in saying that some problems are not fixed by market forces, because people are not aware of the problem at all. I think the evidence and the reasoning are straightforward in this case, but it's hard to convince people who have adopted it as an axiom that whatever the free market arrives at, must be the solution. My favorite single sentence in Lessig's book was, "Put your Ayn Rand away." I could imagine the years of pushing against dogmatic fanaticism that led him to write that sentence, and I knew how he felt.

Comments

Summary

[sorensenbill]

Is there a summary of the summary available?

Re:Summary

[TheMMaster]

According to TFA his list is opt-in only, so unless he's lying about that he doesn't appear to be a spammer.

I've had similar experiences with Spamhaus btw, they decided to nix my upstream provider and when I complained I was told that I should use another ISP because mine wasn't well liked.

I can assure you I have never sent a single spam email in my life.

This is the whole point of TFA though, there's no incentive for companies running mail services to ensure that legitimate mail gets delivered. It's simply cheaper to not bother with false positives at all because the cost of non-delivery is placed squarely on the shoulders of the sender.
This is why Spamhaus could easily force me to switch ISPs, it doesn't cost them anything to put my IP range on a shitlist, but it cost me money and effort to migrate my service.

Re:Summary

[TubeSteak]

Is there a summary of the summary available?

We call them "titles"
Here's one example: Hotmail & Yahoo Mail Using Secret Domain Blacklist

Re:Summary

[preaction]

No, but I can summarize the summary of the summary: People are a problem.

Re:Summary

[AlphaWolf_HK]

That's "hear hear", as in "hear him, hear him!" (which is where that phrase is rooted.)

Re: Summary

[Urza9814]

As a long-time subscriber to his list (at least 6 years), no, he's absolutely not. He provides a fantastic service and does a damn good job of ensuring only those who want the messages are receiving them. And I get less than one message per month from that list. If he's a spammer, so is literally every single person or organization that has ever sent me an email.

Re:Summary

[Anonymous Coward]

"Opt-in" covers multiple sins. Sometimes, it's a genuine "I want you to send me this stuff". Sometimes, it's "enter your email address to have a chance to win a magic unicorn. By the way, we'll also send you some emails", and sometimes it's an "opt-in" with the box checked by default, the way that everyone tries to get you to install browser toolbars or demo antivirus products.

Then you get people, who generally get sent marketing junk from any number of companies that they may or may not remember having dealt with. Typically, people don't go to any effort to try to unsubscribe from these lists, but just hit th e"report as spam" button because they don't want to read it.

Re:Summary

[TheRealMindChild]

Spamhaus != 0 false positives. This guy sends the same email out to tens of thousands of people who tend to use Yahoo or Hotmail. They both block the messages as spam.

Just FYI, I seen this guy bitching about it MONTHS ago. Apparently he still hasn't made a lot of headway. However, if you operate like a spammer (sending the same email to multitudes of folks, while relaying information about open proxy servers as information), then you will be treated like a spammer

Re:Summary

[afidel]

Why? By definition he is NOT a spammer since his messages are neither unsolicited nor commercial. It should be fairly easy for the responsible parties to verify he following best practices and whitelist him but apparently that's too much work for the postmasters at the big 3 webmail providers. Basically the postmasters at yahoo, gmail, and hotmail aren't doing their jobs. I know if our email admin was so bad at rectifying false positives he wouldn't be here for long but because of the scale of these organizations that pressure isn't happening.

Re:Summary

[girlintraining]

I've had similar experiences with Spamhaus btw, they decided to nix my upstream provider and when I complained I was told that I should use another ISP because mine wasn't well liked.

I've had problems like that with them as well. The thing is, Google et al. do provide very good spam filters. Out of the thousand or so spam messages that hit my mailbox every month, only about 5 make it through. A 99.95% success rate is nothing to sneeze at, so credit where credit is due. But the problem here is still architectural -- very few people respond to spam so the odds are very high that responses are to legitimate e-mail. Higher, I would think, than the 99.95% rate above. Multiple responses to the same address should override any spam-rating system they have automatically, and if not, there should at least be a 'white list' option for users to bypass the filter in the event of a failure such as this.

Neither option exists, and there is no remediation pathway available. The author (correctly) concludes this is deliberate and not merely a process oversight. Such is the nature of operations where the profit margins are so tiny that any support would obliterate it. Google only provides gmail so it can mine keywords and phrases from your e-mails to build a marketing profile and then target advertisements at you. Despite the very low rate of success here, it still beats the cost of the hardware maintenance and bandwidth when aggregated over a few hundred million regular users. But the only support incentive here is customer retention, and the support provided is very minimal and highly automated (as the author has discovered). This guy isn't a google customer -- he's trying to contact google customers, which places him in the "liability" column, not the "asset" column. Unless this guy can show that hundreds of thousands of Google customers are impacted and the impact is severe enough for them to switch, or consider switching, to another provider, there is no incentive for Google to even read his complaint, no matter how justified or rational, or easy to fix.

That's the free market problem he's run into: He thinks he's a customer, but he isn't. He's a service. And one that costs google more to support than any potential revenue that may be generated. The business decision here is clear, if not very friendly.

Re:Summary

[fifedrum]

yeah: guy discovers cloudmark domain blacklist is used by two cloudmark customers. At least, that's my opinion. this information isn't new, this list has been around for years, and you don't get on it easily. It takes multiple reports from multiple accounts before they add you.

Re:Summary

[jdavidb]

I've had similar experiences with Spamhaus btw, they decided to nix my upstream provider and when I complained I was told that I should use another ISP because mine wasn't well liked.

"Wasn't well liked" == "complaints had been received that they allowed their customers to send spam."

I agree with spamhaus. This puts pressure on ISPs to police their customers, or else their decent customers will leave. And everyone can choose whether they want to use providers that allow all contact through, or providers that filter out contact from ISPs that don't police their customers.

there's no incentive for companies running mail services to ensure that legitimate mail gets delivered

Well, there's some incentive in that if their customers truly want the mail and aren't receiving it, they'll have to pick a different provider. I purchased a product once to be emailed to me and had to acquire an alternative email address because the seller wouldn't do business with gmail, yahoo, or hotmail addresses. I didn't waste time arguing with him; I just got an email account that would get his mail through.

it cost me money and effort to migrate my service.

That's the price of offering a service. If enough people want it, they will more than make up for the cost of you going with an ISP they consider reputable. If not, the world has no obligation to keep your costs low enough to keep you in business. A much cheaper thing to do would've been to quit offering your service.

Re:Summary

[TheRealMindChild]

As a consumer of email, I would rather the 1% find a better way to communicate rather than stupify the email system even more to accommodate them

Re:Summary

[SomePgmr]

I'm not on his list so I don't know how it actually operates, but there's a lot of variation in these things. I've opted in to lists that don't provide a real opt-out, or in other ways don't comply with the old can-spam guidelines. I usually see this from foreign companies, which will require me to log in to an account on their website to get off the mailing list. Those get the spam button, every time, no questions asked.

Spamhaus does BL domains for no apparent reason, though. We're talking about properly configured mail servers, no open relays, no backscatter, appropriate DNS, with opt-in recipients only and working, simple unsub options right in every email. It is not a perfect service. My experience correlates with his on this though, in that you can have yourself removed pretty easily the first time.

This all ignores a pretty simple issue, though. it's easy to be too optimistic about the reliability of email delivery. It has never been great, there are no simple (and free) solutions, and things like this are going to happen. It's not because the providers are evil or conspiring to keep you from getting your job done, but because they're trying to make an implicit trust system usable over the internet.

And unfortunately, smtp implementations are not going to change in a way that fix all the present shortcomings.

Re:Summary

[afidel]

Why? Listservs are older than SMTP and have always been one of the use cases for electronic communications. Plus it's not like those providers are blocking all listservs, just those that don't pay their friends stupid high monthly fees for the privileged of emailing their users.

Re:Summary

[genner]

Is there a summary of the summary available?

It's a pain in the butt to get yourself removed from yahoo's email blacklist.
The end.

Re:Summary

[Onymous Coward]

Is the Cloudmark list of MTAs or URIs?

Maybe Cloudmark provides both kinds of lists now.

Re:Summary

[MrNaz]

1. Email blacklists are a terible idea, and I really sympathise with this guy's plight. I've been at the nasty side of a Spamhaus issue with my own mail server and I can tell you, those guys are nothing but a bunch of digital thugs who have managed to get themselves a nice big stick that they use to hit people randomly with. My server, being private, had just about every conceivable spam prevention mechanism turned on. SSL only connections, authorised SMTP-submission sending only, properly set up SPF records, PTR records correctly registered against the IP to allow reverse lookup. It got registered with Spamhaus and it took me a LONG time to get them to play ball. I'm still listed with a few older BL's but oh well.

2. If someone in a country wishes to circumvent government censors, why on Earth would they use a proxy? Why would they not just use Tor, which can't be blocked or filtered in that manner? If the government is doing deep packet inspection and will infer illegality from mere encrypted traffic, surely transferring illegal content in the clear is worse? Furthermore, setting up Tor is not materially more difficult than setting up a proxy. Not trolling, genuinely interested to know why one would choose the proxy path over Tor.

Re:Summary

[mystikkman]

According to TFA his list is opt-in only, so unless he's lying about that he doesn't appear to be a spammer.

Except that if even a few people viewing the email click on the "THIS IS SPAM" button in their email client/website, you're going on their shitlist regardless of whether people opted in or not. People tend to easily forget what they signed up for, and in some cases even if they remember, hitting the spam button is way easier than figuring out how to unsubscribe, even if the email has a link to do that.

Re:Summary

[afidel]

Let's use your physical mail analogy, under your idea charitable organizations would not be allowed to mail people who have signed up as supporters unless they went through a commercial mass mailing company paying a huge fee per piece mailed. While that's kind of the status quo for poorly run charities with a high overhead cost none of the charities I choose to support are so stupid, why you would want to reduce the amount of money reaching deserving causes and feed the commercial mass mailers I have no clue.

Re:Summary

[fafaforza]

Spamhaus does have some false positives, but dealing with the amount of data they get, it's inevitable. Every time I've dealt with them, they were fairly responsive. Small blocks are removed automatically. Larger ones (like a /19 for example) take an email or two of back and forth.

As far as incentive, if companies want to retain email users, then of course they have incentive to minimize false positives. At the end of the day, it's a balance. It isn't always easy to tell spam from non spam if you aren't looking at your own, personal mailbox that you're familiar with.

Re:Summary

[TheRealMindChild]

Because my concern for the charities that can't go door to door, cold call, etc (just as effective as junk mail), is Nil. Just the same as most everyone on the receiving end of this garbage. Cost/benefit

Re:Summary

[Onymous Coward]

5 fails out of 1000 is 99.50% success.

My personal setup gets 99.78% success (9 spam delivered v. 4139 spam attempts in November), so we should set the bar higher for large corporations. I bet their rate is well above your miscalculation.

Not to dismiss the rest of your post. I thought it was insightful.

Re:Summary

[lipanitech]

Summary is I have yahoo mail its bad lets hope this helps.

Re:Summary

[Onymous Coward]

I'm not sure I understand you. Are you saying a denied spam delivery attempt is a failure?

Oh, from the spammer's perspective.

I didn't think I needed to be explicit. I do not want to receive spam. My filters are designed to deny spam. 4130/4139 spam attempts blocked is largely a success for me.

Re:Summary

[justin12345]

It sounds to me like he might just be getting blocked due to not having SPF records set up properly for all his domains that he's mailing on behalf of. Many mail servers block email from domains that don't bother with SPF.

Re:Summary

[BitZtream]

Sure:

He runs a collection of open proxies and emails a list of them to people to use.

Yahoo and Hotmail are aware of this and blacklist his domains because he is effectively running spam sources.

He's too stupid to understand why they are doing exactly what they should be.

That should sum it up nicely.

Re:Summary

[BitZtream]

This has nothing to do with HIS list being SPAM.

The problem is that he runs OPEN PROXIES which then are naturally then used by spammers to spam people.

If you intentionally run an open proxy and you're surprised that you're blacklisted, you're pretty much the definition of stupid.

Re: Summary

[BitZtream]

Its not about who he mails the list to, its about who uses the services provided by the list.

The list provides a nice collection of proxies to use to send spam.

He could post the list on a website and never email it to anyone and it would STILL be blacklisted since spammers would still find and use his proxies.

Re: Summary

[Urza9814]

They're not regular proxy servers; just an apache server hosting PHP proxy and some other CGI web proxy scripts. I don't believe there's any way to use them to proxy spam.

Re:Summary

[squiggleslash]

Oh bollocks.

Spammers have no problems whatsoever with this spamless utopia you espouse where legitimate emailers can't send email because they're running their own mail server. My mailbox is full of this crap all the time, and I've met people who work for companies that send spam and do everything they can to stretch the rules as far as possible, resulting in their largely unsolicited "Wait, I don't remember signing up for this" crap getting through.

You are the problem. You are the problem because you accept any idiotic solution to spam control no matter who it inconviences, and no matter how ineffective it actually is. Objectively, nothing this article is about concerns any legitimate means of blocking spam. Yet you're in favor of it, because that's the justification.

What you espouse, your support and your willingness to give full throated apologia for this crap, is undermining the email system. You reduce its effectiveness as more and more legitimate applications become impossible, while spammers continue to find ways around it.

Go away.

Re:Summary

[fafaforza]

That would be a fairly poor decision to make since not very many people bother with SPF. If anything, DKIM would be a better choice, especially for Yahoo.

Re:Summary

[Anonymous Coward]

here, here!

There, there.

Re: Summary

[ranulf]

This can't click "this is not spam" if they never get the message in the first place.

Re:Summary

[thedarknite]

My domain (a .edu.au) started being blocked by the hotmail servers yesterday with BAY0-MC2-F2.Bay0.hotmail.com #550 OU-002 (BAY0-MC2-F2) Unfortunately, messages from 203.xxx.xxx.xxx weren't sent. Please contact your Internet service provider since part of their network is on our block list and it was due to not having an SPF record.
After having one created mail goes through, but they seem to automatically be sent to the Junk folder with This message looks suspicious to our SmartScreen filters even though I don't see how. Maybe it's because I haven't had my domain registered with their third party "Safe Senders" list.

Re:Summary

[tricorn]

He's just commenting on the math, 5 failures (delivering spam) out of 1000 is 99.50% success (of stopping spam), not 99.95% as was stated in the post he was replying to.

Re:Summary

[Obfuscant]

According to TFA his list is opt-in only, so unless he's lying about that he doesn't appear to be a spammer.

But then he mentions the main reason he cannot use Constant Contact is because he sends different email to subsets of his full list, not that Constant Contact is a spammer almost beyond compare and won't remove someone from the lists they spew to even when both the recipient AND the sender tell them to.

I'm on two Constant Contact operated lists and there is absolutely nothing I can do to get off, including getting the companies that put me on to remove my address. At this point, I simply filter all Constant Contact email into the bit bucket.

Re:Summary

[Omnifarious]

Not trolling, genuinely interested to know why one would choose the proxy path over Tor.

Tor is frequently very slow. Totally worth it if you want industrial strength anonymity and use it correctly. But if you just care about your own government censors it's overkill.

Also, you can get very strange an unpredictable results from geographic targetting of Internet services. Oftentimes things will ignore any information you give them about what language you want to see the site in and decide that you should be seeing it in German because the IP you came from was in Germany. But then the next page load will be shown in Russian because the next connection came from a Russian IP. Which is very odd because all the session information is the same. But it still happens.

Re:Summary

[Entropius]

He's not sending mail from the open proxies. He's sending mail telling people where the open proxies ARE.

Re: Summary

[complete loony]

If the email providers a adding "golflanding.com" to their black list, stop sending the entire domain name on one piece. Construct a simple Turing test that the end user can solve to construct the name. If the text of the Turing test starts getting blocked as spam, you may be able to modify the test. Reducing the risk of the name of the proxy becoming well known or getting blocked automatically.

Perhaps he needs to build a new communication channel with his subscribers as his current approach has some obvious downsides. When a user first connects to a proxy give them a couple of proxy names randomly selected from your new list. Perhaps some client software that can silently refresh its list in the background, provided that one link still works.

Re:Summary

[Anonymous Coward]

How the Great Firewall of China is Blocking Tor: http://www.cs.kau.se/philwint/static/gfc/

Re:Summary

[jdavidb]

So your solution to the school bully demanding you give him your lunch money everyday is to just hand over the money?

That analogy doesn't fit at all.

Re:Summary

[girlintraining]

And he'd be right. Look, I'm normally a night owl but this week they got me in training and I have to wake up at 4 in the fucking morning... so if I'm off by a decimal point here or there, it's not surprising. I'm not going to care too much about 0.45% when we're talking about something as petty as spam. :)

Re:Summary

[trancemission]

Nailed it.

I also recall this story - simple point to take from this:

The reason the emails and hence his servers are being 'blacklisted' is due to the *content* of the message.

I am sure there are *many* companies out there that send emails out on this scale (and much more/often). I am to lazy to cite but think about it....

Please bring back the wild west of the internet, not much is free in this world now and I don't just mean in the monetary sense.

Why is this person being censored?

Political
Ineffective algorithms
Incompetence
+ Add your own

*Checks tin foil hat*

Re:Summary

[MurukeshM]

And I report your mail as spam or junk or (if the option is given) phishing. Screw you over, since my response is likely to get your emails to others classified as spam or - worse - phishing. If it takes me more than two clicks to unsubscribe, you're marked spam. Most people won't respond, some will be vindictive like me and very few will be willing to send an email.

Re:Summary

[TheMMaster]

My main issue is that there is no recourse. I did everything right, I had a /19 with all the correct WHOIS information, my server had nothing illegal on it or things that spamhaus doesn't like. (those two things aren't necessarily the same, that's the other thing that really annoys me)

And to pressure my ISP they decided to make it impossible for ME to use the server I paid for. I understand what they are trying to do, but the way they are doing it leaves a lot to be desired. If they actually cared about the damage they did they would have unblocked my /19 for at least a reasonable period of time for me to migrate. It's not like I got a warning or anything, I only found out I was shitlisted after people started to complain.

After that it took me several days to get everything moved over, DNS changed etc, and again: no recourse, no way of temporarily getting my service restored. The only thing I got was a warm fuck you from Spamhaus.

Re:Summary

[TheMMaster]

that was a /29 not a /19 whoops

Re:Summary

[fifedrum]

both, they've provided both for as long as I can remember, at least three or four years. MTAs are covered under the RBL, the URIs under the antispam "cloudmark authority" engine. One blocked at the connect phase, one at the end of the data phase.

Re:Summary

[jonadab]

> Just FYI, I seen this guy bitching about it MONTHS ago.

Furthermore, the rant just posted on Slashdot is a verbatim copy of the one I read months ago (or, at least, the part that I re-read today is verbatim; I declined to re-read the whole thing, on the grounds that I remember it pretty well).

Re:Summary

[LordLimecat]

Summary: Once again, Bennet Haselton gets blocked from sending out automated email detailing how to bypass school and corporate IT policies, and wonders where he went wrong.

Here are a few protips, Bennett:
  * Offering services to get around various kinds filtering will eventually cause you problems on the internet, especially when said activities will make you an "undesirable" for IT stafff in general.
  * Sending out automated emails with automatically generated content likely to be on filters will cause you problems.
  * And biggest of all, you do noone any favors by teaching kids how to violate their school's computer-use agreement. If someone is a "victim" of filtering that they cannot simply uninstall, chances are 99% of the time they have no implicit / irrevocable right to the network / computer resources. Being a guest on the network means you play by their rules.

Re:Summary

[LordLimecat]

The domains he mentions in his automated emails are considered threats by a lot of filtering programs out there. Theyre literally about circumventing acceptable use policies.

Theres not much of a mystery here; I feel like this story comes up every few months, and he still doesnt get why hes so unpopular with IT departments.

Re:Summary

[eugene ts wong]

That guy is criticizing somebody for having a short attention span, but doesn't even have the time to spell and write properly.

yeah, spam blacklists are a poor solution

[Trepidity]

I could maybe see their necessity 10 or 15 years ago, but statistical classification techniques are good enough these days that a blunt tool like a domain blacklist doesn't really make much sense. Heck, Paul Graham was arguing that seven years ago, and it hasn't gotten less true.

Re:yeah, spam blacklists are a poor solution

[Anonymous Coward]

I wonder how many job opportunities I've missed or friends I've drifted apart from because of email dropped by statistical classification techniques. That's why everybody uses Facebook to keep in touch now.

Re:yeah, spam blacklists are a poor solution

[ColdWetDog]

I wonder how many job opportunities I've missed or friends I've drifted apart from because of email dropped by statistical classification techniques. That's why everybody uses Facebook to keep in touch now.

Friends? An AC on Slashdot?

Jobs? An AC on Slashdot?

Not to worry.

Re:yeah, spam blacklists are a poor solution

[niiler]

Mod up. This is a very good point. Closed systems like Facebook seem to work.

Re:yeah, spam blacklists are a poor solution

[pixelpusher220]

yes but maybe not for who think they work for...

Re:yeah, spam blacklists are a poor solution

[AlphaWolf_HK]

The spammers have found various ways around these. Often they throw a bunch of the "high target" key words (e.g. viagra, cialis, penis enlargement) in as images, or they'll use computer generated text that looks somewhat real enough to even fool some human readers in order to throw off those filters. This works because the more words you have, the less likely the small terms will be snagged.

Re:yeah, spam blacklists are a poor solution

[Anonymous Coward]

The problem is, 99% of connections to an SMTP server that hosts mailboxes for the general public are complete garbage. Running a statistical filter on those messages is a huge waste of resources when the assholes are already identified.

Re:yeah, spam blacklists are a poor solution

[Jstlook]

What about the slashdot article a few weeks back about the Dallas Cowboys complaining that Facebook wanted to charge him three grand to send out one update to his facebook friends?
Also, what about the fact that, at that level of users (100k+ish) Facebook *won't* post your update to each of your facebook friends? They just silently drop messages.
I don't know - just a thought.

Re:yeah, spam blacklists are a poor solution

[Onymous Coward]

Paul Graham and yourself are making the same error: blacklists are not all the same.

Don't conflate the mechanism (a list) with the method (how things get on the list).

Graham, showing his misunderstanding (emphasis mine):

Server blacklists tend to go bad, because the power they confer corrupts the people running them. They turn into vigilantes and start blacklisting innocent servers. ...
This is bad news, not just for the SBL but for the whole idea of blacklists. The SBL was started with the explicit aim of avoiding the kinds of abuses that had tainted other blacklists. So if even they are going the way of the MAPS RBL, one has to assume that every blacklist will, eventually.

So not only is Graham stuck on a single concept of what a blacklist is, but is trying to paint all list-providing organizations with the same brush he denigrated MAPS with at a particular point in time.

If you have ever shopped for blacklists with any degree of detail in your search you know that blacklists are not all generated the same way, they are not a homogeneous mass. One list might be composed of systems probed for open relays. Another might comprise only systems that have sent the list developer spams. Another might be of systems that have scanned the list maker's network. Heck, you could have a list that's just of IPs matching the birthdays of the top dozen R&B artists. Just look at the quantity of lists out there, as seen at (the now outdated)dnsbl.info. 80 blacklists by several dozen organizations — are they all going to be produced the same way, carry the same kinds of information?

So, yeah, the fact that blacklists haven't gotten less blunt as a tool for fighting spam has indeed not gotten less true, because mu, it wasn't true in the first place.

Re:yeah, spam blacklists are a poor solution

[Trepidity]

Well, sure, if you're not using good techniques, or training them well. Gmail is an example of doing it right, and has a very low false-positive rate, while not resorting to blanket domain blacklists. Why can't Microsoft and Yahoo match Google's performance there?

No Comparison To China and Iran

[Anonymous Coward]

The blacklists and censorship dealings in China and Iran are directly attributable to their respective governments, there is no similiar connection in hotmail and yahoo's blacklists.

Stop this, you look like fools.

Re:No Comparison To China and Iran

[rudy_wayne]

if you care about users in China and Iran

You had me up till there. At that point I realized you're an asshole and stopped reading.

Re:No Comparison To China and Iran

[Aristos Mazer]

His whole service is serving the dissidents in those countries. He isn't an asshole, he is actively trying to promote freedom in those countries. This isn't the "won't you care for the children" emotional plea that you were expecting. It is a description of the problem he actively works on, and if it is one that you care about, you should be helping here.

Spam is like cancer

[Anonymous Coward]

The only treatment is a deadly poison that you hope kills off the bad parts before the good suffers too much.

People still use Yahoo mail?

[Anonymous Coward]

I think you just wanted to go on a political rant there. Seriously, you spend the post talking about the failings of two companies, ignoring the fact that there are other companies out there (well, you do mention GMail once, but you don't give any supporting evidence for it not being "open"), and act like two companies doing particular things is some kind of "failure of the free market."

So what's your solution? What's to stop a government-owned email provider from using this SmartScreen thing "as a matter of policy?"

You're a bleeding moron

[Anonymous Coward]

Seriously? It's fucking news that there might be domain blacklists that aren't public knowledge?

"Free market" scare quotes

[Freddybear]

What's with the gratuitous complaints about the "free market" not giving some mythical "optimal solution" that lets you send your "100% guaranteed opt-in" spam without interference? I call bullshit. If Hotmail isn't accepting your "really honest it's not spam" mailing list stuff, maybe you should try contacting them about it. The "free market" doesn't magically solve problems without people doing what it takes to address the problems.

Re:"Free market" scare quotes

[Freddybear]

That's just silly. If you can't be arsed to do something about your "honest it's not spam" emails getting blocked, you don't have any business complaining about the people who do the blocking. Stop complaining about "the free market" as if you'd prefer an unfree one.

Re:"Free market" scare quotes

[PlusFiveTroll]

The problem with most be free email providers IS contacting them. You're not paying them, so they don't give a shit. Hell Google is hard enough to get a hold of when you are paying them.

The second problem is spammers lie about everything. This has turned server operators on to the line of thought that 'everyone is a liar'. If you weren't a spammer you wouldn't have been blocked in the first place. Needless to say this causes a number of race conditions.

And yes, I do run outbound and inbound SMTP services for a good number of customers at a small ISP.

Re:"Free market" scare quotes

[c_sd_m]

He did contact Hotmail about it. He got a form letter response, replied again and hasn't heard anything. Yahoo keeps just sending him generic, irrelevant articles from their knowledge base. Neither company is "contact-able" in any useful way.

Re:"Free market" scare quotes

[Freddybear]

Maybe Hotmail blew him off because he acts just like any other spammer. Changing domains and using remailer proxies isn't exactly the behavior of the usual legitimate bulk emailer. And yes, I do subscribe to a few of those, and I use ATT's Yahoo email account and I get my subscribed stuff just fine.

Re:"Free market" scare quotes

[PNutts]

Wish I had mod points. You hit the nail on the head.

Re:"Free market" scare quotes

[MaxToTheMax]

In fact, the author has become part of the free-market solution, by inadvertently "auditing" the quality of Yahoo and Hotmail's email service, and motivating their customers to demand better.

Re:"Free market" scare quotes

[chfriley]

And if he is really concerned about the "free market" not doing enough, there is always the option of starting a competing service that is more "free market" and responsive. Yes, that costs something, but that is the nature of the "free market" - it isn't free to everyone, someone's time costs something, always.

Server load

[betterunixthanunix]

Blacklists are nice because they reduce server loads. Sure, running a statistical classifier for one user is not so hard, but if you have to process hundreds of millions of messages per day, that is a lot of CPU time spent on spam.

Now, I agree that blacklists are bad, but we do need some system that doesn't require large amounts of CPU time or other resources. Hashcash is interesting here, in that the CPU time is mostly spent by clients; one might be able to slow spam down enough to let a combination of statistical filtering and greylisting take over.

Re:Server load

[nabsltd]

Blacklists are nice because they reduce server loads. Sure, running a statistical classifier for one user is not so hard, but if you have to process hundreds of millions of messages per day, that is a lot of CPU time spent on spam.

The CPU time spent on running something like SpamAssassin is insignificant compared to the bandwidth, disk writes, etc., caused by spam. Keeping the incoming e-mail in a RAM disk until you have truly accepted it for delivery (which isn't dangerous even if the server crashes hard) is the #1 thing that speeds up e-mail intake. At that point, scanning takes almost no time.

As you mention, though, greylisting does the best job of keeping your overall load down, since you don't even need to use network bandwidth on the body unless the sending server is known to retry, which basically eliminates every botnet member. Maybe this solution would work now that other more "instant" messaging systems are readily available, but 15 years ago when IM wasn't really a corporate thing, I couldn't use greylisting because "it slowed down e-mail too much", even though it didn't slow it down at all for the "important" clients, as their servers got whitelisted anyway.

Re:Server load

[gamanimatron]

but we do need some system that doesn't require large amounts of CPU time or other resources.

Why? CPU time is dirt cheap if you can concentrate your task. The bandwidth (a much scarcer resource) is already being spent, and better decisions will just tend to reduce your costs there. To me this smacks of laziness, not efficiency.

Re:Server load

[dodobh]

Nope. You can't acknowledge receipt until the message is written to durable storage.

And you have never run a really large email system. Bandwidth isn't really a limitation, disk io is.

Re:Server load

[dodobh]

Bandwidth isn't a scarce resource with email. It's disk io.

CPU time gets really, really expensive when dealing with email at a very large scale.

Bayesian scanners are nice for individuals, but do not work for groups of people with different tastes. False positives are even worse than false negatives.

Re:Server load

[sirsnork]

Not only that, but if your software is smart it only greylists server that it hasn't ever sent an email to anyway. So after the first week of being installed all your important customers are automatically whitelisted

Re:Server load

[nabsltd]

You can't acknowledge receipt until the message is written to durable storage.

Which is exactly what I said. If the message is rejected, everything has taken place in the RAM disk, and you don't care. If the message is accepted, then sendmail has written it to the queue at the very least. As long as the queue is not on the RAM disk, you're fine.

And you have never run a really large email system. Bandwidth isn't really a limitation, disk io is.

Again, I said this too: "CPU time....is insignificant compared to the bandwidth, disk writes". I guess Slashdot has gotten to the point where nobody reads anything before posting. But, having talked with people who run e-mail systems for millions of users, not having to receive the spam in the first place (unfortunately usually through blacklists, but often using greylisting) is the biggest win, as bandwidth is in reality far more of a limitation. It's quite easy to put together a disk array that gives you 500MB/sec throughput, while it's not easy to pay for a 4Gbps inbound line (which it what it would take to saturate those disks). Even assuming 8 bytes written per incoming byte, it's still pretty easy to spread load to effectively 2-4GB/sec worth of disk, while 2-4Gbps of Internet connectivity is pricey.

Re:Server load

[nabsltd]

Not only that, but if your software is smart it only greylists server that it hasn't ever sent an email to anyway.

Unfortunately, inbound and outbound SMTP servers often don't have the same IP address, so this doesn't work in practice.

So after the first week of being installed all your important customers are automatically whitelisted

But this is still very true. Only the first e-mail message is delayed, and that delay is mostly controlled by the retry time set on the sending machine. There are some really annoying ones that retry once a minute for a couple tries, then back off to an hour or more. This is the worst as far as greylisting is concerned.

Once a server is whitelisted, then my implementation allows 40 days of no activity from that server before it drops off the whitelist. This means that even a once-a-month mailing list from an obscure server doesn't see any delay after the first time. About 1/4 of the 380,000 IPs that have ever contacted my e-mail servers are currently whitelisted.

Re:Server load

[dodobh]

You are speaking of sequential io when you say 500MB/s. Email is random io. On spinning rust, your bottleneck is seek time.

You have talked with people who have run systems for millions of users. I *have* run systems for millions of users. Not having to receive spam saves you from spending CPU cycles on email, disk seeks, fancy RAMdisk based architectures (I know how to do this, but there's a good reason I recommend against it).

Re:conspiracy ! deliver my spam

[Anonymous Coward]

If it wasn't for real Viagra costing $25 a pill, there wouldn't be as hot a market for the spam.

Simple summary

[Pollux]

He's saying that Hotmail, Yahoo, and GMail are running a cartel of free online webmail services.

He's trying to get opt-in email to accounts on these systems, and it's not going through. He has evidence indicating these services operate a common hidden blacklist service keeping those emails from getting to the accounts. He cannot reach people within these organizations to open up emails coming from his domains, as he does not have an inside contact to "assist" him with this problem. This leads him to speculate that Hotmail, Yahoo, and GMail are operating like a cartel, where only "approved" email list hosting service companies with inside contacts are able to do business with these services.

Better?

Re:Simple summary

[niiler]

Bingo. Good summary. I gave up using my own server to send email a couple of years ago for precisely these reasons. It wasn't worth trying to get de-blacklisted every few weeks because my server had an obscure domain name. If I recall, when I sent out more than 10 emails in a batch (we're talking maybe as many as 30) to members of a class, this triggered the anti-spam bots. When I did it from gmail or from other major providers, things worked beautifully. I had too many irons in the fire to deal with this, and while I would love to use my own server's email capability, it's not worth it anymore.

Re:Simple summary

[sorensenbill]

Much better, thank you!

Re:Simple summary

[kelemvor4]

Yes, now let's see if someone at dice can replace the article with this actual summary!

Re:Simple summary

[marcello_dl]

> I gave up using my own server to send email a couple of years ago for precisely these reasons

In fact, that's probably what the cartel wants, ultimately.

Re:Simple summary

[Anonymous Coward]

Or, you could just keep using your server as before. People who use providers which block your server could wise up and use something else, rather than let Google harvest all their email for marketing purposes while sometimes letting them see an email they want to see.

When you switch to Google, you become part of the problem.

Re:Simple summary

[SomePgmr]

This is weird... I don't think Google was mentioned in the summary at all.

But regardless, they're not operating with a list of approved senders. I build my own systems and send mail through them all the time. Sometimes just regular mail service, some for mass emailing (legally and legitimately). You'll have to take my word that I don't have super-secret inside contacts at Google, Yahoo and Microsoft to make sure this works.

Now if you meant to say they have anti-spam filters that occasionally throw false-positives and block mass emails from some domains I'd say, "Well no shit, welcome to email on the internet since the 90's".

Re:Simple summary

[BitZtream]

No, you left out the part where it has nothing to do with his list that causing him to get blacklisted and the fact that the list contains sites themselves that are used for spamming.

Yes, open proxies get blacklisted, no shit, its a true story.

Re:Simple summary

[jibjibjib]

It's ok to blacklist email received from open proxies. It's not ok to block legitimate email for just *mentioning* them.

Re:Simple summary

[Gr8Apes]

Having problems with this myself at the moment. It seems that entire blocks (as in Class B blocks) are being listed partly as an effort to remove the ability of "normal" people to run mail servers and force people to services such as Hotmail, Yahoo, and Google. Or, you can pony up the extra money and buy your static business IP(s) and for 10 times the cost for the same service, be "approved". You have to pay to play, it appears.

Re:Simple summary

[drinkypoo]

He's trying to get opt-in email to accounts on these systems, and it's not going through. [...] This leads him to speculate that Hotmail, Yahoo, and GMail are operating like a cartel, where only "approved" email list hosting service companies with inside contacts are able to do business with these services.

This leads me to speculate that it's no longer worth it to try to send mail through legitimate channels, when I can pay someone to just go ahead and spam on my behalf, and they can go through the pain of working out the details of getting my ads to my potential customer base.

It's too bad there's not more awareness of RSS, because it's potentially a much better way to stay connected with a customer base than spamming them.

Re:Simple summary

[LordLimecat]

This is the most ridiculous idea ever. Ive done IT consulting for over 7 years now, and I have dealt with hundreds of tiny companies with bizarre and obscure names who host Exchange on Windows SBS; somehow their emails manage not to get blocked.

Perhaps filtering outbound mail and taking action when your public IP starts sourcing spam has something to do with it. Perhaps making sure that the sent mail was legitimate also contributed.

What you suggest would mean that gmail and yahoo would be completely useless for a lot of small businesses (a HUGE part of business traffic, since larger companies are going to email more internally and less externally).

Re:Simple summary

[LordLimecat]

You skipped the part where the emails are sent in an automated fashion and that their content is very likely to violate computer acceptable use policy in a LOT of places, which I strongly suspect is related to the reason they are blocked. He also mentioned starting up a number of domains to get around the problem, which behavior is quite remeniscent of how spammers rapidly register obscure domains to get around blacklists.

Theres not a story here. People use blacklists, and have for years, and some of those blacklists are shared. If you want to stay off the blacklist, there are simple steps you can take:
  * Dont run an open relay
  * Probably a good idea to have a static ip, especially so that...
  * You have an accurate reverse DNS entry
  * Block outgoing SMTP from network except from trusted sources to prevent spamviruses from getting you blacklisted
  * SPF records wouldnt hurt
  * If you dont use SPF, it might help if an MX record actually exists for the domain you are sending from
  * Make sure your server can handle greylisting (ie, that it will retry after some period if it receives a "server busy" response)

Question that was never answered last time...

[Anonymous Coward]

Are the proxy servers you are sending out on these lists capable of relaying mail onwards on port 25? If so this is probably a significant factor in these blacklistings. If you block outbound connections to port 25 when you set up these proxies, you'll probably find your blacklist problems are significantly reduced.

5 second summary

[IamTheRealMike]

Blah blah blah ...... I sent craptons of mail to people who I'm sure want to receive it ..... but the system is telling me people don't .... blah blah ..... free markets suck.

I have worked on spam filters before. I've heard this story a million times. In case the article poster reads this, here's the blunt reality:

Those half-million people you think really really want new proxy sites all the time? Guess what, many of them don't. They are reporting your mail as spam which is why you're getting blocked (this is domain reputation). You may not understand why, but they are, so deal with it. Expire addresses that signed up a long time ago - some people won't unsubscribe when it's no longer useful for them. Make sure it's a simple, obvious one click operation to unsubscribe, and I mean really one click - not "click, log in, go to preferences" etc. Being able to unsubscribe should be the easiest thing in the world.

If SpamHaus is blacklisting you, they probably think you're sending mail to their spamtraps. Hence the "zero false positives" claim. Are you sure every single address on your list replied to a confirmation mail? All 400,000+ of them? Because it sounds unlikely.

Re:5 second summary

[DRJlaw]

Those half-million people you think really really want new proxy sites all the time? Guess what, many of them don't. They are reporting your mail as spam which is why you're getting blocked (this is domain reputation). You may not understand why, but they are, so deal with it.

You assume that this is case, yet the poster provides a link to management data which at least appears to show that your assumption is incorrect. Did you read the post where it mentions that "[it] showed a 'complaint rate' of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list)," or are you simply going to deny any version of reality that doesn't align with your assumptions.

Expire addresses that signed up a long time ago - some people won't unsubscribe when it's no longer useful for them.

Apparently, deny any version of reality that doesn't align with your assumptions.

BAD 'EXPERT'!

If I sign up to a mailing list, I expect to receive the output of that mailing list until I unsubscribe. I certainly don't want the mailing list silently dropping me, and I'm not very interested in the ISP offloading its mailing list problem onto me by making me affirmatively renew my subscription. Especially when you offer no evidence that 'addresses that signed up a long time ago' make up a disproportionate fraction of the alleged 0.1% spam report rate.

Pushing the problem onto the 400,000+ individual users instead of dealiing with it at the ISP level is exactly the sort of free market failure tha the poster complains of.

If SpamHaus is blacklisting you, they probably think you're sending mail to their spamtraps. Hence the "zero false positives" claim. Are you sure every single address on your list replied to a confirmation mail? All 400,000+ of them? Because it sounds unlikely.

Again, deny any version of reality that doesn't align with your assumptions. He isn't being blocked by SpamHaus. He's being blocked by Hotmail and Yahoo. Just admit that you haven't actually read the post, that you're spouting off about your own personal bugbear, and that your advice has almost no bearing on the actual problem. It'll make you feel better, honest.

Re:5 second summary

[Pope]

Why does he need to send 400,000+ emails in the first place? If it's just a list of proxy domains, why not just have an RSS feed that people can subscribe to? No emails needed.

Re:5 second summary

[Kergan]

+1. TD;DR the article, but the parts I did made this whole story reek of "your unsubscription method isn't braindead obvious enough to end-users, so they're unsubscribing by hitting the Spam button until your emails go away for good."

Re:5 second summary

[amicusNYCL]

They are reporting your mail as spam which is why you're getting blocked (this is domain reputation). You may not understand why, but they are, so deal with it.

That's one possibility, and may even be likely considering his subject material. In this example he says he sent a total of 7 new proxy domains to 420,000 addresses, but only sent 1 domain to each person. So each domain got sent to a random 60,000 people, his reasoning being so that a censor could not subscribe and get a list of all new proxies, they would only get one (per address, at least).

But, instead of them getting those emails and blocking the proxies, it may be more effective for the censors to always report his emails as spam, thereby getting them blocked, and then no one gets any of the 7 new proxies. So the people reporting spam aren't doing it because they don't want the mail, they're doing it to stop other people from getting it.

Obviously, this is 100% speculation.

Re:5 second summary

[magic maverick]

Because then someone from the censorship companies or the censorship departments could easily get all the latest domains and block them automatically. By creating multiple domains and emailing them to a section of his subscriber list, he makes it that much harder to block all of them.

Re:5 second summary

[magic maverick]

At the top of the emails:

[You are receiving this because you subscribed to the Circumventor distribution list.
To unsubscribe from this list, click here:
http://www.peacefire.org/circumventor/cv-unsub.html
or reply with the word "unsubscribe" in the subject.]

Seems pretty easy to me...

Re:5 second summary

[IamTheRealMike]

You assume that this is case, yet the poster provides a link to management data which at least appears to show that your assumption is incorrect

I assume this is the case because, like I said, having actually worked on a large spam filter I've seen this kind of story many times before. These people are always amazed to discover that people are pressing report spam on their wonderful bulk mail. Yet the fact remained that people were doing exactly that. They didn't want the mail.

Look at it this way. This guys screenshot shows Hotmail themselves saying he hit some of their spamtraps. From the SNDS FAQ we can see that "trap hits" means he mailed accounts that don't solicit mail - ever - so we already know his claim that every account is opt in isn't true. What else isn't true?

Pushing the problem onto the 400,000+ individual users instead of dealiing with it at the ISP level is exactly the sort of free market failure tha the poster complains of.

It's not a free market failure at all, these sorts of big webmail spam filters are very effective. If users are seeing false positives they can go and unmark the mail as spam, the system will learn that the user wants that mail and the problem is solved.

Again, deny any version of reality that doesn't align with your assumptions.

My assumption is that this story is much like all the other such stories I've come across - the guy is a spammer and doesn't realize it. This assumption is very, very likely to line up with reality.

Re:5 second summary

[Methuseus]

Because the RSS feed's server will likely get blocked, but the emails are less likely.

Re:5 second summary

[czth]

Um... wow, it's sorta sad that I have to explain this.

Imagine you're the Chinese Minister of Censorship, or the flunky that manages the Great Firewall. You learn about a website with an RSS feed with a continually updated list of anti-censorship proxies. What do you do?

(On the other hand, you haven't blocked Hotmail or Yahoo! or other email providers, because, well, riots are bad for business.)

Re: 5 second summary

[Urza9814]

RTFS. Hotmail confirmed that the portion of users marking it as spam was extremely small.

Furthermore, do you realize how many users will click the 'spam' button when they fully know it's something they subscribed to simply because they can't be bothered to take half a second to click the prominent unsubscribe link or send a reply? These people are trashing spam filters. And I know they're out there, because I got it all the time in college. Ran a student club with a mailing list of around 400 users (out of 40k+ students)...Email address were collected and added by hand, one at a time. Each message was typed and sent by hand from an officer's personal email account. And each one had an unsubscribe link highlighted in standard font at the bottom of the page. We'd still get a handful of 'this message reported as spam' emails with every single message we sent out. Yea, obviously some users didn't want our mailings, but they definitely opted in and we couldn't have made it easier to opt out...instead they chose to try to have our messages blocked from all users of that email service.

Point being, just because some people report it as spam doesn't mean it is. Also, the percentage reporting it for us would have been orders of magnitude higher than in this case and we still never got blacklisted.

Re:5 second summary

[afidel]

Because then the blocking companies would just subscribe to the RSS and the proxies would be blocked as soon as they were posted.

Re:5 second summary

[nabsltd]

Why does he need to send 400,000+ emails in the first place? If it's just a list of proxy domains, why not just have an RSS feed that people can subscribe to? No emails needed.

Because the people who want the proxy list would need to use a proxy to be able to read the RSS feed, as censorship in their country would block access once it was learned what the RSS feed contained.

On the other hand, incoming e-mail can be blocked as spam, but you can't decide to block an e-mail as spam without knowing something about it, and if it isn't coming from the same domain or IP address, and doesn't have the same content as previous spam, it's pretty tough to block. So, once the e-mail gets through, the "damage" is done (as far as the censors are concerned).

Re:5 second summary

[Kergan]

Fair enough. But you know, in my (admittedly convoluted) experience in this kind of stuff, a separate issue is user trust and laziness.

It's one thing to write the above notice to paying customers who you're sending news and updates to; even if users got tricked to subscribed, they trusted the business enough to purchase something, and they're likely to unsubscribe -- even though more than a few actually mark it as spam. It's an entirely different thing to write the above notice to an opt-in list, unless it wasn't double opt-in -- which, insofar as I can tell from TFS/TFA, it isn't. When not, for all you know, you mistakenly gave your email or your email got sold, and you've no clue whatsoever what may happen if you click the link. Will you get even more spam? Will you end up on new lists? And even when it's double opt-in, actually.

Here's a thought: instead of simple or double opt-in, I'd suggest that businesses running mailing lists implement a recurring re-subscription opt-in. By this, I mean send the whole list a quarterly reminder of the lists they're subscribed to. Unless readers click a link to confirm that they wish to continue to receive emails, they automatically get unsubscribed. Never mind that most lists would instantly get trimmed to a notch about zero -- if that -- because the recipients might fail to read that email: the fact of the matter is that you're getting filed under Spam precisely because you're not getting read and you're annoying your audience.

Re:5 second summary

[DRJlaw]

Look at it this way. This guys screenshot shows Hotmail themselves saying he hit some of their spamtraps. From the SNDS FAQ we can see that "trap hits" means he mailed accounts that don't solicit mail - ever

Yes, you've noted three trap hits out of 68,000 messages. Do you want to bet that those three trap hits are signup confirmation emails resulting from (i) typographical errors in the email address submitted by someone attempting to sign up or (ii) 'drive by' sign ups by a third party who has an axe to grind against the list?

There is essentially nothing to prevent someone from signing mike@plan99.net up to a dozen mailing lists in the signup process. It doesn't matter if they're mile@plan99.net or simply a jerk -- the fact that it happens (at 0.0044% frequency) doesn't transform the mailing list operator into a spammer. Even Hotmail notes that "[w]ell-behaved senders will hit very few such accounts because they're generally sending to people who give them their address and because they collect and process their NDRs." They don't expect a zero rate.

so we already know his claim that every account is opt in isn't true. What else isn't true?

That you're not bothering to think through the signup and confirmation process, for one... that your putting claims in his mouth that he never made, for another... "The list is also comprised of 100%-verified-opt-in addresses, meaning that a new subscriber has to reply to a confirmation message in order to be added to the list. That's considered the gold standard for responsible mailing." There's simply no basis for you to say that those accounts were falsely opted-in.

FYI in the complaint rate section the SNDS FAW states that "more than 30% of the IPs sending mail to Windows Live Hotmail keep their complaint rate at less than 0.3% and this represents a good bar to shoot for." He's allegedly at 0.1%. Your expectations are simply unrealistic, and yet again show that you're not willing to deal with the reality of the situation rather than attributing anything other then perfection as being evidence that "the guy is a spammer." Frankly, you're a perfect example of the problem at hand.

If users are seeing false positives they can go and unmark the mail as spam, the system will learn that the user wants that mail and the problem is solved.

Again, deny any version of reality that doesn't align with your assumptions. Reread the actual problem -- the users do not see these emails when sent from this account, and therefore cannot unmark the mail as spam. The problem is not solved. The problem isn't even remotely what you concieve it to be.

Re:5 second summary

[magic maverick]

It is double opt-in. As in, I signed up for the emails. I get an email asking me to confirm. I click reply and send. I get the emails. Text from the confirmation email (with email addresses changed):

Dear webmaster@yahoo.com

We have received a request to subscribe this address to our mailing list, where send out the locations of new 'Circumventor' servers to help bypass Internet censorship.

To confirm that you want to subscribe this address to our mailing list, you MUST REPLY TO THIS MESSAGE without changing the subject line. (The subject line has a 14-digit number in parentheses on the end, and you have to leave that in the subject when you reply.) Just hit 'Reply' and hit 'Send', and that should be enough. This is to prevent people from signing up other people without their permission.

Please do not write any message to us when you reply, since the replies are processed automatically and your message will not be read. If you have any questions, please send a separate message to webmaster@hotmail.com

Once you reply to this message, you will be added to the Circumventor list.

Thank you.

Re:5 second summary

[czth]

They randomize which proxies get sent to which random parts of the list (see what RTFA gets you?). Granted, the Minister signed up enough accounts hed' probably get them all (unless they got suspicious at all the requests from evil-commies.cn). Still, harder than one central website.

Re:5 second summary

[IamTheRealMike]

Once again, what I'm saying is, you're accepting everything the poster says on the assumption it's absolutely true. Spamtrap accounts don't reply to confirmation emails or click on confirmation links - ever. That's the whole point of them. Even if you're a malicious troll who got a list of Hotmail trap accounts from somewhere, how do you get control over them to confirm signup?

The screenshot says more anyway. Judging from what he says the sizes of the mailshots are, it's a fresh IP that hasn't been used before. So the screenshot could have been taken before the reputation degrades. That by itself probably won't help, a new IP that sends links to newly registered domains which have no reputation to huge numbers of users and hits spamtraps is exactly the sort of thing spammers actually do.

Look. It's possible that this guy has done everything totally by the book and somehow has just got unlucky that his behaviour happens to closely match that of actual spammers. Or it's possible that we don't have the full story. Having been on the other side of such stories and investigated cases like these, I think "sender is not following standard mail etiquette" is far more likely than some enormous conspiracy theory against him. After all, plenty of bulk mail senders do just fine.

Re:5 second summary

[BitZtream]

Wrong.

If it requires anymore than clicking a link in the email, its failed. Going to a page, doing more crap, blah blah blah, I just hit 'spam' and move on, so does everyone else. If I don't want it, its spam, period. You as the sender need to make it so A) I want it and B) I don't get bored/annoyed trying to get rid of it after I'm done wanting it.

He also hasn't bothered to setup feedback loops with Yahoo and Hotmail, which would solve his problem and show that he had a clue.

He's also sending a list of open proxies which can be used to ... login to yahoo/hotmail with fake accounts and send spam.

There is nothing about what he is doing that makes him wanted by anyone.

I personally have several accounts subscribed to his list. I use his list to block domains at my mail server, he provides me an up to date set of lists every few days so I can block him.

He's really not that good at what he's doing.

Re:5 second summary

[psmears]

Spamtrap accounts don't reply to confirmation emails or click on confirmation links - ever. That's the whole point of them. Even if you're a malicious troll who got a list of Hotmail trap accounts from somewhere, how do you get control over them to confirm signup?

The malicious troll doesn't need to confirm signup - only to request it, at which point the list server will send an email to the spamtrap, and boom, your reputation takes a hit. All while you're conforming 100% to best practice.

Look. It's possible that this guy has done everything totally by the book and somehow has just got unlucky that his behaviour happens to closely match that of actual spammers. Or it's possible that we don't have the full story. Having been on the other side of such stories and investigated cases like these, I think "sender is not following standard mail etiquette" is far more likely than some enormous conspiracy theory against him. After all, plenty of bulk mail senders do just fine.

I see what you're saying, but he's not actually having his IP blocked in this case. The blocking is taking place based on the content of the message, specifically whether it mentions certain domains set up as relays. The interesting question (from his point of view and ours) is exactly how those domains become flagged as "spammy". For instance, I'd be interested to know (as others have asked) whether the relays allow traffic on port 25, and whether this is a factor.

Re:5 second summary

[DRJlaw]

I see what you're saying, but he's not actually having his IP blocked in this case. The blocking is taking place based on the content of the message, specifically whether it mentions certain domains set up as relays. The interesting question (from his point of view and ours) is exactly how those domains become flagged as "spammy". For instance, I'd be interested to know (as others have asked) whether the relays allow traffic on port 25, and whether this is a factor.

Actually, as I read it, it's the combination of mentioning certain domain names and the fact that the message originates from the mailing list IPs. It seems that other messages from the same IP would be received rather than blocked (not specifically discussed, but implied in the three not-banned domain names) and that messages containing the same domains sent from other IPs and email addresses would be received rather than blocked (specifically discussed in his gmail example, paragraph after the session transcript).

It's a far more specific block, and one that I suspect whitelisting the mailing list email address does not overcome (I'm not a member of the mailing list) -- which would be the ultimate issue here.

Re:5 second summary

[IamTheRealMike]

All spam filters do domain blacklisting. The reason is that the textual content can be randomized for free, but spammers typically want to sell something, which means providing links to their stores. It's much harder to avoid having links in your mail, so it makes sense to measure their spammyness and blacklist. Or at least it used to. The prevalence of link shorteners and hacked websites means it doesn't work as well as it once did.

I suspect there's a rule in Hotmail and Yahoos filters that say something like "if a mail contains a link to a young domain that has never been seen before, and it goes to lots of people, and some of them are marking it as spam, and it hits spamtraps, then it's spam". The act of distributing deliberately fresh domains as censorship evaders would then hit such a rule, especially if you do it at enormous scale via email.

Re: spamtraps, you're still assuming that some malicious entity knows where to find lists of spamtrap addresses. They aren't actually listed anywhere, right? Just scattered around the web waiting for crawlers to find them. So at some point Occams Razor applies.

Anyway, my point is simple. There are lots of safeguards in the big 3s spam filters. Those filters aren't perfect, but 99% of the time people complain, they're actually sending mail people don't want. It's possible this guy has found the perfect storm of edge cases that cause widespread failure - or it's possible that there aren't actually 400k people who want proxies emailed to them.

Re:5 second summary

[DRJlaw]

Once again, what I'm saying is, you're accepting everything the poster says on the assumption it's absolutely true. Spamtrap accounts don't reply to confirmation emails or click on confirmation links - ever. That's the whole point of them. Even if you're a malicious troll who got a list of Hotmail trap accounts from somewhere, how do you get control over them to confirm signup?

The confirmation email sent to the spamtrap account is itself the trap event. If the spamtrap does not confirm and the list does not send anything other than the confirmation email, then both the Hotmail management screen and his statements are still fully consistent. Do you want me to sign you up to his list to prove the point, or are you content with merely being lead to this very obvious conclusion through multiple Slashdot postings?

And yes, I am accepting it as true. It's trivial to follow the list signup procedure, respond to the confirmation message, and note that Hotmail even automatically categorizes the email as one from a newsletter. I unfortunately have to wait for the next mailing to confirm the unsubscribe link, unsubscribe, and then wait to not receive more messages, but it is consistent with everything that has been written, whereas you are merely guessing. And ignoring every other opinion to the contrary, e.g.:

http://features.slashdot.org/comments.pl?sid=3314491&cid=42276705
http://features.slashdot.org/comments.pl?sid=3314491&cid=42276435

The screenshot says more anyway. Judging from what he says the sizes of the mailshots are, it's a fresh IP that hasn't been used before. So the screenshot could have been taken before the reputation degrades.

You really insist on not reading the source material, don't you. "Hotmail allows newsletter publishers to view data about what percent of their messages to Hotmail users are being flagged by users as "spam," and when I looked up the stats for our IP, they showed a "complaint rate" of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list)." The screenshot states that it is for a 24 hour period. With a subscribership of 420,000, he's not going to be emailing 420,000 Hotmail users over 24 hours.

It's possible that this guy has done everything totally by the book and somehow has just got unlucky that his behaviour happens to closely match that of actual spammers.

In that case, why isn't the spam-identified content blocked when sent from other IPs/email accounts? "This only happened when sending from my own IP address at peacefire.org. It didn't happen if I tried sending a message from my Gmail account to a Hotmail address, even if the message contained one of the four banned domain names, so the issue probably won't reproduce if you try sending a test message yourself."

I think "sender is not following standard mail etiquette" is far more likely than some enormous conspiracy theory against him. After all, plenty of bulk mail senders do just fine.

In that case, why is the mailing list not blocked, but only certain content? "It turns out that out of the seven different URLs that I had been mailing to our users, four of the domains in those URLs would generate a "550 Message Contains SPAM Content" error when sent from my IP to a Hotmail address, and the other three did not."

What you keep saying is that you simply will not read what is going on, and will not address the actual problem, but by God you'll fight tooth and nail against anyone who dares to point that out. Bravo. You'll notice that others have picked up on it too. I replied because you were +5 Informative yet clearly wrong. That seems to have resolved itself now, so I'm done with you.

Re:5 second summary

[AvitarX]

But they could randomize the domain of the proxies, and email each one to a subset of customers, then they wouldn't all get blocked.

There are evil forces out there

[SpaceLifeForm]

Read my sig.

Really?

[Anonymous Coward]

Oh, someone is still using Hotmail and Yahoo?

Wouldn't it be just easier to do vice versa, block both and it would be a favor to us all.

gold standard for responsible mailing

[joostje]

Yes, verified opt-in is one requirement. But if you don't want to be marked as sender of SPAM, you should also make it *very* simple to unsubscribe. I know I've subscribed to a few lists, and at first read the emails, then ignored them, and eventually thought "should unsubscribe". But if that unsubscribing is difficult, I'll just hit "spam" in gmail (or whatever). I don't see the emails and more, and the sender gets blocked as spammer.

Re:gold standard for responsible mailing

[magic maverick]

Here's the latest email I got from Mr Haselton (with the email addresses changed though).
It's apparently very easy to subscribe. (Though it's not one click as you do need to enter your email address if you use the webpage option.) Is that good enough for you?

From: Bennett Haselton at Peacefire.org <webmaster@yahoo.com>
Reply-to: "Bennett Haselton at Peacefire.org" <webmaster@yahoo.com>
To: webmaster@hotmail.com
Subject: new Circumventor, in a new format
Date: Fri, 07 Dec 2012 04:00:02 -0500 (07/12/12 10:00:02)
Envelope-To: webmaster@hotmail.com

[You are receiving this because you subscribed to the Circumventor distribution list.
To unsubscribe from this list, click here:
http://www.peacefire.org/circumventor/cv-unsub.html
or reply with the word "unsubscribe" in the subject.]

Happy Holidays everybody -- your early Christmas gift enclosed:

https://www.kitepuddle.com/smart/

This Circumventor site is in a different format but it should work as well as the others. You *must* access this one with 'https' at the beginning of the Web address; it won't work with 'http'.

You can attempt to access the "regular" Facebook through this one, for example, but it might not work correctly; the most reliable way is to enter http://m.facebook.com/ on this Circumventor site, which will take you to mobile Facebook. Unfortunately Youtube still isn't accessible yet but we're working on it.

Don't waste too much time on those school computers - Santa's watching!

Bennett

***

"When I was in high school these twins got mono. They got stereo." -Demetri Martin

Peacefire.org
14615 NE 30th PL #10D, Bellevue WA 98007/blockquote.

Re:gold standard for responsible mailing

[Revotron]

You do realize you're talking about Valve servers, and this person who threw the temper tantrum on the mailing list is probably 12 years old and bought his server with daddy's credit card? It's no surprise, really. If you're looking for foolish, overdramatic, hot-headed people, look no further than Counter-Strike players.

Re: gold standard for responsible mailing

[Urza9814]

I'm on this particular mailing list, so I can confirm that he makes unsubscribing quite easier. Easier than any other list I've ever been on in fact. Every email has the following text as the first paragraph:

[You are receiving this because you subscribed to the Circumventor distribution list.
To unsubscribe from this list, click here:
http://www.peacefire.org/circumventor/cv-unsub.html
or reply with the word "unsubscribe" in the subject.]

Re:gold standard for responsible mailing

[Solandri]

The problem with designing something to be completely foolproof is that people underestimate the ingenuity of complete fools. Compare having to read the first four lines of that message vs. clicking a "spam" button on your email program. A not insignificant number of people are going to click the spam button, at which point the mail hosts start to classify it as spam.

I suspect what's needed is a "verified legitimate mass mailer" list. Sort of an inverse-spamhaus list. Legitimate mass mail services can somehow prove to the satisfaction of the major mail hosts that they're completely opt-in. Then the hosts know that if a user clicks "spam" for one of the mails sent from these services, that the message isn't really spam and the user is an idiot.

Re: gold standard for responsible mailing

[ortholattice]

This is still inconvenient, because the unsubscribe link requires you to enter your email address. This requires a redundant step by the user (who may make a typo).

I have several email addresses forwarded to one place, and when an email says it was sent to "undisclosed recipients" I have no idea which one I need to unsubscribe without a tedious analysis of the header. I don't think an average user could do such a header analysis.

Finally, I'm just plain suspicious of any site asking me to type in my email address. If they already know it, why are they asking?

Is this a repeat?

[rudy_wayne]

I could swear this same guy was complaining about problems with his "I swear it's not spam" mailing list several months ago.

Re:Is this a repeat?

[ADRA]

Sounds like the same guy. At least the exact same scenario...

Re:Is this a repeat?

[Bill Dimm]

Yep, 2 months ago

Re:Is this a repeat?

[PRMan]

Before, he complained about the problem. Now, he is sharing what he found out.

Re:Is this a repeat?

[czth]

No - that was about Spamhaus (which is mentioned again in this current writeup but not the main point); this scenario is about Hotmail/Yahoo! having (he believes) a shared secret domain blacklist that applies to the content of emails.

Re: Is this a repeat?

[Urza9814]

Hence the words "frequent contributor" at the top.

I've been using his service for at least six years. It's as far from spam as you can get. Certainly far less spammy than the emails from newegg or Amazon (which is among the worst!) or any of the others that have no problem at all getting through spam filters. Multiple ways to unsubscribe right at the top of every message, verified opt-in, low volume, no embedded tracking features (all plain text), and legitimate content.

So what the hell else do you want? Should he start collecting phone numbers and personally call each subscriber to confirm before sending each message???

Distribute the load

[betterunixthanunix]

Part of the problem with spam fighting is that we are not distributing the spam fighting load. Hashcash distributes the load somewhat, in that it forces spammers to use more resources to send out their message and can slow them down somewhat. A distributed filtering system that allowed people to volunteer CPU time and bandwidth to filter spam (with some system of gaining the trust of an email server) might also work; imagine if hundreds of millions of people were relaying / filtering 100 messages per day.

Re:Distribute the load

[afidel]

There are several distributed reputation filter systems but they are all commercial AFAIK.

Re:Distribute the load

[nabsltd]

Hashcash distributes the load somewhat, in that it forces spammers to use more resources to send out their message and can slow them down somewhat.

Unfortunately, until you get to a significant number of bits, hashcash doesn't take all that long to compute, and you can pre-compute them.

I use 23-bit hashcash on all my outgoing e-mails, but if the address has been sent to before, there is likely a pre-computed 25-bit hashcash waiting. I use idle server time to pre-compute for any address that has been sent to from my servers. Since the hashcash expires in 25 days, I don't have to do this very often unless the recipient is a frequent one. Then, to keep the database small, I remove addresses from the "sent to" table unless they have been recently active, where "recent" depends on the total amount of activity to that address.

Re:Dude

[jellomizer]

I hate to use the if you were legit then you wouldn't need a proxy argument. However If he was using email the way most services want you to use it, he wouldn't have a problem.

Email was meant for a Person to send a message to another person or a small group of people, usually with people that you have some connection too.

Independent verification of verified/double opt-in

[bersl2]

I used to work security at a major hosting provider. If we got complaints about your mailing list, the first thing we'd do is ask you about how you got your list, to see if it complied with our requirement for verified opt-in lists only. We'd also sign up ourselves or check logs and code, because customers always lie (except when they don't).

Right now, I'd apply the same standard of skepticism. I understand that revealing such things would make your proported aim of censorship circumvention hard, but I'd still like to hear independent verification from someone who can reasonably demonstrate the depth of their commitment to opting in.

apple has a "secret list" too it seems

[crisper]

Apple has a "secret list" too it seems, I had one case of this with one domain. When I called I explained to normal tech support the issue, they had me escalated where I explained the issue in a bit more detail. Within an hour or two I had a call back from Apple support telling me that the domain had been removed, I didn't pry any more I just figured since they have the right to deny email for whatever reason then have the right to do this. This came after looking over logs, and some packet captures, to make sure it was being delivered to their servers before making the call to Apple. Nothing indicated any type of failure/deferred/blocked from looking at those logs/captures.

Not a hard problem to solve for PGP.

[DamnStupidElf]

Even S/MIME might meet your needs in this case. Encryption is cheap enough even for mailing lists now.

Re:Not a hard problem to solve for PGP.

[Megane]

The information is already being sent in clear text. The only reason to encrypt it would be to avoid automated blocking.

Since all the domains are composed of two English words, they could be sent as two words, with a space in between, and possibly another codeword to indicate .com, .org, or some other TLD. That would remove the "scan the message for anything that looks like a domain name I haven't seen before and scan it for open proxies" angle.

Perhaps he could use rot13, or a substitution cipher with the key as the first line. or some similar encoding that is easily decoded both manually and with a 10-line C program. It might even be possible for every message to use a different key, and the automation necessary to do that would also let you give everybody a different random subset of the domain names. These are meant to be good for 1-4 weeks, so taking 5 minutes to decode them should be insignificant.

Getting even more tough about using low-impact steganography, the text could be converted into small 1-bit .gif encodings of the domain names and inserted as MIME data. This would be hard to reject automatically by means other than the fact that they are encoded, but a decent GUI mail reader would be able to display them. If the font is kept consistent, they can even be converted back to plain text.

It's a balance between something easy enough to decode, yet difficult enough to be annoying for automated detection. Remember, true spammers need to have their message visible as plain text because their recipients aren't motivated to decode it, but someone who really wants the data can be expected to take five minutes to decode it.

Re:Not a hard problem to solve for PGP.

[DamnStupidElf]

Elliptic Curve ElGamal encryption is pretty fast.

Re:Dude

[lister king of smeg]

Its not Spam if you opt in. Spam is unsolicited. For this you have to request. Now is it possible the guy is bull shitting that part sure, however if we accept that the articles are bull why bother to read them?

Re:You are a spammer

[glaurungn]

He sends proxy address to people that requested that information. He send it weekly because the proxys are blocked.

You missed the point.

[CaptainNerdCave]

The issue is that no one on the list of recipients got the chance to refuse the message.

How can you be certain he is not part of an internet forum dedicated to anonymity? What if he were sending an email with updates on domains that are security risks to a long list of subscribers to his IPsec newsletter?

There is a very long list of possibilities for what he could have been doing that was perfectly legitimate. Basically, USPS, UPS, FedEx, DHL, $common-carrier should not read your text-only message to determine if there is any information they don't like, and refuse to deliver it based on that alone.

Re:Dude

[Anonymous Coward]

Do you people not understand the concept of an email newsletter? For instance, I am subscribed to NASA Tech Briefs 's email newsletter, which purports to have an audience of over 77,000. Being a newsletter, of course those emails all have "the same web address in them" -- they're the same bloody content. This has been going on for decades (they've been a big thing since home users who never heard of usenet started getting internet access...), and as long as it ONLY GOES TO PEOPLE WHO VOLUNTARILY SUBSCRIBED, it's NOT MOTHERFUCKING SPAM! If your spam filter flags this, your spam filter is broken. Spam= UNSOLICITED bulk email, not all bulk email.

Re:Dude

[magic maverick]

After the last article I signed up for the service of getting emailed the proxy sites. Guess what, I've had no problem. I've not recieved any spam to the email address I used. I've only received emails that I specifically requested.

So, ah.

Dude, you're a fucking idiot. Hotmail and Yahoo are not doing anyone good... Get lost!

If someone is running an incredibly popular opt-in email list, that doesn't automatically make them a spammer. In fact, because it's all opt-in it makes them the opposite. It's solicited, not unsolicited. Mr Haselton is one of the good guys, and you are a moron if you can't see that.

Re:Dude

[crypticedge]

I have to use a mail proxy, not because I spam (we send about 20 emails a month) but because verizon blocks port 25 outbound, and won't let me get a static IP at home for my mail server.

I pay 20/year for my mail proxy, gives me 200/mo that we never hit.

did you think of...

[sithlord2]

- Implementing DKIM?
- Implementing SPF?
- Make sure the sender address doesn't bounce?
- Make sure you don't open thousands of connections to the receiving party for each recipient ? (in case of yahoo, hotmail, gmail, ...)
- The contents of the e-mail is not considered spam? (provide unsubscibe link, no big images included, etc...)


Setting up a mass-mail infrastructure is not to be taken lightly. There are lots of reasons why you could be listed as a spammer. That's why most companies outsource their their mass-mailing to 3rd parties like MailJet, MailChimp, SendGrid...

Re:did you think of...

[nabsltd]

That's why most companies outsource their their mass-mailing to 3rd parties like MailJet, MailChimp, SendGrid...

As an e-mail server operator, I'm glad they do, as it makes it easy to block all the spam from those companies at the server level.

I've been added to e-mail lists without my permission quite often because I had to provide an e-mail address for the company to send me a bill or other actual important e-mail. They pass the e-mail address on to these third party companies without any confirmation on my part. Then, when you do go through the unsubscribe process, those companies claim that it might take 4-5 days to remove you from the list, when in reality we know it should happen instantly. This is so that when you keep getting the spam for the next week or so, you might not report it, or maybe if you are truly stupid, you might buy some of the junk that is advertised.

In addition, once you have any dealing with a company like MailChimp, they can play the "previous business relationship" card when sending you "something you might be interested in". And, since none of the companies you list are confirmed opt-in (which requires that they send you a first e-mail when you supposedly subscribe, and unless you click a link or reply to that e-mail, you do not get added to the list), they can play fast and loose with claims that you signed up for something that you did not.

Re:You are a spammer

[niiler]

His behaviors are _similar_ to those of a spammer in number only. Having visited his site: http://www.peacefire.org/ it seems that he gets his email list from people subscribing to it on his site. If I understand it correctly, people who sign up for this list are looking for regular updates to proxies so that they can avoid censorship. As proxies are discovered by governments or certain companies , they are blacklisted, and new proxies must be created and sent out to the interested masses:

"Of course, employees of blocking software companies have gotten on this list as well, so they add our sites to their blocked-site database as soon as we mail them out, but in most places it takes 3-4 days for the blocked-site list to be updated. So the latest one that we mail out, should usually still work. "

Now it could be that there is a better way of doing this, but it seems to me that no matter how this game is played, constant updates to users should be the norm...

Now that I think of it, perhaps a Firefox extension could do the trick. Signed extensions can be updated automatically. The extension could have obfuscated URLs that are decrypted with something like this: https://addons.mozilla.org/en-US/firefox/addon/domcrypt/ and then wired in to automatically select an available proxy from the current batch. Not perfect by any stretch of the imagination, but it solves the "spam" problem. Also, it maybe easier for users and harder for censors? Crap... now I'm not going to get any work done...

I don't understnd the animosity here

[Anonymous Coward]

Early on (before I quit reading) the OP said:

  It turns out that out of the seven different URLs that I had been mailing to our users, four of the domains in those URLs would generate a "550 Message Contains SPAM Content" error when sent from my IP to a Hotmail address, and the other three did not. The message didn't have to contain the banned domain in the From: address; the message would get blocked if it even mentioned the domain anywhere in the message body.

It seems to be treating his email as spam even when he sends one email to a single address.That isn't spam.

Re:I don't understnd the animosity here

[coofercat]

But it is spam if you repeat that same process 10000 times (or however many). Just about all abuse detection works this way - it's a matter of counting how many times you do something per unit time (even if the thing you're doing is supposedly legitimate). Once you trip the threshold, you get banned for a set period of time. It's possible that once you trip the email threshold you're banned for weeks, months, years or permanently (unless a human revokes the ban). Heck, I do something similar on my website to get rid of the vast swathes of Chinese botnets that seem to want to tell me all about Ugg boots and the like.

It's entirely possible also that this guy could have completely successfully emailed the entire list of 7 proxies to his friend in a single email from his 'dodgy' domain, including the words "viagra" and "gold dust" and "nigeria" when the domain was first set up. He might even have been able to send the same list to 50 of his friends. However, when he sent it to the 51st, he tripped some automated checks, which presumably he failed, and so got banned.

It's possible he'd reduce his spamminess to these providers if he spent a week sending his weekly email. That is, if he's sending to 700,000 people, send to 100,000 people each day, and spend 5 seconds or so sleeping between sending each one. Even that's probably enough to trip the thresholds, but you get the idea.

Use DKIM

[pr0gr3sR]

Had a similar problem with Yahoo... Implemented domain keys and signed all my outbound mail and it fixed the problem.

Re: Dude

[Urza9814]

I've been on his list for around six years, and as far as I can tell, everything he says in the article is 100% accurate.

Also worth noting that he submits articles about these things to Slashdot quite regularly. I recall one a few months back where he was first considering this exact experiment. I'd go find it, but I'm posting from my phone.

Optimal != Perfect

[Anonymous Coward]

The "Optimal solution" isn't "perfect".

There are always tradeoffs, and the power of the free market is that it is relatively effective at weighing different options.
It basically brute forces the answer to any question. It's messy, ugly, often inefficient, but it works.

Re: Dude

[Urza9814]

FWIW, I'm on that list. And if I was using hotmail or Yahoo I would be PISSED about missing those messages. Been on it since highschool where I used them to bypass the school's web filters (occasionally teachers would even promote these sites because we literally couldn't do our work without them); today I still use them for testing and occasionally at work if, for example, I need a document from scribd (why that is blocked I'll never understand...)

Re: Dude

[Urza9814]

[Quote]Email was meant for a Person to send a message to another person or a small group of people, usually with people that you have some connection too.[/quote]

[Citation needed]

Email is Electronic Mail. You have large mailing lists like these with physical mail; you'd have to be an idiot to have thought something similar wouldn't be developed with email.

Re:Dude

[Rakarra]

I hate to use the if you were legit then you wouldn't need a proxy argument. However If he was using email the way most services want you to use it, he wouldn't have a problem.

Email was meant for a Person to send a message to another person or a small group of people, usually with people that you have some connection too.

Then how do you send a message to a large group of subscribers (let's ignore the spam angle for now and say these people want the updates) notifying them of site updates, special offers, alerts, or whatnot. I don't think it's enough to say "well they should just go to the site and check it when they want to." First, I don't want to call up every web site I might have signed up with every day. I just don't want to go through that hassle. I would end up not doing it. Email is perfect for me, I can scan it quickly for things that interest, delete it all when done... no hassle for either myself or the services. What would you suggest to replace this that would work better?

Re:Dude

[wendyg]

Bennett Haselton is no spammer. He's been involved in anti-censorship for nearly 20 years; he began in high school by investigating the block lists operated by the filtering software installed in many schools and libraries.

Not a spammer.

wg

Top secret :Google gmail has faulty BLACKLIST too!

[Anonymous Coward]

Top secret fact :Google gmail has faulty blacklist too!

I have has a website I have had for over 12 years at niftyspot.com

Its mostly ARM CPU assembler link info and has been for ages

Gmail blocks mail from its server from google group lists I subscribe to among friends! Some messages delayed for 5 hours, some sp*m blocked

why? because the ONLY domain I know not listed in the first 1000 results is my website!

Google has a vendetta against me... my domain!!. I used to be number one search result (www.niftyspot.com) on EVERY search engine I know, and various listings sites, for the phrase "niftyspot" and also for phrase "nifty spot"

Now I am BANNED and have been for months, but chinese google has me number one result. As do 4 other search engines. But google DELISTED MY SITE ENTIRELY FOR MYSTERIOUS REASONS!!!

Google has a blacklist. And it has NOTHING to do with anything logical. The domain has been a good citizen and compliant for over 12 years, and the emails sent are a handful a week to a handful of people.

TRY IT YOURSELF IN BING; Then try finding my site in google’s crappy search engine. Or try other search engines besides bing

My www.niftyspot.com is delisted as destination search ENTIRELY off google, but for years and today ALWAYS number **ONE** in other search engines (http://bing.com, http://yahoo.com, http://duckduckgo.com, http://www.baidu.com)

Oddly enough, I stopped being number one and was delisted on just Google and blocked in google gmail after a stranger possibly in India wanted to negotiate to but niftyspot.com from me. I don't accuse him, but I sold it to him this morning because I do not know how to fight google.

I am enemy number one to google

even islamic terrorist websites are listed, and asian drug distributers are on google, and competitors, but not www.niftyspot.com

check the other searach engines and chinese google if you doubt me.

All i say is 100% fact and true. I am and was number one everywhere , but BANNED and DELISTED off of googles faulty or evil technologies.

I am the only known domain delisted off google AND harrassed in gmail as well

So quit picking on Yahoo and Hotmail. Google does evil too!

Re:Dude

[mcl630]

He has even sued spammers.

Re:conspiracy ! deliver my spam

[multicoregeneral]

Dude, you're spelling it wrong. That's going to be flagged for sure.

His emails simplify the blacklister's job

[Goldenhawk]

Ironic. Almost all blacklist providers keep proxy sites on their default "bad sites" list. Were I running URLBlacklist or similar, I would simply sign up for his email service and make a point of adding every web domain spotted in his emails. Almost an instant kill for the blacklist provider; by the time email recipients can act on the information, it's already been blacklisted.

Re:You are a spammer

[kimvette]

Want to know who sends mass email in batches like that?

Apple, Microsoft, NewEgg, Amazon, Zappos (an amazon company), Woot (another Amazon company), ZD Net, and so on.

Not every large volume emailer is a spammer.

Re:Dude

[PlusFiveTroll]

Why not use Verizon's mail server?

Most likely, because it sucks great big donkey balls. Now, that said I don't use version so I don't know for sure. What I do know from working for one of the top 10 ISPs (size wise) in the country is, most big ISP mail servers suck. Send any attachments of any size and they're apt to be blocked, get stuck in the queue, or just go in to the blackhole. Other issues are that the ISP might flag or block as your messages as spam because you want to send 200 messages on friday. And you have to put up with their filtering and blocking choices, that may not meet your needs.

Re:Dude

[Anonymous Coward]

I think your scenario is exactly what RSS is for, right?

Except the summary is probably wrong.

[Medievalist]

I wouldn't be suprised if it's just Bayes. The majority of messages with links leading to those registrars' domains were categorized by human readers as spam, so automated bayesian analysis picked it up.

As long as you have Internet governance that is primarily concerned with eliminating certain forms of political speech (Great FireWall of [insert name of nation here]) rather than ensuring a free market and fair trade, you're going to have this problem. The same low-rent registrars are going to be used for criminal spam as for legit filter avoidance technologies, because they are looking for the same service (temporary domain names at minimum price).

Re:Except the summary is probably wrong.

[AdamWill]

"I wouldn't be suprised if it's just Bayes. The majority of messages with links leading to those registrars' domains were categorized by human readers as spam, so automated bayesian analysis picked it up."

Try reading the longer summary.

"It's conceivable that one or more of the domains might have gotten blacklisted as a result of Hotmail or Yahoo users clicking their "This is spam" button. However, Hotmail allows newsletter publishers to view data about what percent of their messages to Hotmail users are being flagged by users as "spam," and when I looked up the stats for our IP, they showed a "complaint rate" of less than 0.1% (usually the rest of people hitting 'Junk Mail' to unsubscribe from the list). Assuming that the complaint rates are similar for Yahoo Mail, it's unlikely that the domains got blacklisted as a result of user complaints, unless the blacklist trigger has a ridiculously low complaint threshold."

Re:Except the summary is probably wrong.

[SomePgmr]

That's not what he was suggesting. Reread the post you replied to carefully.

2 days

[asmkm22]

How in the world did it take you two days to figure out Spamhaus was blocking your stuff?

Save yourself some time down the road and just go to mxtoolbox.com. Enter the domain name it and can check all kinds of things for you. If a list is blocking it, you can get details as to why. In the past I've seen various reasons, but most are pretty detailed and provide quick access to the forms you need to get removed.

As for your idea of a secret shared blacklist between hotmail and yahoo, it sounds more like it's just a dynamic content filter that pulls data from spam lists to prevent the propagation of bad links. I doubt it's 100% realtime, so after getting Spamhaus to unlist you, yahoo and hotmail need to wait for their next content filter update to see the changes.

Anyway, I just thought it was a little weird that it took you 2 days to even get to step two of your problem.

Re:Dude

[crypticedge]

Because they won't let me relay my own domain through it, it sucks big fat donkey balls and it's subject to far tighter restrictions than what I use to send outbound.

I run all my outbound mail through a good spam filter (that forces all outbound to be scanned, regardless if it makes it through the mail server) and have a fairly open file size limitation (20MB, compared to last I tested Verizons 5 MB)

My outbound proxy has no size limitations, my outbound proxy handles all blacklist issues, and my outbound proxy handles all RFC compliance issues (it's dyn, they aren't a bit player)

In the end, filtering it through dyn just works, trying to send through verizon ends in frustration.

Re: Independent verification of verified/double op

[Urza9814]

Been on the list since late 2005 and I never delete an email, so I can confirm.

You subscribe at his website and you get a confirmation request email. You confirm, and it sends another message confirming that you've been added. The content is legitimate, the volume is fairly low, every email gives two unsubscribe methods in the first paragraph of the message (click a link or reply with unsubscribe) and all messages are plain text.

Re:Web page

[Ksevio]

Read the summary and you'll find out!

Here's the real issue

[CanHasDIY]

Preface: I am not taking the side of the spammer here. You keep that shit out of my inbox, fucker.

That said, the real issue is the censorship of people's messages without their knowledge or consent. Granted, nobody wants to have to filter through millions of V1@gr@ ads just to read their mail, but on the same note, nobody wants someone else going through their mail and arbitrarily deciding what will and will not be delivered. I understand the purpose of the spam filter, and am glad it's there - but a secret spam filter? Not cool - as far as I know, those who administer said filter may decide, 'you know what? I vehemently disagree with the political philosophy of Grassfire/MoveOn/other political group, let's add them to the secret blacklist.'

Real world analogy - think of Yahoo/Hotmail as UPS - just because it's a private company doesn't mean they have the right to go through the shit you ship (or do they? I, personally, don't ship a lot of stuff...).

Nothing new

[wukka]

For years I've noticed that Yahoo email will bounce emails I try to send which contain certain URLs. Legit websites, no spam, no malware. Next year Ixquick.com/StartPage.com are supposed to offer StartMail.com as a webmail alternative. I am anxious to try it out.

Re: All I can say is

[Urza9814]

Double confirmation of opt-in, two methods to unsubscribe at the top of every message, legitimate content, low volume, and no tracking features (all messages are plain text).

I've been a subscriber since 2005 (just looked that up for a different comment; I never delete email) and this list is as far from spam as you can get. Shit, I've replied to those messages and gotten a response from him personally (Still have those too.)

If you still think this is spam, then apparently by your definition I have literally never received an email that wasn't. He provides a great service and runs a clean list. I'm a huge fan if you haven't noticed...

Re:Web page

[c_sd_m]

Because (1) the webpage would get blocked for the people who need to use proxies and (2) you don't want to give everyone the same proxies.

You are missing a fundamental point here

[larwe]

(If you want a tl;dr version of my post: Don't whine that a free service run by someone else isn't guaranteed to meet your business objectives). Actually a couple of points. The first one is, bulk email is bulk email and it really doesn't matter how "good" an actor you are, whether you are opt-in, etc. You are not sending individual hand-calligraphed invitations to a royal wedding, or any other kind of lovingly crafted correspondence; you are sending bulk flyers which most people won't read. It is possible, maybe, to classify CONTENT algorithmically. It is impossible to classify intent (CE vs UCE). The second one is, email is NOT a guaranteed service. Broken cable. Bad MX record. Squirrelly hard drive in a mail relay somewhere and your email is gone. Why are you complaining about one particular failure here? It seems that people who actually want to have a certain level of service guarantee (in this case, avoiding a legitimate antispam measure - regardless of how effective it is) - need to pay the piper. You resent this, and believe you should get an above-baseline level of service for free. Can you hear my boiling tears raining down on the volcanic desert of the Internet? The third one is: you paid nothing to transport the email, you only paid to squeeze it out the urethra of your computer into the public internet - from then on all the transport is ON SOMEONE ELSE'S DIME. If they choose not to forward it, that's your bad luck. You have no recourse. Stop whining. I have zero sympathy for any problems experienced by anyone distributing bulk email for any purpose. If you told me your correspondence with Aunt Franny was being swallowed by a demon at Gmail, or something of the kind, I'd be more sympathetic, because that's a realer problem (though my second and third points above still apply to the Aunt Franny case). It is specious, maybe arrogant, to pontificate that consumers "lose" or the "free market fails" when bulk email doesn't reach its (mostly uncaring, if not downright resentful) endpoints reliably. In summary: Grrrr.

Re:Dude

[sexconker]

Its not Spam if you opt in. Spam is unsolicited. For this you have to request. Now is it possible the guy is bull shitting that part sure, however if we accept that the articles are bull why bother to read them?

They're spam because people are clicking the "THIS IS SPAM" button.
They're clicking that button because they don't want the fucking emails and he keeps sending them.

WTF upvotes for baseless aspersions

[Onymous Coward]

This man is running a list (among many other activities) supporting individuals' rights to information freedom under repressive governments and you're implying he's either incompetent or, worse, underhanded?

This is inane.

And how much effort is required to fucking test this?

Thank you. A confirmation message has been sent to address redacted.
YOU MUST REPLY TO THAT MESSAGE, in order to be subscribed so that we can notify you when new Circumventors are set up. Almost 50% of our subscribers forget to reply, and as a result, do not get added to the list. If you do not reply to that message, then your address will not get added!

What causes rudy_wayne and those who upvoted his post to like the idea that Bennett Haselton is spamming and lying about it? And is their credulity what keeps them from performing such an easy test? Whatever the cause of the inanity, how can we discourage this problem in the future?

Re:WTF upvotes for baseless aspersions

[Onymous Coward]

You are correct. Thank you for your precision.

The primary method of subscription was presumably the thing in question, and so was the thing tested.

More detail: this test proves that the subscribe form on the peacefire.org site does require opt-in. I assume this is the primary, possibly only publicly accessible means by which persons can attempt to add addresses to the list. Your precise reckoning does highlight the possibility that there may be other means and that they may have been subverted. I believe this is unlikely, and I think it's likely Mr. Haselton has investigated the possibility.

Perhaps Mr. Haselton will do something like a binary (or 6-ary) search for which addresses may be reporting to the URIBL and trace how those addresses were added, should future domains listings happen.

Re:WTF upvotes for baseless aspersions

[theArtificial]

Follow the can spam laws and ya have nothing to worry about.

An associate of mine created an email delivery system which many companies use and one of their larger clients sends out about 25million messages a day to their opt-in lists (it's a newsletter). There are users who opt-in and then mark the message as spam (forgetting they signed up, or perhaps even mis-clicked). This does happen. What then? As a company you're following the rules...

Re:WTF upvotes for baseless aspersions

[Onymous Coward]

>inanity

You keep using this word. It does not mean what I think you think it means.

Correct. Rather, it means what I actually think it means.

Re:You are a spammer

[mrbene]

Now it could be that there is a better way of doing this, but it seems to me that no matter how this game is played, constant updates to users should be the norm...

Now that I think of it, perhaps a Firefox extension could do the trick. Signed extensions can be updated automatically. The extension could have obfuscated URLs that are decrypted with something like this: https://addons.mozilla.org/en-US/firefox/addon/domcrypt/ and then wired in to automatically select an available proxy from the current batch. Not perfect by any stretch of the imagination, but it solves the "spam" problem. Also, it maybe easier for users and harder for censors? Crap... now I'm not going to get any work done...

There are multiple benefits of email delivery that aren't present in the Firefox Addon model:

• It's push notification - the updates only go out once. Firefox Addons are a pull - a server has to handle all the clients requesting updates (and sending the appropriate subset!).
• It's more difficult for the people that this list is supposed to enable to bypassing of to automate the immediate blocking of the new set of domains.
• It natively enables two-way communication at a human level.

If I were the OP, I'd consider moving to an encrypted blog method of delivery (still via email), but doing it while being very conscious of the level of technical know-how of the target recipients.

Re:Hotmail and Yahoo

[Megane]

Their very lameness and ubiquity is what makes them perfect for people living under oppressive regimes. When "everybody" uses a mail service, it becomes harder to block it without a lot of people noticing and getting pissed off. When they are so ubiquitous that even members of the regime use them, it's even better.

It's spam, of course.

[Animats]

If you want to distribute a "newsletter" to real subscribers, set up an RSS feed, or a Twitter feed for little stuff. Readers can then subscribe if they want, and they can unsubscribe without having to beg to be taken off the list.

This clown sent 420,000 emails. Of course he's a spammer.

Re:It's spam, of course.

[Anonymous Coward]

Then so is Scott Adams with his Dogbert newsletter. According to Wiki, he used to send out 500,000+ mails at times.

The bastard!

Re:It's spam, of course.

[Anonymous Coward]

Sending 420,000 emails to opted in subscribers does not make him a spammer.

What I've figured out with Hotmail

[Predius]

I admin a few mail servers. I've run into trouble with Hotmail. Here's what I've learned:

First, there are a ton of url / domain blacklists available out there, no need to suspect a conspiracy within Hotmail and Yahoo. That said, I know they also maintain in house IP and domain based blacklists, along with full url blacklists. No idea if they share but I actually doubt it as that potentially weakens their competitiveness with the other email providers. Hotmail also uses a paid whitelist service too via an 'independent third party', although certain blacklist levels can even override that paid service.

Second, Hotmail splits mail up into three categories now, legit mail and spam which we're all familiar with, plus what they've dubbed 'graymail'. In short, graymail is legit opt-in mail that the user just never bothers to read. Thats right, your quadruple opt in email can be treated like spam by Hotmail if your users never bother to look at it. Generate too much, you're treated as a spammer. Can SPAM compliance or not, they don't care.

Third, if you manage to get on Hotmail's IP blacklist, there is no recourse that I can find. Their policy is tough expletive, move your mail server to a new IP or go away.

As far as the complaint level stats you can view through their Postmaster tools, they only show two of the three stats their system works on at the IP level, the complaint rate (people flagging mail, I *think* VIRI mail also counts in this column) and filter hits percentage, although this one is obfuscated to try and defeat spammers trying to tune around it. The missing stat is IP reputation, based on those first two stats over time along with external and internal RBL data. So when you DO setup on a new IP, it'll take awhile for their system to actually accept mail from you. You can subscribe to a feedback loop program, but that shows another issue with Hotmail:

They have no concept of traditional mail relays, they expect all individuals to be sending via Hotmail, Gmail, Yahoo, etc. All other port 25 traffic destined to them must be from commercial list serves. At least that's the impression I've gotten from going through all their postmaster policies and dealing with their ticket system. If you try to explain the idea of an ISP relay for use by people within that IP block, they just ignore it and resume pestering about opt-out notices, etc.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Screw that

[pavon]

And should the HFH and ACLU and all the other newsletters I subscribe to be blocked as spam as well? They send far more than 400k emails a month. Email is more convenient than RSS or worse Twitter, and is newsletters are a perfectly legitimate use of the medium.

Re:Screw that

[BitZtream]

And yet they seem to be able to do all the right things to not get blocked.

Perhaps you should consider that before you say anything else.

who cares about blacklists?

[someones]

they have a very long history of false positives and actually noone should be using them today anyway.

Re:Seriously...

[RKThoadan]

I've never wanted mod points more than I do right now to mod you down as well as the many other idiots like you who have made similar comments.

Read his entire article and you'll understand why he's doing that. I'm not going to tell you because your brain obviously needs the exercise. If you're too lazy to read then don't bother commenting. I'd also recommend actually reading the rest of the comments for the many people here who have subscribed to his list for years and verified that he's 100% legit.

There is nothing commercial or unsolicited about his service and it's a vital and important tool if value minor things like freedom. A single one of his e-mails is probably more valuable to the world than your entire pathetic existence.

Perform listwashing, just like spammers do

[Khopesh]

Ironically enough, you can isolate the "moles" by listwashing, just like spammers do for spam traps.

You've already started the process: you know that three sevenths of your subscriber base is probably safe. In your next run, make sure each of the remaining four groups is subdivided again. Each time you find a group that isn't a mole, you've reduced the potential mole list. Eventually, you'll have just a few accounts and you can silently drop them from your service (or confront them, your call).

There was also an earlier comment on spammer abuse of your proxies that I'd like to expand upon. While it asks you about proxying port 25, there's also the potential for abuse with respect to port 80/443: 419ers are increasing their use of proxies to hide their identity from free webmail providers so they can get free passes on sending spam. If you're better at cracking down on them (by e.g. blocking access to yahoo and hotmail on your proxies), you'll probably have better luck overall.

Maybe you can combine the above two ideas: groups of subscribers known to contribute to getting blocked will get domains whose proxies can't use freemail.

Wrong.

[kjs3]

> Pretty much the entire financial cost of sending email, is attributable to the failure of the "free market" to motivate email providers to deliver non-spam emails into their user's inboxes.

To quote Pauli, this is not even wrong. The central fallacy to this entire anti-capitalist rant is that there's some nearly perfect solution to spam that the "market" participants are conspiring to deprive the consumer of. This contention is, not to put too fine a point on it, as deliberately dishonest as similar claims about running cars on water or perpetual motion machines. Spam is an arms race, not a problem with a "solution" that we've just been too lazy to find. Once you dispense with that fallacious premise, this entire screed can be summarized as "I'm butthurt because Spamhaus/Yahoo/Hotmail blocked my spammy-but-not-spam-because-I-said-so emails and they won't take my call" all wrapped up in a "won't someone think of the children...err...dissidents!" bow.

Email account providers have as many automated, heuristic-based blocking techniques as blacklist based. Have you considered that you might have tripped one? Like...a domain that was registered less than a week ago, first mailing we got from them was a carpet-bomb, content we've previously spotted and identified as spam? I mean, it's a lot less sexy than claiming there's a villainous corporate cabal in the back room twirling their mustaches as they condemn some hapless dissident to a life of Internet ignorance, but it is possible.

Re:Wrong.

[jibjibjib]

RTFS. He's not claiming that there's an almost perfect spam filter being suppressed by a conspiracy.

He's making the very plausible claim that spam filters naturally err on the side of false positives, to the detriment of the users, because false positives are a less visible problem than false negatives.

Re:Wrong.

[kjs3]

I did read the article, thanks. He's claiming that there's a solution to that problem that isn't being pursued. That's false. Visibility is a red herring, which you nicely went after.

Re: Dude

[BitZtream]

So you fail to understand why running open proxies gets you black listed?

Hell they don't have to have a 'secret domain blacklist' he freaking mails them the domains to ban.

Re:Dude

[BitZtream]

Not a spammer != Not facilitating spammers.

What do you think spammers do with a nice list of open proxies? THEY SPAM FROM THEM. He's distributing a list of spam producing sites and you're shocked that the list gets blocked?

Re:Sigh

[psmears]

If the guy does not want his stuff flagged as spam he should try sending e-mails with the same address people opted in for.

He is doing that - but the mail is being blocked purely because it mentions certain domains in the body of the message.

Re:If you follow the Can Spam Laws

[BitZtream]

Its impossible for him his lists content to follow CAN-SPAM laws ... he lists open proxies so people can not be censored ... do you think the spammers don't use them to get around blocks as well?

He facilitates spamming (not intentionally, but does none the less) so he's being blocked as such.

Math

[Cow Jones]

I registered seven new domains and sent each domain to one seventh of the list; the list contains about 420,000 addresses, so each one went to about 60,000 people. (Each new site is only sent to a random subset of the list, so that a blocking company can't just subscribe one address to the list and block all new sites as soon as they're mailed out.)

So somebody who wants to block all the proxies would have to subscribe several times in order to get the full list (it's not like multiple subscriptions would be noticed on a list with 420k recipients). I was wondering how effective this method was. Here are my results, in case anybody else was wondering:

With 20 subscribed addresses, the chance of getting the full list is 70%.
With 30 subscribed addresses, the chance of getting the full list is 93%.
With 40 subscribed addresses, the chance of getting the full list is 98.5%.
With 50 subscribed addresses, the chance of getting the full list is 99.7%.
With 100 subscribed addresses, the chance of getting the full list is 99.9999%.

Seems like this method of evading the censors is only effective if they're not smart enough to write a couple of simple scripts.

CJ

Re: Dude

[RR]

I need a document from scribd (why that is blocked I'll never understand...)

The McAfee block thing tells me that Scribd is a piracy website. Scribd hosts user-uploaded documents, and some of those documents are copyrighted by various companies.

Half of these people

[future assassin]

ragging on the article OP probably haven't got a clue that people actually use other methods than Twitter/Facebook to update users about their service. Why is everything have to be in the open social communication. If someone was tracking his twitter it would take a few seconds to block all the proxy domains.

Re: Dude

[Urza9814]

These aren't regular proxy servers. They're web proxies. Just a regular web server running one of the standard web proxy scripts -- used to use cgiproxy, then phpproxy, now it looks like he's switched to Glype proxy. It's just a generic web server running a generic PHP script -- I fail to see how that is cause for a ban.

Long-Lime Anti-Blocklist Crusader Still At It

[thenick58]

Good grief. Is Bennet Haselton still at it? I first crossed swords with him over a decade ago when I was Executive Director of Mail Abuse Prevention Systems, the famous or infamous, you pick, maintainers of the Realtime Blackhole List. Haselton should stick to what he knows best, and that is blocking of *websites*. I see his knowledge of blocking technology by email service providers is as dismal as ever. Hotmail and Yahoo have every right to block email they perceive to be spam. If they did not do so, their servers would crash under the barrage of email arriving every second. They spend countless expensive CPU cycles just *blocking* the spam from their networks. Is the system perfect? No. That is why both organizations have staff to deal with the "false positives" -- another needless expense for which you can thank the spammers. I know the anti-spam staff at both Hotmail and Yahoo. Members of their staff spend their entire days reviewing and responding to complaints about false positives, as well as tweaking the anti-spam filters. Forgive me if I don't shed any tears if the staff doesn't respond to Mr. Haselton's demands just because he stamps his feet. They're dancing as fast as they can. And you know what? Both Yahoo and Hotmail are *free* services! Imagine that! Do you know what else? There are many, many mailing list owners who will not accept subscribers with Hotmail or Yahoo email addresses *because* of deliverability problems at these two services. Both services are very well known by savvy mailing list managers for delivery problems. An entire industry of deliverability consultants has emerged to deal with email delivery problems at Hotmail and Yahoo. No, resolving email delivery problems at Hotmail and Yahoo is not for the faint of heart. I also see that Mr. Haselton has not lost his fondness of conspiracy theories. I seriously doubt that Yahoo and Hotmail are sharing their blocklists. They are competitors, after all. And suppose they did share information about their blocklists. What of it? It is entirely within their prerogative to do so. I would even say it represents efficiency for the two organizations to share their blocklists. But I'm reasonably confident that they don't. I believe Mr. Haselton has a fundamental problem with blocklists, period. I believe he has taken his philosophy at Peacefire about blocking web sites and is naively attempting to apply it to email. I base this belief on email exchanges and conversations I have had with Mr. Haselton. He does good work with Peacefire. I think he should stick to doing what he knows best and stop his crusade against blocklists. Until spam is eliminated -- which will never happen -- they are here to stay. And that is a Good Thing because email would be unusable otherwise. Nick Nicholas

How do you say unsubscribe in Turkish?

[tepples]

We're talking about properly configured mail servers, no open relays, no backscatter, appropriate DNS, with opt-in recipients only and working, simple unsub options right in every email.

In how many languages are these unsubscribe options presented? I can't find the unsubscribe in a lot of Turkish mailing lists that I have ended up signed up to.

Spammers have access to bigger machines

[tepples]

Hashcash is interesting here, in that the CPU time is mostly spent by clients

So how do you allow legitimate mail sent from pocket-sized, battery-powered mobile devices without allowing mail from spammers who have access to bots running on compromised always-on PCs capable of running hashcash on beefy GPUs? I was under the impression that Bitcoin mining was a form of hashcash, and GPU-accelerated Bitcoin mining trojans were spotted over a year ago.

How do you say "remove me" in Turkish?

[tepples]

Seems pretty easy to me...

If you speak English. I have ended up subscribed to plenty of mailing lists in Turkish, and I can't read Turkish to find their unsubscribe processes. Besides, some spammers have long been known to see a "reply with the word 'unsubscribe' in the subject" as a request to sign up for all the spammer's other lists.

RSS with authentication

[tepples]

How many RSS readers support authenticated feeds? And how would a site requiring authentication let a web-based RSS reader log in to retrieve the feed without letting the RSS reader impersonate the user in other ways?

Address != person

[tepples]

as long as it ONLY GOES TO PEOPLE WHO VOLUNTARILY SUBSCRIBED

Addresses that voluntarily subscribe != people who voluntarily subscribe. Say someone cancels his Hotmail account, and later, someone else registers the same name. Who subscribed, and who continues to receive mail?

Hacky workaround

[halcyon1234]

1) Before sending out the mass email, send a test email to your own gmail, yahoo and hotmail account. See what bounces back.
2) Either drop that domain and register a new one OR
3) Base64 encode the URL in the email, with the provisio "Run this through base64decoder.com to get the address"

Second DKIM, SPF, RFC-compliant

[nullchar]

Using SPF is a bare-minimum for even a self-run SMTP server that only has a handful of users with no mailing lists. Anything larger needs DKIM.

One other thing Bennett can try is to have a pipeline of registered domains ready to be used. Each domain in each URL is scanned by spam filter, and young domains (via registration age in WHOIS or the daily registry-published zone-file which yahoo/hotmail/gmail all have) are more "spammy" than older domains, simply because spammers do the same thing here - register new domains and mass-email them out as URLs.

Advertisements

[NewYork]

They call it spam. I call it advertisements.

Re:Seriously...

[PortHaven]

Well, gee, than MAYBE the summary should post clearly what he is doing. Instead of saying

"Well, I am not doing anything illegal and really really it's not spam"

There is NOT a single thing in that WHOLE post that says he's doing this for freedom. Nearly every spammer claims their list is double opt-in. So how the !@#$% is anyone to know. There is one link to Circumventor List - sorry, that's blocked by work. So it gave no info.

Don't frickin flame me....flame the idiots who post a slashdot entry without giving an iota of background info.