Slashdot Mirror
To Beat Spam Filters, Look Like A Spammer?
Lest you think that spam filters only rarely make mistakes any more, recall the instance in which after I mailed out a group of 10 proxy websites to my own mailing list, the British "anti-spam" outfit Spamhaus blacklisted two of the domains, which caused the registrar (Afilias) to disable all 10 of the domains en masse, so that the sites simply disappeared from the Web. (This happened even though our mailing list is 100% closed-loop confirmed-opt-in; users have to reply to a confirmation message in order to join the list, so the actual emails were not "spam.") It took several days to find out what happened and restore the domains, during which Spamhaus and Afilias refused to answer any of my inquiries, and have to this day not reached out or explained what they're doing to avoid similar screw-ups in the future. And this was just the latest in a long line of headaches caused by spam filters including filters at Hotmail, AOL, Yahoo, and Gmail, which had regularly categorized our emails as "spam" and caused users to miss them.
So when the email deliverability company WhatCounts announced their October 16th webinar on how to avoid having your mails blocked as spam, I watched in real time with some interest. The webinar (which you can view here), was presented by Brad Gurley, the "Director of Deliverability" for WhatCounts, who has worked in the email "deliverability" industry for 10 years. While email deliverability services is one of the products that WhatCounts charges for, the presentation didn't contain any blatant plugs for their own services, so I'm taking the contents at face value. Even if any statements in the webinar happened to be incorrect, it's still safe to assume that the presentation represents mainstream thinking in the email deliverability industry, which will determine what recommendations are made to email senders.
I hasten to add that WhatCounts should not be blamed for any of the recommendations that they made that I'm counting as "eroding privacy"; their job was to answer the question, "What is the best way to make sure my emails don't get blocked as spam?", and they answered it. The fault, if any, should lie with the spam filters which encourage these practices. Furthermore, I'm only saying that the practices encouraged in the webinar are eroding user privacy, not violating it. (If you ask every new subscriber for their name and geographic location, I would call that an "erosion" of privacy if it normalizes the practice of collecting more user data than you need, but it's not a privacy violation as long as the user willingly gives it to you.)
The webinar begins with some recommendations that are actually good netiquette, such as cleaning subscriber lists regularly (removing bouncing addresses), and displaying a prominent "unsubscribe" link for users who want to leave. If you run a newsletter, and good netiquette isn't a compelling enough reason to put an "unsubscribe" link near the top, here is a direct quote from the webinar:
"The Unsubscribe link should be prominently placed within the message body. Unsubscribe links that are hidden or hard-to-find will generate spam complaints from unhappy users who want to unsubscribe. Placing the link in the preheader has been shown to reduce spam complaints in many cases."
That's one reason that every message that I send to my own newsletter, contains this text at the top:
[You are receiving this because you subscribed to the Circumventor distribution list. To unsubscribe from this list, click here: http://www.peacefire.org/circumventor/cv-unsub.html or reply with the word "unsubscribe" in the subject.]
(I give people the option of replying with the word "unsubscribe", even though that creates some hassle for me to process those requests manually, because many of our users are on censored networks and cannot access the unsubscribe link on the peacefire.org website.)
But, on to the less-stellar news: the presentation also says that the key to getting users to keep opening your emails -- and hence to signal to the email providers like Hotmail and Yahoo that your mails are not "spam" — is "engagement." Gurley suggests that senders "tailor mailings to segments of subscribers based on demographic data," including segmenting users based on city or zip code. Nothing sounds wrong with that, except that to "tailor" the mailings based on demographic data, you have to have that demographic data -- i.e. ask users for their age, sex, location, income bracket, or other information at the time that they join the list.
As I said, I don't consider this a violation of privacy if the user gives their information voluntarily, it's just an erosion of privacy, because it normalizes the process of asking users for extra data when there's no clear reason why it's necessary. In the late 1990s, you could join most companies' email lists without providing any more information than an email address; if you were asked for more information, it was for an obvious reason (such as filling out a profile on match.com, or ordering a product to be shipped). The less information about users was stored all in one place, the less opportunity there would be for the company to abuse it, or to be bought out by some other company that would abuse it, or for someone to hack into their servers and steal the information outright.
Our mailing list in particular serves a segment of the population who are particularly privacy-conscious -- they're using our proxy sites to circumvent Internet blocking software, so in almost all cases, just the simple act of being our mailing list could get them in some amount of trouble with somebody (although the severity would vary). So by design, we collect the minimum amount of information -- the email address -- necessary to send new proxy sites to the users. The more information that we asked for, the less likely the user might be to sign up in the first place.
Again, companies are within their right to ask for this information, but I don't think the rest of us newsletter publishers should be penalized for not asking for it.
The presentation goes on to say that email providers such as Hotmail and Yahoo judge whether an email is "spam" based on what proportion of the time users open an email from that sender. As Gurley says, "Give people a reason to open your email and keep opening it." The trouble is that this penalizes email notifications where you can fit all of the relevant content into the subject line -- many of my emails say something like "new Circumventor: badbadger.info", and for most users, that's all they need to see. Some subscribers have specifically said that they always want to see the new proxy site name in the subject line, because they're on a network where they are blocked from accessing their full email inbox, but they can use other webpages to see the subject lines of recently received emails. (For example, Yahoo Mail users might be on network where Yahoo Mail is blocked, but if you're signed in to yahoo.com you can see the subject lines of your last few emails on the www.yahoo.com front page.) If I'm being penalized by spam filters because user's don't open my emails, then obviously that's incentivizing me to do the users a disservice, by putting the proxy site name only in the message body.
(This might be an issue that is highly specific to my particular mailing list, because most people don't run email newsletters where they can fit all of the relevant content into the subject lines. However it's easy to think of other web applications that have a need for subject-only notifications -- Google Calendar sends me an email whenever one of my calendar events is coming up -- and those shouldn't be penalized just because the user never opens them.)
Finally, the presentation suggests that senders unsubscribe any user who hasn't opened the last 50 emails you sent them. This might set off mild alarm bells with tech-savvy readers, who know that the only way to tell if a reader has opened your message, is to embed images into the messages -- and if your newsletter content doesn't lend itself to images, you have to plant a surreptitious "web bug" image into the email, a tiny image that serves no purpose except that if you open the message and the image loads, it tells the sender that the message has been read. (For this reason, if you open an email message that does contain images, most email clients will not display them unless you click "Show images" or something similar -- because otherwise, if images always loaded automatically, spammers could use web bugs to tell who was opening their emails. So in fact, if a user opens your message and doesn't click "Show images", you generally can't tell that they opened your email.)
Again, I would consider web bugs to be an erosion of privacy more than a violation of it, on the order of asking for the user's zip code at the time they join their newsletter -- in both cases, the reason being that you are collecting more information than is strictly necessary for the operation of your mailing list. (In the case of web bugs, the "information" you're collecting is whether the user opened your message or not.)
Some people feel more strongly about it. A recent message posted on MIT's "liberationtech" mailing list had this to say about "web bugs", to a person who was asking about why his newsletter was being blocked:
You do not appear to use web bugs in your mailing list messages. A wise choice: web bugs are malware, they're invasive and abusive, and they actively degrade the security of recipients...which is a pretty crappy way to treat one's audience.
I think this is over the top -- all that a web bug does, is tell the sender whether you opened their message -- but, whether this opinion is valid or not, some people out there feel that way, and using web bugs in your email might piss them off.
Although before you cut loose the users who haven't opened your last 50 emails, Gurley's presentation also suggests trying to win them back with one last message with a "teaser" subject line like "We're saying goodbye...", or "Are we not going to talk to you any more?", or "Are we breaking up?". I hate subject lines like that, whether from spammers or from people I've signed up to get mail from. (Although now that I think about it, I doubt I'm really that mad about the 1 second of my time that they wasted; I think I just resent the fact that even just for that 1 second, they actually had me fooled, and I thought it really was a message from a friend.)
But again, we can't kill the messenger: Brad Gurley's job was to do a presentation on how to get your emails past the spam filters at the major email providers, and if using "come-on" subject lines works, because it gets more users to open your messages, then that's part of the answer. (Remember, this presentation was aimed at opt-in email senders, not spammers.)
So, I don't know that I can do anything differently with my list as a result of the presentation. I think it would be too off-putting to users to ask for their age and zip code, and in any case it wouldn't do any good for all the users who have already signed up. I probably couldn't use web bugs even if I wanted to, because the web bugs would have to load the image from a website, and if the user opened the email from a network where Web access was censored, the network's filter might block the website that the web bug loaded the image from. And for a list with many members who are still in high school, and whose parents might read their email over their shoulder, I don't feel like trying to get their attention by sending them an email with the subject "Are we breaking up?"
The more important takeaway here, though, is that there's no reason to expect the free market to deliver spam filters that are optimal from the user's point of view. In a world where users had perfect information, if Hotmail told their users, "We're going to start flagging the newsletters in your inbox as 'junk mail' unless the sender asks for your zip code when you sign up, and uses teasing subject lines to get you to open the message, and uses web bugs to verify whether you've opened it," their users would likely say, "Screw you, I'm going to Gmail!" (Which many of their users have apparently said anyway.) If this doesn't happen, it's because the vast majority of users don't have enough information for the market in spam filters to function effectively. And thus there's nothing to stop Hotmail and Yahoo from imposing arbitrary conditions on senders through their spam filters, which will lead to more legitimate senders resorting to "come-on" subject lines and web bugs -- ironically, looking more like the spammers they're trying to differentiate themselves from.
Get yer own blog, Bennie!
Spam filtering not a solution. E-mail has a monopoly on a lot of functions today. Getting accounts on most websites, getting receipts and confirmations from online purchases, recovering passwords, and countless other functions of the Internet. One thing they all have in common is that not only are they E-mail, but they are also unencrypted and can be spoofed with minimal effort.
A free market solution would be to offer more options. Automatic, universal encryption or digital signatures applied to everything genuine would be a legitimate solution to spam, and everything else gets dropped by your server. There are some minor obstacles, but if every mail server also serves the keys for the accounts it holds, it would be a simple matter to verify what current keys to accept at the recieving end.
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
Where'd that come from? Last I checked, "free-market forces" weren't capable of programming anything. Programmers do. Nothing's preventing anyone from making a better filter.
The "free-market forces" non-sequitur bespeaks an author with an ax to grind.
Article can be summed up as, "Sending mail people actually want is soooooo hard, I have to do all kinds of privacy-invasive things and that makes me a spammer!"
I've not seen such rambling nonsense for a long time. The guys domains appeared in spamhaus because - reality check - they are open proxies. Every single open URL redirector on the internet gets ruthlessly pillaged by spammers who are trying to avoid domain name blocks, so a URL like "http://my-proxy.com/render?url=http://buy-cheap-meds.info" inevitably lands my-proxy.com on spam-filter blacklists, because they learn that 99% of the time my-proxy.com appears in an email, that email is unwanted. URL shorteners are especially vulnerable to this.
As to the other ideas - hey, here's a great one. How about instead of using image bugs to try and figure out if your last 50 (!!) mails were ignored, why not ask users to re-opt in every so often if they want to continue receiving your mails? Was that really so hard? Keeping a good reputation with spam filters really isn't magic, so it blows me away that people host webinars on the topic - send mail people want. That's pretty much 95% of it. The other 5%? Avoid sharing resources that get abused by spammers - like URL shorteners.
I think Bennet may just have to give up on what he's trying to do here. If his proxies get abused by spammers to work around spam-filter URL domain reputation, then communicating lists of open proxies via email is inevitably going to break.