Slashdot Mirror
To Beat Spam Filters, Look Like A Spammer?
Lest you think that spam filters only rarely make mistakes any more, recall the instance in which after I mailed out a group of 10 proxy websites to my own mailing list, the British "anti-spam" outfit Spamhaus blacklisted two of the domains, which caused the registrar (Afilias) to disable all 10 of the domains en masse, so that the sites simply disappeared from the Web. (This happened even though our mailing list is 100% closed-loop confirmed-opt-in; users have to reply to a confirmation message in order to join the list, so the actual emails were not "spam.") It took several days to find out what happened and restore the domains, during which Spamhaus and Afilias refused to answer any of my inquiries, and have to this day not reached out or explained what they're doing to avoid similar screw-ups in the future. And this was just the latest in a long line of headaches caused by spam filters including filters at Hotmail, AOL, Yahoo, and Gmail, which had regularly categorized our emails as "spam" and caused users to miss them.
So when the email deliverability company WhatCounts announced their October 16th webinar on how to avoid having your mails blocked as spam, I watched in real time with some interest. The webinar (which you can view here), was presented by Brad Gurley, the "Director of Deliverability" for WhatCounts, who has worked in the email "deliverability" industry for 10 years. While email deliverability services is one of the products that WhatCounts charges for, the presentation didn't contain any blatant plugs for their own services, so I'm taking the contents at face value. Even if any statements in the webinar happened to be incorrect, it's still safe to assume that the presentation represents mainstream thinking in the email deliverability industry, which will determine what recommendations are made to email senders.
I hasten to add that WhatCounts should not be blamed for any of the recommendations that they made that I'm counting as "eroding privacy"; their job was to answer the question, "What is the best way to make sure my emails don't get blocked as spam?", and they answered it. The fault, if any, should lie with the spam filters which encourage these practices. Furthermore, I'm only saying that the practices encouraged in the webinar are eroding user privacy, not violating it. (If you ask every new subscriber for their name and geographic location, I would call that an "erosion" of privacy if it normalizes the practice of collecting more user data than you need, but it's not a privacy violation as long as the user willingly gives it to you.)
The webinar begins with some recommendations that are actually good netiquette, such as cleaning subscriber lists regularly (removing bouncing addresses), and displaying a prominent "unsubscribe" link for users who want to leave. If you run a newsletter, and good netiquette isn't a compelling enough reason to put an "unsubscribe" link near the top, here is a direct quote from the webinar:
"The Unsubscribe link should be prominently placed within the message body. Unsubscribe links that are hidden or hard-to-find will generate spam complaints from unhappy users who want to unsubscribe. Placing the link in the preheader has been shown to reduce spam complaints in many cases."
That's one reason that every message that I send to my own newsletter, contains this text at the top:
[You are receiving this because you subscribed to the Circumventor distribution list. To unsubscribe from this list, click here: http://www.peacefire.org/circumventor/cv-unsub.html or reply with the word "unsubscribe" in the subject.]
(I give people the option of replying with the word "unsubscribe", even though that creates some hassle for me to process those requests manually, because many of our users are on censored networks and cannot access the unsubscribe link on the peacefire.org website.)
But, on to the less-stellar news: the presentation also says that the key to getting users to keep opening your emails -- and hence to signal to the email providers like Hotmail and Yahoo that your mails are not "spam" — is "engagement." Gurley suggests that senders "tailor mailings to segments of subscribers based on demographic data," including segmenting users based on city or zip code. Nothing sounds wrong with that, except that to "tailor" the mailings based on demographic data, you have to have that demographic data -- i.e. ask users for their age, sex, location, income bracket, or other information at the time that they join the list.
As I said, I don't consider this a violation of privacy if the user gives their information voluntarily, it's just an erosion of privacy, because it normalizes the process of asking users for extra data when there's no clear reason why it's necessary. In the late 1990s, you could join most companies' email lists without providing any more information than an email address; if you were asked for more information, it was for an obvious reason (such as filling out a profile on match.com, or ordering a product to be shipped). The less information about users was stored all in one place, the less opportunity there would be for the company to abuse it, or to be bought out by some other company that would abuse it, or for someone to hack into their servers and steal the information outright.
Our mailing list in particular serves a segment of the population who are particularly privacy-conscious -- they're using our proxy sites to circumvent Internet blocking software, so in almost all cases, just the simple act of being our mailing list could get them in some amount of trouble with somebody (although the severity would vary). So by design, we collect the minimum amount of information -- the email address -- necessary to send new proxy sites to the users. The more information that we asked for, the less likely the user might be to sign up in the first place.
Again, companies are within their right to ask for this information, but I don't think the rest of us newsletter publishers should be penalized for not asking for it.
The presentation goes on to say that email providers such as Hotmail and Yahoo judge whether an email is "spam" based on what proportion of the time users open an email from that sender. As Gurley says, "Give people a reason to open your email and keep opening it." The trouble is that this penalizes email notifications where you can fit all of the relevant content into the subject line -- many of my emails say something like "new Circumventor: badbadger.info", and for most users, that's all they need to see. Some subscribers have specifically said that they always want to see the new proxy site name in the subject line, because they're on a network where they are blocked from accessing their full email inbox, but they can use other webpages to see the subject lines of recently received emails. (For example, Yahoo Mail users might be on network where Yahoo Mail is blocked, but if you're signed in to yahoo.com you can see the subject lines of your last few emails on the www.yahoo.com front page.) If I'm being penalized by spam filters because user's don't open my emails, then obviously that's incentivizing me to do the users a disservice, by putting the proxy site name only in the message body.
(This might be an issue that is highly specific to my particular mailing list, because most people don't run email newsletters where they can fit all of the relevant content into the subject lines. However it's easy to think of other web applications that have a need for subject-only notifications -- Google Calendar sends me an email whenever one of my calendar events is coming up -- and those shouldn't be penalized just because the user never opens them.)
Finally, the presentation suggests that senders unsubscribe any user who hasn't opened the last 50 emails you sent them. This might set off mild alarm bells with tech-savvy readers, who know that the only way to tell if a reader has opened your message, is to embed images into the messages -- and if your newsletter content doesn't lend itself to images, you have to plant a surreptitious "web bug" image into the email, a tiny image that serves no purpose except that if you open the message and the image loads, it tells the sender that the message has been read. (For this reason, if you open an email message that does contain images, most email clients will not display them unless you click "Show images" or something similar -- because otherwise, if images always loaded automatically, spammers could use web bugs to tell who was opening their emails. So in fact, if a user opens your message and doesn't click "Show images", you generally can't tell that they opened your email.)
Again, I would consider web bugs to be an erosion of privacy more than a violation of it, on the order of asking for the user's zip code at the time they join their newsletter -- in both cases, the reason being that you are collecting more information than is strictly necessary for the operation of your mailing list. (In the case of web bugs, the "information" you're collecting is whether the user opened your message or not.)
Some people feel more strongly about it. A recent message posted on MIT's "liberationtech" mailing list had this to say about "web bugs", to a person who was asking about why his newsletter was being blocked:
You do not appear to use web bugs in your mailing list messages. A wise choice: web bugs are malware, they're invasive and abusive, and they actively degrade the security of recipients...which is a pretty crappy way to treat one's audience.
I think this is over the top -- all that a web bug does, is tell the sender whether you opened their message -- but, whether this opinion is valid or not, some people out there feel that way, and using web bugs in your email might piss them off.
Although before you cut loose the users who haven't opened your last 50 emails, Gurley's presentation also suggests trying to win them back with one last message with a "teaser" subject line like "We're saying goodbye...", or "Are we not going to talk to you any more?", or "Are we breaking up?". I hate subject lines like that, whether from spammers or from people I've signed up to get mail from. (Although now that I think about it, I doubt I'm really that mad about the 1 second of my time that they wasted; I think I just resent the fact that even just for that 1 second, they actually had me fooled, and I thought it really was a message from a friend.)
But again, we can't kill the messenger: Brad Gurley's job was to do a presentation on how to get your emails past the spam filters at the major email providers, and if using "come-on" subject lines works, because it gets more users to open your messages, then that's part of the answer. (Remember, this presentation was aimed at opt-in email senders, not spammers.)
So, I don't know that I can do anything differently with my list as a result of the presentation. I think it would be too off-putting to users to ask for their age and zip code, and in any case it wouldn't do any good for all the users who have already signed up. I probably couldn't use web bugs even if I wanted to, because the web bugs would have to load the image from a website, and if the user opened the email from a network where Web access was censored, the network's filter might block the website that the web bug loaded the image from. And for a list with many members who are still in high school, and whose parents might read their email over their shoulder, I don't feel like trying to get their attention by sending them an email with the subject "Are we breaking up?"
The more important takeaway here, though, is that there's no reason to expect the free market to deliver spam filters that are optimal from the user's point of view. In a world where users had perfect information, if Hotmail told their users, "We're going to start flagging the newsletters in your inbox as 'junk mail' unless the sender asks for your zip code when you sign up, and uses teasing subject lines to get you to open the message, and uses web bugs to verify whether you've opened it," their users would likely say, "Screw you, I'm going to Gmail!" (Which many of their users have apparently said anyway.) If this doesn't happen, it's because the vast majority of users don't have enough information for the market in spam filters to function effectively. And thus there's nothing to stop Hotmail and Yahoo from imposing arbitrary conditions on senders through their spam filters, which will lead to more legitimate senders resorting to "come-on" subject lines and web bugs -- ironically, looking more like the spammers they're trying to differentiate themselves from.
Get yer own blog, Bennie!
Spam filtering not a solution. E-mail has a monopoly on a lot of functions today. Getting accounts on most websites, getting receipts and confirmations from online purchases, recovering passwords, and countless other functions of the Internet. One thing they all have in common is that not only are they E-mail, but they are also unencrypted and can be spoofed with minimal effort.
A free market solution would be to offer more options. Automatic, universal encryption or digital signatures applied to everything genuine would be a legitimate solution to spam, and everything else gets dropped by your server. There are some minor obstacles, but if every mail server also serves the keys for the accounts it holds, it would be a simple matter to verify what current keys to accept at the recieving end.
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
Who's dick is Bennett Haselton sucking to get so many of his rants posted here?
I know I'd unsubscribe from Bennet if I could.
I was wondering what to dress up as, for Halloween!
I'll be the low sodium one!
If your clients really want to get your spam, simply instruct them to whitelist you during the registration process.
Having said that, I don't really have much sympathy for someone who's trying to help students and employees circumvent network policy. They can watch their porn or check facebook on their own time.
Where'd that come from? Last I checked, "free-market forces" weren't capable of programming anything. Programmers do. Nothing's preventing anyone from making a better filter.
The "free-market forces" non-sequitur bespeaks an author with an ax to grind.
The webinar begins with some recommendations that are actually good netiquette.
Is this webinar on the Information Super Highway?
Article can be summed up as, "Sending mail people actually want is soooooo hard, I have to do all kinds of privacy-invasive things and that makes me a spammer!"
I've not seen such rambling nonsense for a long time. The guys domains appeared in spamhaus because - reality check - they are open proxies. Every single open URL redirector on the internet gets ruthlessly pillaged by spammers who are trying to avoid domain name blocks, so a URL like "http://my-proxy.com/render?url=http://buy-cheap-meds.info" inevitably lands my-proxy.com on spam-filter blacklists, because they learn that 99% of the time my-proxy.com appears in an email, that email is unwanted. URL shorteners are especially vulnerable to this.
As to the other ideas - hey, here's a great one. How about instead of using image bugs to try and figure out if your last 50 (!!) mails were ignored, why not ask users to re-opt in every so often if they want to continue receiving your mails? Was that really so hard? Keeping a good reputation with spam filters really isn't magic, so it blows me away that people host webinars on the topic - send mail people want. That's pretty much 95% of it. The other 5%? Avoid sharing resources that get abused by spammers - like URL shorteners.
I think Bennet may just have to give up on what he's trying to do here. If his proxies get abused by spammers to work around spam-filter URL domain reputation, then communicating lists of open proxies via email is inevitably going to break.
is this a non-problem?
--fatboy
This is just the latest in a series of Slashdot posts in which he explains why spam is that which he does not do.
He's a spammer. Hence he's recommending that spammers do the kinds of things spammers do.
Is that why all of his posts make it through the firehose?
The free market doesn't really apply when there is near-zero cost to sending an email other than actually typing the message.
I got greatly annoyed by a colleague who attended a seminar from a training company that had been spamming our company. Buying anything from a spam message promotes spamming, but it's clearly effective for spammers.
Woah there! Don't start a flame war, buddy.
Given past experiences with slashdot front page posts consisting of a wall of text, I'd have to assume that this is a nobody spouting insightless drivel or ranting against a cautionary principle he clearly doesn't understand.
That said, beating spam filters is easy. Ordinary non-spammy emails get through fine. It's only when you doing something borderline spammy that the spam filter catches you.
In this case, the asshole was running a mailing list.
Then when he asked why... no answer. I originally checked this post out thinking there might be value, and I was wrong.
Whenever I send or receive a URL in the first email exchanged, I wind up checking the spam folder in webmail (Yahoo, Gmail) because that's where it winds up half the time. After having it transferred to the Inbox, there's rarely another issue of getting any mail from them. Meantime, we've all had outright spams get through the filters, server-side or client-side, because the author tried hard to make it seem more like a human sent something you wanted to see. But I do wonder how a spam reply from Craigslist can wind up in the spam folder while a legitimate reply can make it to me, seeing they both have the same subject line, a legit-looking email address (some of the time), and part of the body content.
Opt-in direct mailing shouldn't be affected by spam filters because despite being sent in bulk no one receiving it is complaining, and you'd think cloying titles like "Are we breaking up?" would trip filter triggers (or at least human brain triggers) quicker than "Weekly Report for 10/21/13".
Laughter is the Spackle of the Soul.
I don't really have much sympathy for someone who's trying to help students and employees circumvent network policy. They can watch their porn or check facebook on their own time.
People who live at school are often subject to filtering even in the dorms. So what is "their own time" to you?
You have a newsletter and problem being misfiled as spam? Put each new issue online (you probably do already) and offer an RSS feed with it. Some people greatly prefer RSS to a periodic email, and you can point people to it if they tell you the emails are getting blocked.
Trust the Computer. The Computer is your friend.
Then I saw that it was from our favorite "never has a real clue" "contributor" Bennett Hasselton. I'd rather go and read all of "War and Peace"...
to get past spam filters to allow kids to look at porn at school. Brilliant
I don't think Bennie's quite ready to be trusted with an ax of his own.
I'm not even sure he's allowed metal spoons, since The Unfortunate Incident At Dinner.
They expose the location and user agent of the readers location to the sender.
The are also vulnerable to surveillance by anyone between the reader and the sender.
See story number 3: http://www.infoworld.com/print/222831
You know, if you get frequent run-ins with anti-spam tools, then maybe they are all stupid and broken and need to be re-examined - or, maybe, you need to re-examine the way you work, including the tough question of maybe you ARE a spammer?
The #1 red flag for any conspiracy theory, crackpot or pseudo-science is always the attribution of blame exclusively to outside forces. If nobody listens to you, it must be because of a conspiracy to cover things up, or the establishment trying to put you down, or whatever.
As other posters have outlined: You had open proxies, thus you rightfully belonged on the blocklists. If you re-examine your other problems, you might also find that everything works as it should in the anti-spam world, except for the spammers.
Assorted stuff I do sometimes: Lemuria.org
stop using email for mailing list subscriptions entirely, this would be more appropriately handled through RSS. however that would require actual opt-in instead of "we got you to click on yes so you are opted in"
Snowden and Manning are heroes.
"all that a web bug does, is tell the sender whether you opened their message"
Actually, it tells much, much more: the IP address, approximate geographic location of the receiver and precise times when the email was opened; his operating system, browser and other technical data that can be used to infer demographics and even mount a cyberattack against him, or further refine a social engineering attack. Web-bugs will also link two otherwise disparate email aliases, say petraeus.d@army.mil and loverboy69@aol.com, thereby compromising privacy.
Web-bugs are a form of malware in that they exploit a vulnerability in the recipient's user agent software in order to subvert control of his computer, make it submit personal data the recipient might not agree submitting, while hiding this fact.
Can you please quit posting this morons thoughts like he is someone who matters? Is he one of the DICE flunkies or something? No one gives a shit about his ignorance. Just because he created a couple websites doesn't mean he has a clue or is authoritative on any subject, including the ones he's created the websites for.
Yes, I know who he is.
Yes, he's a fucking idiot. Stop posting his ridiculous diatribes.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
No one here gives a shit. My advice is go talk to people who have the most to gain from allowing opt-in content. Namely, the major mail providers.
Bennet went to some marketing demo, got his panties in a bunch, and then as usual complains to Slashtards. We can't help him.
So yeah, non-problem.
I tried not to reply, but asshattery is hard to not reply to.
Tom,
Hey, it's been a while. Remember me? We were friends on MySpace a few years back. I've moved on to a new social service. Do you want to join me on Friendster?
Take care,
Seth
$5 / month hosted VPS on linux = awesome!
That will go a long way to stopping spam on /.
And my spam filters aren't filtering those.
The webinar begins with some recommendations that are actually good netiquette.
Is this webinar on the Information Super Highway?
Can you work the 'cyber' prefix in there somehow?
Jesus was all right but his disciples were thick and ordinary. -John Lennon
Bennet Haselton isn't Jon Katz 2.0.
Katz's mindless ramblings were at least occasionally interesting.
The editors had the good sense to list Katz as an editor himself so that he could be filtered away.
Curse myself for not noticing the submitter before clicking the link. Curse /. and especially soul kill for making it necessary for me to read who the submitter is.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
jeeze
Spam is an economic problem. People will respond to this by praising their favorite spam filters, and ignoring the obvious fact that the filters don't solve the problem, and never will solve the problem. Spam is present not to piss you off but because spammers make money by sending it out. If you truly want to stop spam, no number or combination of technical fixes, legislative proposals, public executions, user education, or forum posts will do. The one and only way to stop spam is to prevent the spammers from getting paid. We have ways to do this, that have been demonstrated. We just need to actually follow through with it. If spammers don't get paid, they will get out of the business.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
I give people the option of replying with the word "unsubscribe", even though that creates some hassle for me to process those requests manually, because many of our users are on censored networks and cannot access the unsubscribe link on the peacefire.org website
Oh, if all mailing lists were so insightful. Besides, not all of your users are reading your mail "in the browser".
Following up on myself:
You do not appear to use web bugs in your mailing list messages. A wise choice: web bugs are malware [...]
I think this is over the top -- all that a web bug does, is tell the sender whether you opened their message -- but, whether this opinion is valid or not, some people out there feel that way, and using web bugs in your email might piss them off.
Well, I think it's not over the top, but as far as I'm concerned, I never "open" any mail, since my MUA can't load images or any other links and can't do active content. Heck, my browser's javascript is disabled by default most of the time.
"Are we breaking up?"
Yes, it seems we have already.
That litany just got flagged by my internal filter... [$MaxLength >> x]
Spam is in the eye of the beholder, and that's not you.
So chill out, accept that your newsletter isn't the best thing since sliced bread, and that the fact you're sending it to someone who was probably tricked into subscribing, but changed their mind once they read the first paragraph, doesn't make it legitimate for all time or any time at all.
The Internet doesn't owe you a living. Don't send out your messages, make a website and leave them there. If people want to read them, they'll come. Peace.
In my humble opinion BH does some truly admirable work documenting abusing blocking and documenting/creating ways around blocking.
I'm a strong supporter of the old cyberpunk credo: "Information want to be free". I'm opposed to all filtering and blocking, no matter if it's stupid parents that think that their child benefits from living in a rose-colored bubble completely unaware of the real world and possibly unable to find support for whatever 'deviant' thoughts he or she might have, or employers that think that their minions gets more productive if they are forced to check their Facebook accounts from their phone in the bathroom instead of their work computer... Creating work-arounds to combat blocking is Admirable Work in my world, and BH has been doing that for a long time. Sure, he might be annoying at times, but I can live with that.
"For every complex problem, there is a solution that is simple, neat, and wrong." -- H.L. Mencken (1880-1956) --
I block Bennett's stories from appearing.
Bennett gets someone else to post his drivel.
Bennett is getting around a filter that was put in place, via Slashdot's own system....and is therefore evading. To talk about how to get around filters.
Anyone else see the irony here?
Supernaut
How do I block it?
If people are not clicking on and reading your 'newsletter', then it is spam. It probably means they never really wanted it in the first place. If unsubscribe isn't obvious - and ocrable (so smart people can put a filter to send unsubscribe stuff to a junk folder), then you are spam. As for web-bugs, they are a huge invasion of privacy, not an 'erosion'. Smart filters reject all web bugs. If you want to know if they opened your email you put in a big, obvious image. The only reason to use a 'web bug' is because you don't wan them to KNOW you are tracking them. In addition I bet
1) he use a default opt-in is a real opt-in (it isn't - it's an attempt to trick the unwary from mistakenly opting in)
2) he includes an opt in as part of a registration process (even if it is is 'optional')
If the sender uses an image in the email to bug the recipient, then it's a web bug. If the sender doesn't keep track of who opens the image, then it's not a web bug.
If the image is 1x1 in the same color as the background, it's pretty much guaranteed that the sender is using it as a web bug, because about the only other thing you can do with images like that is try to tweak kerning or fill in a table entry that gets misaligned if you don't, or something like that.
Twitter's web page constantly tells me it thinks I'm not receiving its emails correctly, and offers to send me more test emails, because they're using web bugs, and I use an email client that shows me email as text, not as HTML, and they so thoroughly assume that everybody uses buggable web mail that they don't even include a URL link saying "Please click this in a browser to confirm we got your email correct." And the banner on their web page that offers to send me a test email doesn't have a choice for "Yes, I'm receiving it just fine, stop whining." Idiots.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I maintain a small announcement list for about 200 mostly highly tech-savvy people. We've been around for 25+ years on a range of different platforms, and are currently using a hosting platform with Linux and mailman (as opposed to the previous home Linux box and majordomo), but we still occasionally get spamblocked. It's text-format mail, no automated verification, and it's possible that some mailbox services are blocking us silently instead of bouncing, but most of the bouncegrams I get these days claim that the recipient's mailbox is full (maybe true, sometimes not), or the usual things you get when somebody moves and their forwarding breaks. Occasionally I get a burst of greymail-grams. The site that seems to do the most silent drops is pobox.com, which is annoying because it's where I do my own mail, so I have to have a couple of duplicate subscriptions of my own just in case it's cranky again.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Yes, it's nice to be able to receive images from people who are actually your friends, not spammers, and who don't overdo sending annoyingly cutesy images (e.g. that cousin who forwards stuff to everybody.)
But being an old guy doesn't just mean that I want you to send text email and stay off my lawn, it also means I want to set the font I use to read email with so it's easy for me to read, instead of having you pick a font that you think looks great to you on your screen, because I need a font that's big enough and dark enough to read easily, and if I'm reading mail on a phone instead of a full-sized screen, I *really* want to have my choice of font size, not yours, and while maybe you think Comic Sans is cute or <BLINK>want to send your Halloween party announcement in a blood-red font that's bleeding down the page</BLINK>, I'd much rather be able to read what you wrote.
And because I'm an old cranky security guy, I really really don't want my email client trying to run your Javascript on my machine, thank you very much, even if all you think you're doing is trying to center the text neatly in ways that might look good on a 24" monitor but utterly fail when I'm reading in an SSH session or on my phone.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Dude, it is a free market, for most people in the world; if you're a draftee into some army that only uses X.400 email, or your country only allows unencrypted SMTP to pass through their Great Firewall, then I'm sorry, and I can recommend some good anti-censorship tools for you, which you can get from a guy named Bennett Hasleton.
But otherwise, you're free to use tools other than SMTP/POP/IMAP/Webmail, and we'll be happy to see your running code and give you opinions about whether you'll get rough consensus from anybody else about using it.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
You're confusing the difference between my suggestions and my level of care. I'm perfectly happy having abandoned email for any meaningful or important communication.
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!