I would have thought that spammers wouldn't expend much effort at trying to get around sophisticated anti-spam techniques. After all, if you go to the trouble to block spam, you're probably not going to respond.
But of course some of the spammers get paid based on how many 'eyes' (or HTTP requests) are generated, so if they can just get through to an Outlook Express preview pane, it's worthwhile....until 'marketers' wise up.
By virtue of having my own domain name, outside of the United States, I now receive 1200+ spams a day (and noticeably increasing). People who advocate 'just hitting the delete key' make me fume. That's a lot of delete key. And a lot of time. I've now reached the point where false positives on spam detection by automated software are less likely than me hitting delete one too many times. Thanks to DNSBL I can reduce spam from 1200+ a day to 10 a day, and Paul Graham's Bayesian filtering reduces that down to 2 or 3 a week.
I'd like to share some recent observations I've made - I haven't seen this referenced elsewhere but maybe I don't know where to look (so feel free to point me where this is mentioned elsewhere).
First a minor observation that spam increases markedly on the weekends - because peop,e aren't around to close down open relays or spamming accounts?
Secondly, spammers have started adding non-spammy words (eg capacitor) and constrcuted nonsense words (capacitorsggg) inside their messages. I can only see this as a direct response to Paul Graham's approach. I don't see it as working - the rest of the message is just TOO spammy - but it sugegst to me that spammers see such an apprroach as a threat. I've seen these words sprinkled at the start of plain text emssages and after the/body>/html> of HTML messages.
Thirdly, what I've recently noticed is that a spammer will connect to my mail server, say HELO, do a MAIL FROM: and then QUIT. Then they connect to my system again and use a HELO command that is my OWN IP address. They also include a fake Received header that makes it look as though the message originated from my own machine. Nice try you scummy spammers. SpamCop is smart enough to see through that ploy. I wonder how other system's will respond.
Fourthly, I've noticed that often when I complain to SpamCop I become the victim of a JoeJob. Currently I'm getting all the delivery failures coming back to random alphanumeric usernames at my domain. Sigh. Time to strip off my domain when I lodge SpamCop submissions eh?
To ensure the fairness and accuracy of disclosure of business terms and conditions to the user public and community in general;
To improve the standard of conduct within the industry;
To provide public access to complaint handling and cost-effective redress mechanisms;
To impose and regulate industry standards;
To improve customer relations;
To protect rights of access and free speech;
To ensure that information and procedures are in place for the protection of minors from accessing objectionable material over the Internet;
To ensure that the information and procedures are in place so Internet users know how to limit access to protect a user from accessing inappropriate or objectionable material
Note the positive phrasing - to protect the rights of access and free speech. I like that. In fact I like all of it. It seems very customer friendly. Which is why I expect Telecom's Xtra doesn't like it - it talks about the right of services being forwarded if you change providers - Xtra wouldn't like that.:-)
The onus is on the users, not the ISP to know how to protect themselves from objectionable content.
It suggests ratings systems like PICS. I have to say - it all looks good to me.
Agreed. Pages 55 onwards are specific to the Internet and the summary of recommendation are on page 68.
My summary on the key points made in the Internet section of the report are:
Child pornography and other obscene material is more readily available because of the Internet (and is already illegal material).
Peer to peer systems makes it harder to control the distribution of such material - suggests law changes to make clear offering files on a P2P system is "supply"
Recommending that filter software be made available (ideally free of charge)
Education of users should be encouraged
Live shows are not covered by censorship/classification rules - "let's change that"
Ability to execute a search warrant on grounds of possession, rather than trading
Have ISPs adopt a code of practice - if they don't (and NZ Telecom's Xtra is noted as dragging it's heels), then force one on them.
Nothing in there that alarms me too much.
Child pornography is the bogey man, and the vehicle on which everything else rides, if anything extreme is going to be introduced.
What makes you think they'd stop here? This Earth? Just because we share the same name (and some mixed up mythological names)...this is not the planet of advanced technology lost colony that will save you from the Cylons...move along.
The reality is that BG would go right past Sol III and continue searching for the real lost colony of similar (or more advanced) technology and write us off (like the other humanoid planets they passed) as being a side-show.
Russian wives. I was surprised directmailorderbrides wasn't picked up, but as it turned out, that's the first time that word (token) has appeared in any of my email.
Note that while the Paul Graham rating of 0.999999999999999 is high, in practice I use Gary Robinson's calculations (more refined and use even infrequently occurring tokens - I get better, less extreme results). Gary Robinson's spam rating on this is: 0.61705129961986
That may seem relatively low, but is on a different scale and is firmly indicative of spam.
Unlike Paul Graham, I don't parse out (and ignore) HTML comments. I find all information is useful, and I find it just as effective (and simple) to treat the text as a straight byte stream.
I got the same spam. No problem with my naive Bayesian filter:
nrc 0.9322408
200 0.8996861
-0500 0.9530694
5329 0.9897271
u 0.9130855
s 7.409731E-02
communigate 0.9296147
contrast 4.253653E-02
codewks 0.9789982
f 0.9006658
nacjack 0.9789982
wlink 0.98057
cithara 0.9689967
feb 0.9688099
2003 0.9078071
Spam ratings:0.999999999999999
The headers are still comprpmising, and with no real English words, there's nothing to weight this message against being spam.
Interestingly very few people sending me real email use the letters "f" and "u" on their own.:-)
My point remains valid. Because there is a direct cost to the spammer to adapt.
If they bulk up their spam that's going to slow them down, increase their costs (even if bandwidth costs aren't going to be passed back to them now, the more they use, the more visible they become). They become more visible.
Or they continue on their way. The reality is that they concentrate on the easy targets - you and I will never purchase their services so people taking this approach aren't really in their target audience anyway. I know this is (surprisingly) less true than one might think. Spammers do work to overcome basic obstacles, but that adds more costs and time - they don't work hard to avoid tar pits, because there are so few of them.
So I still see it as a win...large emails are very unlikely to be spam. If that changes, well so be it, but that will hurt the spammers. In the meantime I reap the benefit of fewer false positives and faster spam filtering.
Final comment - over the last six months I've seen spam get slightly larger (from about 32k peak size to about 45k peak size). But I haven't been analysing for any trends - just the outliers.
Spam tends to be short. The shorter the spam, the more messages they can put through. So spammers would be loathe to add 21 pages of text to their spam.
I have
Const maxspamsize = 42695
in my spam filter - I've only receive one piece of spam larger than than in the last 12 months (a giant promotion for a Korean trade show). It speeds up my spam filter processing and lets large newsletters (with false triggers like this) through without a problem.
Anything that constrains or reduces the field of play of spammers aids in the detection of spam. Even if the compliance is voluntary.
I can filter for "residents of CA" and "105th U.S. Congress" to eliminate large amounts of spam. Thanks to spammers everywhere using
"This message is sent in compliance with the new email bill section 301. Under Bill S.1618 TITLE III passed by the 105th U.S. Congress this message cannot be considered Spam as long as we include the way to be removed, Paragraph (a)(c) of S.1618. Further transmissions to you by the sender of this email may be stopped at no cost to you by sending a request to be removed to __"
Now that I'm using that Bayesian-style analysis of spam this gets even better...
Token, Spam Hits, Real Email 105th, 462, 0
Congress, 636, 8
transmissions, 632, 2
residents, 342, 11
CA, 10240, 606
So forcing/encouraging spammers to add "some crap" to their email (even if it is trying to avoid the effects of a law) just makes it easier to pick them up.
Somewhat related is this approach I've been trialing quite successfully for the last month. I haven't been able to find any reference to anyone else doing this, and would welcome any comments.
If it's a 'new site' (not dealt with regualrly and not seen recently) and it shows up clean on the variosu DNSBL's I use, then I send a temporary error code back. If they retur (after a suitable time delay - I use 15 minutes) and still come up clean, then I let it through.
Advantages:
* many spammers don't retry - ever (perhaps they get shut down, or someone closes their open relay, or they concentrate on more receptive targets)
* those that do retry (often many hours later - average is 7.6 hours for spammers) are usually listed on the DNSBL's by then
* I get to collect the list of mail addresses they are trying to send, and if they hit one of my spam traps (and there are many obvious dictionary attacks) then they immediately get marked bad even if they are not DNSBL'd
* Doesn't waste bandwidth (or the hijacked resources of a open relay 'victim') which continually using a tar pit does
Disadvantage
* Genuine email from a new/infrequent source gets delayed 15 + (until their servers retry) minutes. Most geuine ISPs try at reasonable intervals - though some wait an hour.
I'm willing to wait an hour for mail from someone new, who's not on my whitelist, given the amount of spam this simple technique filters.
Obviously if everyone adopts this approach then spammers would deliberately work around it - but it would complicate matters for them - the time delay and reptetive nature of their attempts would make them even more obvious as spammers, and more easy to shut down. And they can't avoid the spam traps.
Forgive me if this is obvious and well known - I'd appreciate any pointers to where this has been applied and any comments.
Since background radtion levels are around 0.1 mSv/hr (eg in Hong Kong)
then all of us should be near enough radiation to "test if the watch even works".
Slashdot Mirror
551 User not local; please try
So far no takers. And after 2 months no spam at all. A 14 character alphanumeric address
We just want to borrow your brain.
We have to get it out first.
It has to be prepared
...treated...
.............diced!
It can be replaced if you think it's important. A shell script would suffice. A simple one would do.
But of course some of the spammers get paid based on how many 'eyes' (or HTTP requests) are generated, so if they can just get through to an Outlook Express preview pane, it's worthwhile....until 'marketers' wise up.
By virtue of having my own domain name, outside of the United States, I now receive 1200+ spams a day (and noticeably increasing). People who advocate 'just hitting the delete key' make me fume. That's a lot of delete key. And a lot of time. I've now reached the point where false positives on spam detection by automated software are less likely than me hitting delete one too many times. Thanks to DNSBL I can reduce spam from 1200+ a day to 10 a day, and Paul Graham's Bayesian filtering reduces that down to 2 or 3 a week.
I'd like to share some recent observations I've made - I haven't seen this referenced elsewhere but maybe I don't know where to look (so feel free to point me where this is mentioned elsewhere).
First a minor observation that spam increases markedly on the weekends - because peop,e aren't around to close down open relays or spamming accounts?
Secondly, spammers have started adding non-spammy words (eg capacitor) and constrcuted nonsense words (capacitorsggg) inside their messages. I can only see this as a direct response to Paul Graham's approach. I don't see it as working - the rest of the message is just TOO spammy - but it sugegst to me that spammers see such an apprroach as a threat. I've seen these words sprinkled at the start of plain text emssages and after the /body> /html> of HTML messages.
Thirdly, what I've recently noticed is that a spammer will connect to my mail server, say HELO, do a MAIL FROM: and then QUIT. Then they connect to my system again and use a HELO command that is my OWN IP address. They also include a fake Received header that makes it look as though the message originated from my own machine. Nice try you scummy spammers. SpamCop is smart enough to see through that ploy. I wonder how other system's will respond.
Fourthly, I've noticed that often when I complain to SpamCop I become the victim of a JoeJob. Currently I'm getting all the delivery failures coming back to random alphanumeric usernames at my domain. Sigh. Time to strip off my domain when I lodge SpamCop submissions eh?
That's OK. I expect to have mine returned marked "Insufficient Postage".
And so do two of it's moons!
The listed aims are:
- To ensure the fairness and accuracy of disclosure of business terms and conditions to the user public and community in general;
- To improve the standard of conduct within the industry;
- To provide public access to complaint handling and cost-effective redress mechanisms;
- To impose and regulate industry standards;
- To improve customer relations;
- To protect rights of access and free speech;
- To ensure that information and procedures are in place for the protection of minors from accessing objectionable material over the Internet;
- To ensure that the information and procedures are in place so Internet users know how to limit access to protect a user from accessing inappropriate or objectionable material
Note the positive phrasing - to protect the rights of access and free speech. I like that. In fact I like all of it. It seems very customer friendly. Which is why I expect Telecom's Xtra doesn't like it - it talks about the right of services being forwarded if you change providers - Xtra wouldn't like that.The onus is on the users, not the ISP to know how to protect themselves from objectionable content.
It suggests ratings systems like PICS. I have to say - it all looks good to me.
My summary on the key points made in the Internet section of the report are:
- Child pornography and other obscene material is more readily available because of the Internet (and is already illegal material).
- Peer to peer systems makes it harder to control the distribution of such material - suggests law changes to make clear offering files on a P2P system is "supply"
- Recommending that filter software be made available (ideally free of charge)
- Education of users should be encouraged
- Live shows are not covered by censorship/classification rules - "let's change that"
- Ability to execute a search warrant on grounds of possession, rather than trading
- Have ISPs adopt a code of practice - if they don't (and NZ Telecom's Xtra is noted as dragging it's heels), then force one on them.
Nothing in there that alarms me too much.Child pornography is the bogey man, and the vehicle on which everything else rides, if anything extreme is going to be introduced.
The reality is that BG would go right past Sol III and continue searching for the real lost colony of similar (or more advanced) technology and write us off (like the other humanoid planets they passed) as being a side-show.
Note that while the Paul Graham rating of 0.999999999999999 is high, in practice I use Gary Robinson's calculations (more refined and use even infrequently occurring tokens - I get better, less extreme results). Gary Robinson's spam rating on this is: 0.61705129961986 That may seem relatively low, but is on a different scale and is firmly indicative of spam.
Unlike Paul Graham, I don't parse out (and ignore) HTML comments. I find all information is useful, and I find it just as effective (and simple) to treat the text as a straight byte stream.
I got the same spam. No problem with my naive Bayesian filter: nrc 0.9322408 200 0.8996861 -0500 0.9530694 5329 0.9897271 u 0.9130855 s 7.409731E-02 communigate 0.9296147 contrast 4.253653E-02 codewks 0.9789982 f 0.9006658 nacjack 0.9789982 wlink 0.98057 cithara 0.9689967 feb 0.9688099 2003 0.9078071 Spam ratings:0.999999999999999 The headers are still comprpmising, and with no real English words, there's nothing to weight this message against being spam. Interestingly very few people sending me real email use the letters "f" and "u" on their own. :-)
My point remains valid. Because there is a direct cost to the spammer to adapt.
If they bulk up their spam that's going to slow them down, increase their costs (even if bandwidth costs aren't going to be passed back to them now, the more they use, the more visible they become). They become more visible.
Or they continue on their way. The reality is that they concentrate on the easy targets - you and I will never purchase their services so people taking this approach aren't really in their target audience anyway. I know this is (surprisingly) less true than one might think. Spammers do work to overcome basic obstacles, but that adds more costs and time - they don't work hard to avoid tar pits, because there are so few of them.
So I still see it as a win...large emails are very unlikely to be spam. If that changes, well so be it, but that will hurt the spammers. In the meantime I reap the benefit of fewer false positives and faster spam filtering.
Final comment - over the last six months I've seen spam get slightly larger (from about 32k peak size to about 45k peak size). But I haven't been analysing for any trends - just the outliers.
I have
Const maxspamsize = 42695
in my spam filter - I've only receive one piece of spam larger than than in the last 12 months (a giant promotion for a Korean trade show). It speeds up my spam filter processing and lets large newsletters (with false triggers like this) through without a problem.
Ok, what about my Outlook PST file - 600 Mb and changes every minute or so?
I can filter for "residents of CA" and "105th U.S. Congress" to eliminate large amounts of spam. Thanks to spammers everywhere using
Now that I'm using that Bayesian-style analysis of spam this gets even better...
Token, Spam Hits, Real Email
105th, 462, 0
Congress, 636, 8
transmissions, 632, 2
residents, 342, 11
CA, 10240, 606
So forcing/encouraging spammers to add "some crap" to their email (even if it is trying to avoid the effects of a law) just makes it easier to pick them up.
Zombies: Brains. Brains! BRAAAIIIINSSSS!
[Zombie's tap heads of /. readers and Homer J.]
[FX: Hollow echoing sound]
[Zombie's exit stage right in pursuit of more fruitful sources...]
Yeah, brain donations from /. - that'll work....plenty to spare...
Somewhat related is this approach I've been trialing quite successfully for the last month. I haven't been able to find any reference to anyone else doing this, and would welcome any comments. If it's a 'new site' (not dealt with regualrly and not seen recently) and it shows up clean on the variosu DNSBL's I use, then I send a temporary error code back. If they retur (after a suitable time delay - I use 15 minutes) and still come up clean, then I let it through. Advantages: * many spammers don't retry - ever (perhaps they get shut down, or someone closes their open relay, or they concentrate on more receptive targets) * those that do retry (often many hours later - average is 7.6 hours for spammers) are usually listed on the DNSBL's by then * I get to collect the list of mail addresses they are trying to send, and if they hit one of my spam traps (and there are many obvious dictionary attacks) then they immediately get marked bad even if they are not DNSBL'd * Doesn't waste bandwidth (or the hijacked resources of a open relay 'victim') which continually using a tar pit does Disadvantage * Genuine email from a new/infrequent source gets delayed 15 + (until their servers retry) minutes. Most geuine ISPs try at reasonable intervals - though some wait an hour. I'm willing to wait an hour for mail from someone new, who's not on my whitelist, given the amount of spam this simple technique filters. Obviously if everyone adopts this approach then spammers would deliberately work around it - but it would complicate matters for them - the time delay and reptetive nature of their attempts would make them even more obvious as spammers, and more easy to shut down. And they can't avoid the spam traps. Forgive me if this is obvious and well known - I'd appreciate any pointers to where this has been applied and any comments.
Since background radtion levels are around 0.1 mSv/hr (eg in Hong Kong) then all of us should be near enough radiation to "test if the watch even works".