Slashdot Mirror
One Third of Email Now Spam
Himanshu writes "The volume of spam received by business has doubled over the last two years and it's going to get worse.
Analysts IDC reckons that spam represented 32 per cent of all email sent on an average day in North America in 2003, doubling from 2001. That figure is less than the 50 per cent or more junk mail statistic commonly cited by email-filtering firms like MessageLabs and Brightmail but it still represents a serious problem,"
I get a ton of spam, check out some of my recent spams and a frequency plot. starting from when I began saving and filtering them. Many thanks to Paul Graham for his plan for spam, or I would be buried by 350 spams per day by now. It is only going to get worse! Based upon how many I get, the probability is more like 95% percent of my email is spam.
By using filters and mail forwarding, I haven't gotten any spam in the past 2 months, so the increase in spam is certainly news to me.
... that 1/3 of email is *not* spam. Where do they get these figures from? Is there a computer that tallies all the spam up, and if so, why can't it just kill the spam along the way?
Though he seems to get most of the spam in the company. (Thankfully, the rest of us aren't as plagued.)
Anyone know a good challenge/response program that works with Exchange? (And before you suggest a free alternative, he refuses to migrate, so I have to work with what he wants.)
libertarianswag.com
I'd like to have a statistic on how much of that spam is do to worms relaying themselves from infected networks. 80% of the spam I now filter has a worm or trojan attached. I rarely get the marketing spam anymore.
As more spam gets sent, the rate of response to spam will decrease. Which means spammers have to send EVEN MORE spam emails to get the same return on investment that they did a few weeks before.
I'm surprised it took this long for the ratio of spam to real to reach the level it has.
So things are better than the last time slashdot ran this story?
I doubt it.
-Colin
anyone know how these stats compare with standard mail?
Here's my idea that I don't have any capital for:
Run an Internet backbone that lets all traffic through except for mail. Nope, sorry, we can't transfer mail packets over. You'll have to use some other company.
Okay, so it won't make me tons of money, but think of how stress-free the support staff will be. Or maybe not.
If spam is costing corporations millions every year, there is a HUGE opportunity for arbitrage between the amount spam costs them and the amount one could charge for a, effective spam filter.
Yes, yes, I know about baysian filters etc, but no current solution is near 99.9% perfect.
I presume the problem is that a solution requires cooperation among a lot of people (ISPs, advertisers, users) who are not naturally likely to work together, and for whom as individuals there is not a significant gain from blocking spam. It's a bit like litter: few people like it, but lots of people drop it, and everyone has to live with it.
Internet providers need to configure their mailservers to accept e-mail from authenticated servers and hosts only.
Finally, digitally-signed messages should become the norm, not the exception, where it's easy for Joe Newbie to check the signature against known databases.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Right now the mail server that I admin, which has only about 7 active users, we catch about 25% spam.
I've got spamassassin installed, and it does a good job. One thing from the article that reinforces something that I've been thinking about implementing is reducing the time spent dealing with spam. Since I have a good spam filter, I was thinking of deleting the obvious spam, and then delaying the more questionable spam to be spooled until one time a day and then put in the users' mailboxes at one time. That way the user would only have to go through the scan the inbox and delete spam once a day instead of incrementally throughout the day. This will also reduce the "You've go new mail" at all if the only new mail is spam or possibly spam. The only false positives that I've seen have been solicited mass mails like newsletters, and sometimes a mail in the spamassassin mailinglist will get flagged as spam for obvious reasons. Having these false positives mailed with the other questionable spam with a delay would not be a problem.
The CIO of the company I just left always claimed that sooner or later, all professional email correspondence will take place by allowing recognized correspondence as opposed to blocking known spammers. Presumably, a person would have to go through some process to request the ability to communicate via email with someone within another company.
I don't claim to know everything, but this seems a bit far-fetched to me. Not to mention crippling a technology that has the potential to be an effective collaboration tool. I'd be interested to hear what you folks think, though.
My sig sucks.
For those who are thinking that 32 percent is a low number, note that the original post says, "...spam received by business". This actually makes some sence since business email throughput will be a lot higher than personal email throughput. For example, I typically send/receive around 3 legit emails per day from home, but I that number jumps to around 10 emails at work. If each address receives the same amount of spam, the business address will show a significantly lower percentage.
It does eight (yes, eight) tests on the subjects of every message. I havent even added body checking yet, and it catches most spam. I even tried replacing these 8 tests with the SpamAssassin engine and found that it was less good at detecting spam mails. The tests are so simple:
The blacklist is checked after first collapsing spaced-out words like "V I A G R A" and removing the above-mentioned obvious obfuscation. It's regex-based and contains the typical stuff like "meds" "medication" etc, but also a test for a subject that ends in 3 or more spaces followed by a string of random consonants.
When it detects SPAM, it simply changes the subject line to indicate that the message is spam.
In addition to spam-checking, it also removes all HTML mark-up (removes the tags leaving plaintext behind), deciphers MIMEd messages and recompiles them into multipart/mixed format (so images etc. are attachments) and renames many-extensioned attachments, so girl.jpg.pif becomes girl.pif.
It's still in dev, but it'll be available on baxpace.com in the next week or so for Win32 (as an exe) and UNIX platforms. It's written in Perl.
If every Linux and Windows machine ran Postfix with CRM114 by default (and with manpages and documentation), this would help. Maybe a new anti-spam Linux distribution is needed. MacOSX ships with Postfix, but not CRM114.
Do you have any idea how many open-relays still exist? Why does SMTP software allow '*' open-relays in the first place? Do you know how many proxy servers are out there on the Internet? How many SOCKS4&5 proxies that just allow any SMTP to be bounced? How many are seemingly closed but available with the CONNECT method? Let's close some of our holes, and prevent software from opening them in the first place.
Also - know your enemy. Why haven't people dissected the software these creeps are using. The majority of spam comes from a program called DarkMailer or DM. Let's reverse engineer this application and figure out how it works, so our defenses can be built around the enemy's weapons and not just generalizations about spam.
Finally, let's set some ethics and procedures about how to deal with spammers. Too many is the case that people just want to beat their heads in with baseball bats or delete all their files on all their computers. This activity is not productive. It's my firm belief that if you take away their tools and educate them, less spam will be out there. You make it a war -- and that's what you'll get. Passion drives creativity and efficiency.
IPv4 allocations for hobbyists? join the ipalloc-l mailing-list! www.operations.net/mailman/listinfo/ipalloc-l
While we've surely all seen enough spam, this is about the most thorough bit of spam I've seen in a long time. And its short - way more crap per line than usual.
Not only is it spam, it claims to be consistent with the CAN-SPAM act. How wonderful is that?
It has the usual set of junk words intended to try to disguise itself from the normal anti-spam software. And it has the usual image to load that contains my email address so it will know I visited there. And it encourages me to send it to all my friends. And it has the usual "visit here to get off our list".
Even better, if you go to their web page you'll find a pointer to a page where they say "It has come to our attention that ..." spammers are advertising their product, and you can complain by filling in a form. And, of course, giving them your email address! For those who are amused by such things, look at the source - its obfuscated to the point of absurdity and does not seem to like running under mozilla.
See my journal for more info, including the source of the mail, the urls involved and a decoding of their web page.
I spoke with IDC for a short article I'm writing on this release for InformationWeek. The difference between IDC's figures (32%) and those of anti-spam vendors like Brightmail (63%) comes from the sample. IDC's sample included internal corporate mail sent by respondents to each other. As might be expected, mail sent from employee to employee tends to include fewer mentions of Viagra. Brightmail's statistics are based on mail traversing the Net.
It's a similar situation here in the States. Although I don't work for the United States Postal Service, those that I know that do are nice, hard-working people. I would estimates that 80% of our USPS mail is junk. At least the senders are paying the USPS to deliver it!
Mind you, that would require too much work for their pea brains.
I do have another solution, though. Since I control the mail server user accounts where I work, I can just create a new email account every week and invalidate the old one. Or create an email account just for usenet postings :-)
We are using a new product called GWGuardian that we spotted at Brainshare. On average I was recieving somewhere in the range of 1500+ SPAM messages a week. With the GWG I have had 1 Spam mail make it into my inbox. Have to love it.
While the economics of email favor spam, spam will flourish. It's as simple as that.
To get rid of spam, we need to change the economics of email.
However, most systems proposed are too simple in that they serve to make a lot of the legitimate purposes of email too expensive, Maillists being a primary one, as well as mail from new potential customers.
Essentially, we can arrange email into a grid of Expected or Unexpected vs Desired or Undesired. We need a way to freely receive all Desired mail whether it is Expected or not, while making it expensive for mail that is both Unexpected and Undesired.
To address this, I believe a system where the promise of payment is encoded into the delivery may solve the problem. Note that the promise of payment doesn't mean that payment will be necessarily be required. However, having the promise encoded into the email does require that it be possible to place a charge on that email by the recipient. This would require verification at intermediate servers that the mail came from a known system that allows payment to be made before relaying it on.
Legit users send out so few emails that they could easily send out mails with promise of payment encoded, companies would not require the payment be made (as what a great way to lose a potential customer) so the status quo is preserved, and friends who they send mail to similarly would not bother requiring payment. Of course, if payment is required (you get into a fight with your friend) it should be a small enough amount (sub-dollar range) that it is not an extreme hardship even then.. provided you're only getting charged for one or two.
Mail-lists could be sent without the promise of payment, but since they are typically subscribed to directly, it becomes very easy to implement a white-listing solution for all the lists you're on.
Spam could not be sent using promise of payment -- if it was, the costs would quickly dwarf the profits since it is only the very low cost of email that makes spamming possible. Anybody receiving the spam would simply click the "Require Payment" button or some such, and the spammers credit card would be automatically charged the amount. Assuming only 25% of the recipients are actually able and willing to require payment, since the typical spam run sends out hundreds of thousands of email, the charges mount significantly quickly. Yet if spam was forced to not promise payment, since all legitimate email is using promise of payment, it becomes very easy to whitelist the spam out of existance.
Essentially, the promise of payment system allows unexpected but desired mail to proceed as normal, while unexpected undesired mail incurs a fee. Expected mail can use the standard email system with whitelists, or still use the promise system with no difficulties.
That Jesus Christ guy is getting some terrible lag... it took him 3 days to respawn! -NJ CoolBreeze
I have to assume IDC based its studies on mail filtering reports and technologies using servers that at some point, started deferring SMTP traffic and didn't actually compile complete stats on spam. There's NO WAY the spam-to-legit ratio is 33%. It's more like 85%, especially for any boxes hosting e-mail addresses which may be on file with domain records.
That study is flat-out inaccurate. When they use those lame content-based filtering systems, their mail system slows down so much, they cannot handle all the inbound connections so they never really know how much SMTP traffic they actually get. Spammers hit their lame servers, get deferred, and don't come back. I guess this might be one reason why you might want to use MS Exchange: it's so slow it can't actually process all the spam sent to it, and then you get incomplete figures on mail traffic and spam.
IDC estimates that each worker would spend an average of 10 minutes a day dealing with spam.
That seems a bit low to me. Maybe with content-based filtering in effect. But they should also ask IT managers how much time is wasted per-employee looking for legitimate messages that have been held up by the inbound mail filtering/flagging systems that erroneously trap legitimate mail. I bet that figure is much higher.
RBLs work. Content-based filtering doesn't. This whole study is basically a shill for promoting more ineffective "strip-searching" of e-mail content as a "solution" [sic] to the spam problem.
If you're lucky enough to get a valid email address, feed it in to your other spam (using their handy verify^H^H^H^H^H^Hunsubscribe link). Also useful for abuse/postmasters who do nothing.
Seriously though, nothing will happen as long as China (and a few other countries) don't care. A spammer recently picked up my cable address (which I don't use), and hits me 2-3 times a day. I've traced it back to china, contacted the appropriate admins, and received a "abuse mailbox full" bounce.
I use Macs to up my productivity, so up yours Microsoft!
Seriously- if you think about it, spam may be our last hope for privacy on the net. The more legal measures we put against spammers, the more freedom we lose ourselves. So why not just accept spam as a fact of life and find some useful purpose for it, like camoflage for stego. I know there's several stego programs out there that disguise their transmissions as spam- if we get rid of the spam, no more camoflage. Don't get me wrong, I don't like getting ads for pr0n at work any more than anyone else, but I think there are other ways of dealing with it- without legally screwing ourselves in the end. (pun intended)
DISCLAIMER: This post was not checked for speling and grammar- if you complain- you're a whiner
>>I know, this is against the old rule "never respond to spam, Spamers will pick it up and use it for spamming"
I hear this a lot, like people saying the 'remove' link is to just verify your email address.
I don't think I buy it. I don't think they care or have a reason to care if the address is good or not.
What happens if they don't get a response? They just send more. They don't care if the address is valid or not.
It doesn't cost them any more to send to them, in fact once they have a connection to send spam they don't have any reason to purge the bad addresses. Why spend time doing something that won't save you money?
I really think the 'remove' hits are just ignored, doesn't make any difference if they are valid or not. And, they can always claim they have the 'un-subscribe' that many laws require.
Here in Vancouver, I got the postpeople to put a "no junk mail" marker on both my home and post-office mailboxes, as a result I don't get any of the flyer-type junk mail. Also in credit applications, I always report the lowest income I can get away with to discourage direct-marketing mailings. That and asking everybody for no direct-marketing contacts cuts my junk mail down to almost zero.
Interestingly, when I asked USPS to put a "no junk mail" marker on my American post-office box, they laughed at me.
> Fewer complaints
They are anonymous. All the information is forged. They never hear the complaints.
> far less likely that they would end up in court for spamming.
Court where? in China? in Russia? Who do they send the supoena to? See above.
> they would at least try to target interested people. They would honor unsubscribes. They would put legitimate info in their header.
Why? What would they gain by going to these difficult lengths? It doesn't cost them anything more to target EVERYONE. The interested people get the spam.
> None of that would make it acceptable to me,
Me neither. I hate them. I hate my overflowing mailbox. But I am pointing out the realities of the situation.
> if most spammers did that, congress wouldn't be passing laws about spam, and far fewer people would complain about it.
They don't care. The laws and the complaints don't affect them.
> they are forcing people to get decent spam filters
Now THIS is true. Our filters are getting better, which cuts down their audience. But of course they are in this for the quick buck and their business has no happy medium with "considerate marketing". But of course their profits trickle down to hackers for hire who keep sneaking through the spam filters.
> they get a lot of complaints.
No they don't. Lots of people are complaining. It's not the same thing.
- For the complete works of Shakespeare: cat
Spam levels must get worse, and will, before anything changes. Email will become useless (if it hasn't already) before. I imagine that by that time though, we may be using another form of communication (Instant Messaging?) as our primary means instead.
I know I'm about to beat a dead horse here, but...
SMTP must evolve or be eliminated.
We cannot keep duct-taping a protocol that is clearly not suitable for the internet of today and hope for any real success.
internet email has become absolutely worthless now due to spam, IM programs have become the better tool for personal communications, that and private messages in forums. Restricted corporate email is the only truely usable email to use now due to their ability to filter everyone but the companies own domain and those in a list.