Slashdot Mirror
Forty Percent of All Email is Spam
PCOL writes "There's an interesting article on spam in today's Washington Post which includes an inside look at AOL's spam control center in Northern Virginia. The story reports that roughly 40 percent of all e-mail traffic in the US is now spam, up from 8 percent in late 2001 and nearly doubling in the past six months; that AOL's spam filters now block 1 billion messages a day; and that spam will cost U.S. organizations more than $10 billion this year from lost productivity and the equipment, software and manpower needed to combat the problem."
I administer a Spam filter for a state University in Tennessee. Since I began filtering, I have trapped about 42% of all email bound for faculty and staff. Some spam still gets through, but the impact on our pop and imap servers has been greatly reduced.
550 Spammer Go Away!
So, we all agree that Spam is a problem. We all agree that legislating Spam out of existance isn't going to work, due to the international design of the Internet. So what needs to be developed is a backwards-compatible mail transfer protocol that authenticates the user to the sending server and forwards the message to the recieving server, who contacts the sending server back and verifies the user's identity.
I'm no software designer, but surely we could find some concept for migrating off of SMTP and POP and to a better, more secure protocol.
Other thoughts?
-cheezus_es_lard
The real problem with spam is the economics: it costs next to nothing to send a message, the only real cost (time) is borne by the recipient. Fix that problem and spam will go away. It doesn't need legislation, which in any case could apply in just one jurisdiction.
A system like Hash Cash could solve the problem. The most popular free mail clients could start including hash-cash postage with each sent message, and then in a couple of years' time start to drop incoming messages that don't have postage paid. AOL could include hash cash in their mail client easily. *Easily*. That spam-detection centre they run is not cheap. Even Microsoft would add hash cash to Outlook, Outlook Express and Hotmail, since it's another encouragement to upgrade to a new Outlook release (which of course requires a new Windows version).
Getting the whole world to upgrade its mail clients is a hard task, but getting every government in the world to pass anti-spam laws and enforce them is much harder. Goodness knows it's bad enough trying to get _one_ legislature to take a sane view on anything technology-related.
-- Ed Avis ed@membled.com
My theory: most spammers are the cyber equivalent of "flashers" - sexual deviants who derive thrill from shocking unsuspecting citizens. I believe that the products offered are largely irrelevant. It is the shock value which motivates the spammer. Perhaps they could be prosecuted under similar sex crimes laws that allow us to go after the "flasher".
According to POPFile only 18% of my email messages are spam, but it's 46% when you take the file sizes into account. The total memory fraction would seem to be a more relevant measurement if you're an ISP concerned about spam's costs.
So, when they say 40%, is that by number of messages or total size?
Liberal (adj.): Free from bigotry; open to progress; tolerant of others.
i run a small isp's mail server system (~30k accounts) and just our dnsbl blocks about 60% of all incoming e-mail. spamassassin and various other techniques pick out about 5-10% more of the overall.
Blocking spam before it gets to our main mail server has extended the life of our mail server indefinately. The less we have to spend on hardware, the more time and energy we can spend on building quality of service for our customers. That keeps the customers happy, and keeps the business people doubly happy, since they don't lose customers and don't have to buy new hardware every year for a mail system.
Why read the article when I can just make up a snap judgement?
AOL does no filtering on the content only on the header information. It does nothing with the content of the email messages. It forwards every mail that is accepted by its mail servers to the users. Thats why AOL only blocks about 50% of the stuff. Even if they accepted the mail, they should be deleting or giving me the option of deleting without seeing every mail that wants to increase my unit's size or my wife's boobs and the pharmacy come ons and the Norton junk. But AOL continues to act like a single lost email is the end of the world. Well give the users some tools and let them decide. No wonder they are losing subcribers, they don't know how to deal with the number one annoyance on the internet today.
And then what do we do what a company hires an untraceable spammer to send out a million messages with its competitors names?
I know as a youth, one of my hometowns stores fliered the city with a competitors name and fake coupons for a rediclous amount off to give them a bad name when their competition was at its worst.
It finally came out the other guys had done this, but the other store decided to make a promo out of it and honor the coupons anyways...backfiring on the others.
In a smaller town, this sort of thing can be traced back to the source rather easily. On the internet, how are you going to police the fact that PacificMed's greatest competitor (would that be AtlanticMeds) by doing the same sort of thing? Find a spammer in Asia (or one that works for your local college that will simply use Asian relays) and pay them $1000 to send out a million spams either to get them in legal action or simply to give them a black eye in the public's mind.
clif
We use BrightMail and are very happy with them. If anyone can give you fairly accurate stats, it is them due to how they work.
They monitor a LOT of mail boxes...many customers plus many created mailboxes for spam. If a message hits a number of mailboxes in a short time span that message is forwarded to their NOC. A person looks at it and decides if it's spam. If so they tag it as spam before sending it to other customers that receive it.
It works very well. We now block almost all of the spam we receive and have not had ONE single false positive.
On the server I administer, I have a nightly cronjob set to parse the spamassassin logs, and email me the stats.
Since the logs were cycled on Sunday morning, there have been 8332 messages, 5824 of which were spam, for a percentage of 69.89%.
This number has increased substantially over the last 3 weeks. This time last month we were below 50%.
"The guide is definitive, reality is frequently inaccurate."
My life as a spammer (in brief):
/. on the subject. Anyone interested in reading about how I turned a malicious spammer into an honest netizen?
Started working for new company under contract. Help the bossman w/ his spam. Make him do it legitimately by unconfirming all lists and sending reconfirmation notices. Result: 60% reconfirm (including people who had reported us for spamming before). Now we have nice, clean lists and the reply-to/return-path headers are actually LEGIT! Imagine that... an honest bulk mailer. Too bad our rep is already soured. We even have people who are afraid to click on the unconfirm links for fear of being added to another list.
I'm thinking of writing an (anonymous) article for
-- S
Corporate speech and individual speech are equally protected under the First Amendment.
Seriously, what gave you that idea? Are corporations citizens? Do you think they have the right to vote? Does the second amendment apply to them? Does a sufficiently old corporation have the right to run for president, if it was founded in this country?
My impulse is to think that was an incredibly asinine statement, but I do not claim to be an expert on constitutional law. In fact, "mildly informed" is putting it too strongly. So educate me, back up the claim that "Corporate speech and individual speech are equally protected under the First Amendment."
God is real unless declared integer
I could be wrong on this but having looked at the hash cash site I think that no communication from receiver to sender is necessary. The problem is based on the message body and the recipient name. The sender knows these at the beginning.
The costs to ISPs in the short term will be no worse than at present. In the long term costs to ISPs will fall as spam traffic declines.
You are right that adoption is a problem but that is no reason not to start now. Of the 10% of messages I get that are not spam, almost all are from relatively knowledgeable people who can upgrade to the latest version of Pine or whatever to get hash postage. For other users, it just needs AOL or Microsoft to put out a new release, which as likely as not will be an automatic update. Attaching postage to your message increases CPU load, but only for a few seconds per message sent, and even that can happen in the background.
The advantage over the status quo is that legitimateness of a message can be checked *automatically*. That is the point, you don't have to have your time wasted by checking and deleting spam, this job can be done by the computer. Children do not have to look at pornographic messages, etc etc. Saving time for humans, not computers, is the most important thing. Though like I said, in the long term making spam uneconomical will reduce the load on ISPs as well.
And unlike Bayesian filtering there is no way around it, the message has to cost a few seconds of CPU time or else the postage will not be valid. (Assuming the hash function is cryptographically secure in the sense there is no easy way to get either partial or total collisions with a given hash value.)
-- Ed Avis ed@membled.com
I work for a medium-small ISP in FR. We host around 6500 domains and 150k mailboxes.
Our abuse department is manned by one person 365 days a year, a bunch of scripts, a largish database integrated with our customers database, and lots of red tape. This person calls our customers when they are the source of spam or other non UCE conforming use of our network (including running an open-relay). He explains the situation politely and asks the customer to conform to the policy written in the contract. If the customer does not comply after the first warning, he must look for another ISP to do business with, for we send him an official letter (with official receipt acknowledgement)each time we interact with him.
All in all, given our company size, a bit over 1% of our costs are burnt by our abuse department. Needless to say, we relay these costs to our customers, as do most of our competitors.
This is only half of the cost of spam from our point of view. Our mail servers farm is sized in order to perform well even with 40% of the mail being spam. These are larger human and hardware costs associated with spam as well (though more diluted and thus difficult to pinpoint).
Spam costs people and companies a lot of money, we feel the need for the Internet mail system to be reengineered in order for the cost of sending email to become high enough so that spammers don't get away with their offense.
The Brightmail report is not a big surprise.