Slashdot Mirror
One Third of Email Now Spam
Himanshu writes "The volume of spam received by business has doubled over the last two years and it's going to get worse.
Analysts IDC reckons that spam represented 32 per cent of all email sent on an average day in North America in 2003, doubling from 2001. That figure is less than the 50 per cent or more junk mail statistic commonly cited by email-filtering firms like MessageLabs and Brightmail but it still represents a serious problem,"
One-third of e-mail is spam? But nine out of ten of my e-mails are spam... Nobody loves me. :~(
I get a ton of spam, check out some of my recent spams and a frequency plot. starting from when I began saving and filtering them. Many thanks to Paul Graham for his plan for spam, or I would be buried by 350 spams per day by now. It is only going to get worse! Based upon how many I get, the probability is more like 95% percent of my email is spam.
The only thing more annoying then people who post just to say they got the first post.
By using filters and mail forwarding, I haven't gotten any spam in the past 2 months, so the increase in spam is certainly news to me.
Then who is getting the other 66.6% of my email?
-------
Support Indy Music. Buy
From the logs of our anti-spam appliance, over the last six weeks or so:
That's right, about 96% of our email is spam, viruses, or otherwise ungood.I'd be delighted if the spam dropped off so it were only 32% of our mail. Think of all the things I could do with that extra bandwidth...
In fairness, the study says they were looking at businesses, and this is at a small ISP, mostly residential customers. But it's a good number to chew on nonetheless.
spam really needs to catch up. I know that over half the snail-mail I get is junk mail...
... another 2/3 to go then our job is done.
Sanford Wallace
I've had the same domain name for around ten years with a catch all email acount. 1 in 3 is nothing, for me its closer to 99 out of 100.
"Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
I am tilting the scale the other way... My spam is more like 10 spam 1 normal mail. I guess I don't send so much email -- quite a bit of IM, phone, and the age old, walk to the person :)
S
... that 1/3 of email is *not* spam. Where do they get these figures from? Is there a computer that tallies all the spam up, and if so, why can't it just kill the spam along the way?
One third of my regular mail is junk mail, and it's been that way ever since I can remember. Why should email be any different?
It's about time it started going down.
"After I received 80,730 different emails trying to sell viagra, I started to wonder: How many different ways are there to spell Viagra?"
http://cockeyed.com/lessons/viagra/viagra.html
I think that they goofed. 1/3 of it is virus infected, another 1/3 is spam, and the remaining 1/3 are jokes from people that you barely know that are not that funny.
Though he seems to get most of the spam in the company. (Thankfully, the rest of us aren't as plagued.)
Anyone know a good challenge/response program that works with Exchange? (And before you suggest a free alternative, he refuses to migrate, so I have to work with what he wants.)
libertarianswag.com
I get about 2500 spams a week to my work address, and I can't change my work email. It's on my business cards, and as a DB geek they won't get me new ones :(
Because of the extreme amount of spam that I get, my Bayesian spam filters are pretty strict. I lose valid email all the time!!!
Why just this morning, I came in and was going through my spam folder, and found that my good friend Gooshot Moneyface has been trying to get in touch with me! I was wondering why I hadn't heard from her for so long.
Why should I argue rationally with someone being irrational? I'll just mock them instead.
According to this article the problem is worse
"A diplomat is a man who always remembers a woman's birthday but never remembers her age." -Robert Frost
I'd like to have a statistic on how much of that spam is do to worms relaying themselves from infected networks. 80% of the spam I now filter has a worm or trojan attached. I rarely get the marketing spam anymore.
People who FAIL it are pretty annoying.
As more spam gets sent, the rate of response to spam will decrease. Which means spammers have to send EVEN MORE spam emails to get the same return on investment that they did a few weeks before.
I'm surprised it took this long for the ratio of spam to real to reach the level it has.
They must really love you
:-)
They must think the sun shines right out of your arse, sonny!
I'd love to only get 1/3 of MY mail as spam
Ooh ooh ooh, my idea of heaven is to only get 1/3 of MY mail as spam
What I wouldn't give to have only 1/3 spam.
Nail them up I say!
(With apologies to MP
Simon.
Physicists get Hadrons!
So things are better than the last time slashdot ran this story?
I doubt it.
-Colin
OK, so some I can understand, like how to make millions of dollars by investing in some guy in Nigeria. Or increase the size of your sexual organs (though I'm disturbed by the ones that state "I went from 2" to 6"!" I mean, my 2 year old son is 2", you know? What of freaks are in these testimonials?)
But the ones I really don't understand are the "stop spam with this email!" It's like the phone company selling you caller-ID systems that block unlisted or telemarketers numbers - then sell the telemarketers systems to get through those.
That would never happen, right?
52 Weeks, 52 Religions with John Hummel
As a person who likes to eat spam, I am offended!
anyone know how these stats compare with standard mail?
http://www.idc.com/getdoc.jsp?containerId=VWP00020 4
What kind of crappy article is this? IDC has an article labelled "40 years of mainframe", and the only OS they mention is s/360? And a quote from this article:
" As it works to raise processor utilization rates to acceptable levels, the mainframe environment has been able to prioritize and balance the workload needs through well-established operational automation and virtualization techniques."
To acceptable levels? You won't find a single OS that is as capable of processor utilization then z/OS.
IDC says SPAM will increase? WOW, what an amazing prediction, do they have Nostradomus working for them or what??
Mod +5 Drunk
i don't understand why free email servers like hotmail don't do more to avoid spam. Firstly, i think the way hotmail asks it's users to identify and then block spam mail is seriously flawed. One, on an average 1 in 2 mails are spam. Two, and then identifying and blocking that mail is a serious inconvenience to the user. Thus, most spam mail goes through the inbox.
Alternatively i like microsoft's idea: to charge a penny for every email sent. I'd gladly use such a system. It gets so frustrating to just delete the spam mail everyday and everyhour of the day. I've almost stopped using hotmail because of the spam.
Comment removed based on user account deletion
I get between three hundred and four hundred spams a day. I get, 50-100 valid emails a day. Thank god for Spambayes.
currently running around 70% at my work domain - and that's not counting the fact I don't process email for non-existant users. When I do it's more like 85%.
Oh I wish I only 1/3 of my email spam..
One of three? Seems awfully low!
CDE open sourced! https://sourceforge.net/projects/cdesktopenv/
Here's my idea that I don't have any capital for:
Run an Internet backbone that lets all traffic through except for mail. Nope, sorry, we can't transfer mail packets over. You'll have to use some other company.
Okay, so it won't make me tons of money, but think of how stress-free the support staff will be. Or maybe not.
Comment removed based on user account deletion
I heard about it here on /. and installed it the same day. At first it marked ALL my mail as spam because I'm on a few list servers, but the adaptive learning function of it is getting much better. After I "unlearned" my list mails as spam, it'd still let about 60% of spam through. Now it gets about 40 out of the 42 spams I get a day. I don't mind deleting two (or hitting "j" for junk), and recent searches through the junk folder show no false positives.
Check it out...
If spam is costing corporations millions every year, there is a HUGE opportunity for arbitrage between the amount spam costs them and the amount one could charge for a, effective spam filter.
Yes, yes, I know about baysian filters etc, but no current solution is near 99.9% perfect.
I presume the problem is that a solution requires cooperation among a lot of people (ISPs, advertisers, users) who are not naturally likely to work together, and for whom as individuals there is not a significant gain from blocking spam. It's a bit like litter: few people like it, but lots of people drop it, and everyone has to live with it.
See
Happy Spamiversary!
Celebrating Spam's Ten-Year Anniversary
U.S. is World Leader in Spam
This is by no means a good list of all the spam stories that have hit slashdot, just a list of the ones that seem to have no point, are glaringly obvoius, or are redundant.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
Filtering doesn't mitigate the problem.
So what if I don't have to see the mail? That doesn't mean my mailserver isn't using cycles to talk to some originating server, transfer, store and eventually delete that spam. The only saving grace is I don't have to pay for bandwidth on a usage basis (cable modem is still, happily, "flat rate").
But what happens if that volume gets to be high enough that it starts to affect my ability to use the bandwidth for other things?
What we have available are basically work-arounds; we need a concrete solution that addresses the basic problem.
So what is the problem? People soliciting without you opting in? Deceitful mail designed to make you open it thinking it is from a friend? The sheer volume?
The real problem is we haven't found an effective way to trace this crap back to the people supposedly "making money" with these schemes.
Solve *that* issue... put a name, address, and bank account to that spam, and we'll clean this stuff up in a hurry!
Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
My work email accounts have never recieved a spam message. Why? I don't forward crappy joke emails, I don't accept crappy joke emails, etc. I have a "spam" email account setup so I can use it for registering on websites, etc., but the funny thing is, THAT email address only gets about 1-2 spam emails a week! I have no idea why this account gets so little spam?
Mod +5 Drunk
Currently running 96% spam at home! Fortunately, I'm running POPFile which identifies 99% of it. Then Eudora moves it to my trash folder. Still, it's VERY annoying - I'm thinking of moving to a white list.
[Insert pithy quote here]
we run about 52% spam (measured what our spam filter catches and the server rejects as invalid recipient), so I think their statistics are a bit off.
if they at least spell checked their spam??
Exact quote from my latest spam:
we ever meds people of stuff Miracles it don't stuff the saw that Miracles catch stuff is all where alot overlook u alot of me best make more is happen don't want later later be meds of do Don't best self saw do Belive products told is is Don't u sure yourself best this . them is miss believe don't man later life told make be u alot things don't is be sweety sweety peace the is later of stuff is at want man the thing this peace of Disrepect Belive meds best life things should of want believe happy
Deletes every third email. No mess, no fuss.
From si20.com:
Newsflash: Spammer's fake the return address!
Probably because that other 18% is bounce messages and virus reports going to innocent addresses.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Internet providers need to configure their mailservers to accept e-mail from authenticated servers and hosts only.
Finally, digitally-signed messages should become the norm, not the exception, where it's easy for Joe Newbie to check the signature against known databases.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
The only problem is that it doesn't take into account "Viarga" and "Vagira", for which I get advertisements all the time.
where do they get their numbers... I have been working closely with my isp and thy are seeing 80% to 90% of the email they get throught their mail server as know spam/spam-bounce traffic, this they round-file immediately, in the 10% left over, we the users still recieve spam, albeit not in the MASS QUANTITIES as before, eh Beldar.
A computer once beat me at chess, but it was no match for me at kick boxing. Emo Philips
It could be that 33% is correct, as there are some lucky souls like myself that don't get spam, but do process a lot of email.
But for those who use filters, it is likely to be >50%, since why would they be using filters if they didn't have a spam problem? I don't use a filter b/c I don't get spam, but others who are overwhelmed with it will be using filters.
I know what you mean. I have had my permanent email address since 1993 or so, and I am not going to change it. I imagine it was on one of the first email address CDs that were for sale. I get so much SPAM that I don't bother obscuring my email address for netnews, etc, since I doubt it would help at all.
(S(SKK)(SKK))(S(SKK)(SKK))
1/3 of TV is advertising
1/3 of my mail is junk
1/3 of my paycheck disappears
1/3 of the day is spent sleeping
and now 1/3 of email is spam
Proof of a higher power that is laughing at us.
"Can I finish? Can I finish?
My email account of nearly 10 years old receives about 300 email messages a day. I would have to say at least 90% of the messages that come in are SPAM. I guess I shouldn't have signed up for all that free crap back in the mid to late 90's. *sigh*.
The CIO of the company I just left always claimed that sooner or later, all professional email correspondence will take place by allowing recognized correspondence as opposed to blocking known spammers. Presumably, a person would have to go through some process to request the ability to communicate via email with someone within another company.
I don't claim to know everything, but this seems a bit far-fetched to me. Not to mention crippling a technology that has the potential to be an effective collaboration tool. I'd be interested to hear what you folks think, though.
My sig sucks.
For those who are thinking that 32 percent is a low number, note that the original post says, "...spam received by business". This actually makes some sence since business email throughput will be a lot higher than personal email throughput. For example, I typically send/receive around 3 legit emails per day from home, but I that number jumps to around 10 emails at work. If each address receives the same amount of spam, the business address will show a significantly lower percentage.
...of spam through those tiny little wires...
I have enough to feed all of China... You'd think those Chinese spammers would wan't to keep all the spam to feed their families...
It reminds me of a Spaulding Grey line..."Once I was in Russia, and before I went on stage, I was told the audience might throw tomatos at me... I got up and mentioned the fact that I was flattered they would waste such precious items on me..."(or something like that)...
We apologise for the fault in this post. Those responsible have been sacked. -- Signed RICHARD M. NIXON
My mail provider is Yahoo. Boo all you want but I do have to say that Yahoo does a superb job in spam filtering. It is a very rare spam that gets past their filtering. I have quit looking at my bulk mail folder when on the webmail interface anymore because I have seen virtually no false positives there either.
On my home systems I NEVER see the spam at all. I have postfix, procmail, and spamassassin setup to handle it and handle it they do. First off, procmail directs ANY email that has the Yahoo X-filtered-bulk header in it to /dev/null. Anything that gets past this is handled by one of several handy procmail recipes and gets /dev/nulled. Anything that gets past that is handled by spamassassin and gets /dev/nulled. I might see 1 or 2 spams a month, TOPS, that manage to run the entire gauntlet...but then doing "sa-learn" on it brings those particular guys to the /dev/null world.
My wife gets dozens of spams a day at her job, where the network nazis require her to use outlook and wont allow her to install any personal filtering software ala spamassassin. They tell her "Sorry, we feel your pain but we are doing our 'best' to handle spam..." I encourage her to get a laptop to take to work upon which I would install linux for her AND set it up so that she rarely ever gets any spams ever again. When she gets tired of penis enlargement or breast enlargement messages to delete she may take me up on the offer.
On spam filtering, does Snotmail not do something similar to Yahoo with its bulkmail/spam filtering?
In Bushworld, they struggle to keep church and state separate in Iraq as they increasingly merge the two in America.
Comment removed based on user account deletion
A recent study finds two thirds of the Internet is pr0n, and penis enlargement does not work as advertised...
I have some Vic@d1n that might help you out with that.
to get that much spam???
I tried to get as much spam as possible in order to test spamassassin. I posted my email address on usenet and on all porn sites i've found. I have also tried installing spyware and toolbars. Internet explorer now crash on all sites but no spam so far.
Now, i resort to post my address on slashdot
sm@bigserver.hopto.org
It does eight (yes, eight) tests on the subjects of every message. I havent even added body checking yet, and it catches most spam. I even tried replacing these 8 tests with the SpamAssassin engine and found that it was less good at detecting spam mails. The tests are so simple:
The blacklist is checked after first collapsing spaced-out words like "V I A G R A" and removing the above-mentioned obvious obfuscation. It's regex-based and contains the typical stuff like "meds" "medication" etc, but also a test for a subject that ends in 3 or more spaces followed by a string of random consonants.
When it detects SPAM, it simply changes the subject line to indicate that the message is spam.
In addition to spam-checking, it also removes all HTML mark-up (removes the tags leaving plaintext behind), deciphers MIMEd messages and recompiles them into multipart/mixed format (so images etc. are attachments) and renames many-extensioned attachments, so girl.jpg.pif becomes girl.pif.
It's still in dev, but it'll be available on baxpace.com in the next week or so for Win32 (as an exe) and UNIX platforms. It's written in Perl.
And one third of Slashdot posts are First Post
Here's what I do: Bitty Browser & Andromeda
If every Linux and Windows machine ran Postfix with CRM114 by default (and with manpages and documentation), this would help. Maybe a new anti-spam Linux distribution is needed. MacOSX ships with Postfix, but not CRM114.
Do you have any idea how many open-relays still exist? Why does SMTP software allow '*' open-relays in the first place? Do you know how many proxy servers are out there on the Internet? How many SOCKS4&5 proxies that just allow any SMTP to be bounced? How many are seemingly closed but available with the CONNECT method? Let's close some of our holes, and prevent software from opening them in the first place.
Also - know your enemy. Why haven't people dissected the software these creeps are using. The majority of spam comes from a program called DarkMailer or DM. Let's reverse engineer this application and figure out how it works, so our defenses can be built around the enemy's weapons and not just generalizations about spam.
Finally, let's set some ethics and procedures about how to deal with spammers. Too many is the case that people just want to beat their heads in with baseball bats or delete all their files on all their computers. This activity is not productive. It's my firm belief that if you take away their tools and educate them, less spam will be out there. You make it a war -- and that's what you'll get. Passion drives creativity and efficiency.
IPv4 allocations for hobbyists? join the ipalloc-l mailing-list! www.operations.net/mailman/listinfo/ipalloc-l
I think that number might be a little low. For every 1 message that I get that's legitimate there are at least 5-10 that aren't. That would put the % closer to 85-90%.
What are other peoples observations?
One day I'll simply snap and actually contact a spammer with the following order:
From: my@email.com
To: spammer@email.com
Subject: Order req'uest for >X<@n4x and V1agro! fxfj aspll cps
Dea'r Si:rs,
I w.ould l1ke t0 pl@ce 4n 0rder for tw0 p.ortions of Xa:n:aX' and v_i_a_g_r_a. P13aS.e sh1p im:medi`ate1y, 1 h@ve an><1ety and ne:ed a -bo-ner-
Y0urs s.incerel'y
S@vvy 1nvest0r
akdf k- dfks. dfk v9iew casoji ropdfk hork
aso, ckdo ofgkf opwerk- mmos odkaok s
w eofk, eoro gksod bz o-
Comment removed based on user account deletion
I'm sorry 32% is hardly 1/3. Can the editors check the math please.
Has anyone here checked into the program ChoiceMail One by Digiportal http://www.digiportal.com? This program seems to be a good solution although it doesn't eliminate the spam it places the burden on the sender to prove the mail is legit. Check it out and let me know what you think.
Make something ID10T proof, you'll make a better ID10T.
Here at work we only have about 120 e-mail users, on average we get 3200-4000 e-mails per day. I installed a Barracudda Spam Firewall on the 4th of this month. Since then we have gotten 61,938 messages of which 49,455 were blocked as spam, 1,552 were blocked for viruses, and 397 were tagged as bulk mail and passed through the filter. Which leaves 10,534 valid messages. Considering I am still training the filter several spam have obviously made it through. This sure looks like a lot more than 32%. Not sure who they are surveying, but they should broaden their scope a bit.
Just my 2 cents
Their stats don't line up with mine -- the only thing I do agree with is that it is getting worse. It continuously has since March of last year it seems. Back then my base was about 500 a day THEN. Today it's much different, but let's digest some numbers.
:).
:)
:) yesterday alone (a typical day). This includes the harvested messages -- which now puts the email traffic at almost 98% being generated by spam.
... you do know me, right? :). Yesterday's already blocked address' attempting to send even MORE spam in was 2,251 for a total of 2 email address' which may send/get 6 emails in the same time frame. Now we're at 99.7% of the potential email traffic was all generated by spam. .3% was real.
Forgetting work -- let's just look at my home domain. Hosting my wife and myself I'll look at my email alone. In the last week we've sent/received 42 legit emails. That's about 6 a day between the two of us. In the same week the average _daily_ traffic looks like this:
I'll start by saying that actual junk mail that may make it to the Inbox in front of me is maybe 1 a week. I find even that annoying. Yesterday, an average day -- there were 109 messages harvested by spam sucking address'. Our daily average [last typical week] at home was 6 emails (sometimes less, sometimes more
By my numbers that is almost 95% of my email traffic which is simply not wanted, nor allowed.
There were also a total of 291 subnets blocked (for various other noticeable offenses
Of course, once blocked there's a URL sent back (-0- lookups in the same time frame) which tells you what to do (email a unblocking address or pickup the phone and call me
They're numbers, well -- just don't jive with my real life experiences.
I realize that it's only an option on si20. I'm complaining about the general concept of authentication, and the selfishness of the people who choose to use it... Not si20.
Though I doubt this figure (it seems more like 90% of email is spam), snail mail has far more spam, nearly 100% of my snail mail is spam, the rest is bills.
While we've surely all seen enough spam, this is about the most thorough bit of spam I've seen in a long time. And its short - way more crap per line than usual.
Not only is it spam, it claims to be consistent with the CAN-SPAM act. How wonderful is that?
It has the usual set of junk words intended to try to disguise itself from the normal anti-spam software. And it has the usual image to load that contains my email address so it will know I visited there. And it encourages me to send it to all my friends. And it has the usual "visit here to get off our list".
Even better, if you go to their web page you'll find a pointer to a page where they say "It has come to our attention that ..." spammers are advertising their product, and you can complain by filling in a form. And, of course, giving them your email address! For those who are amused by such things, look at the source - its obfuscated to the point of absurdity and does not seem to like running under mozilla.
See my journal for more info, including the source of the mail, the urls involved and a decoding of their web page.
And this is one of them. In the past 7 hours, I have had 56 new emails, of which 1 of them was not spam.
... is if they count the volume of "intranet" mail.
Corporations deal piles of mail on the inside, that never gets out to the genpop: HR crap, memos, meeting notices, etc. etc.
Customer relationships also generate piles of e-mail, but that should be visible to your average slashdotter who buys stuff.
I wonder if they're counting automated, machine-read e-mails such as SEC filings and other things that humans never read?
Design for Use, not Construction!
Maybe if we sent some terrorists, murderers, etc. the addresses of all known spammers, nature would take it's course.
I spoke with IDC for a short article I'm writing on this release for InformationWeek. The difference between IDC's figures (32%) and those of anti-spam vendors like Brightmail (63%) comes from the sample. IDC's sample included internal corporate mail sent by respondents to each other. As might be expected, mail sent from employee to employee tends to include fewer mentions of Viagra. Brightmail's statistics are based on mail traversing the Net.
Then there's the idiot spammer who keeps sending messages with subjects like "$random_1 $random_2 $random_3"...
What I'm listening to now on Pandora...
i do desktop and server support for small buisnesses that cant afford a fulltime IT staff. some of them run their own mail servers are are getting hammered pretty bad. Can anyone suggest a decent, doesn't have to be perfect, server side anti-spam filter? my only requirement is that because they're small buisnesses they don't have alot of money, so its gotta be for windows2000, and easy (1-2 hours) to install and setup so im not there all day because thats costly, and free as in beer?
Everybody denies I am a genius--but nobody ever called me one!
No way there is 1/3.
:->
Try 99.99%
Without serious filtering there is just no way email is even useable anymore.
The problem is bad.
Couple that with blacklisters blocking legit emails and you have a completly fuked system that only the savy can traverse.
What is even more entertaining is you have Google entering the market. Do they even have a strategy?
So far the best laid plans have been laid to waste here on Slashdot.
Maybe I should stop posting my email: kevin@qualico.ca around here?
So who gets all that? I do no no spamfiltration at all and I get about 2-3 spam mails a day. That is mails that are commercial unsolicited email.
:-(
Other types of junk is much more annoying. Error messages from servers claiming that I sent some mail. Not to speak of virusmails!.
I get 25 a day, from the same person, who still has that virus after 2 months. ISP notified, but no reaction. Now, I started sending bills for the handling of virus mails, 200 USD each... But I doubt that they will pay
And irrelevant discussions on maillinglists, could be nice to block a thread on a mailinglist...
The email accounts that are "visible" (i.e. on the webpage and used to register online) get close to 75% spam. It is worse for the individuals who do a lot of internet surfing. However, on the "low visibility" accounts, those only released to customers and suppliers during phone calls, it is more like 1%. I guess it would balance out to 35-50%.
I was taking one day at a time, but then several days got together and ambushed me. (from a Rhymes with Orange comic)
My spam rates aren't nearly as bad as most, and it might also have something to do with an additional step I take:
I use multiple email address. I use [name]@[domain] for my main email. I haven't had to change it in years. If I sign-up anywhere online, I use [name]2@, and if I buy anything online, I use [name]3@, and if I post a resume, I use [name]4@. I've had this in place for about 5 years now (mostly to manage my sanity), but I haven't even had to increment the numbers yet.
Works pretty well, in additional to SpamAssassin.
90% of spam is easily recognized by even the most basic spam filter. So, is this still a problem?
Has anyone noticed that stopping spam seems to be just as difficult as stopping file sharing? The two problems are very similar, and the funny part is that so many people think passing laws against spam is going to matter.
Business isn't willing to pay for products, innovation and careers, so we get brands, mortgage commercials and layoffs.
Define "near". My current Bayesian solutions(POPFile) is 99.33% perfect. 96 bad classifications in 14,544 emails. 70 of those were spam marked as good. Most of the rest of those 26 came from not resetting the statistics since installing. The first time I checked email after installing everything was "wrong", since it obviously had no data to work with. I can think of perhaps 2 or 3 emails (in 14,544) that were legitimately classified wrong, and one of those was a stupid forward that might as well have been spam.
Oh dear god, if you have never seen one of these, you might be best served by gouging out your eyeballs now, just in case.
I have misplaced my pants.
I'd give my left ball for getting only 32% spam. My current rate is over 70%.
Engineering is the art of compromise.
I think there are two kinds of population or, more precisely, email addresses : those who are on spam lists and those who aren't.
The first receive little or no spam, little is when they give out their email to a corporation that resells it or when their email is discovered through brute force guessing. For that category, spam is one message in a while.
The second have, in some way, published their email on the internet. Their adress has been catched by spammers at some time. They will receive more and more spam as their email "circulate" among spammers. After a few months, it will get to more than 100 messages a day which is more than 1 third of the legitimate messages unless you receive a lot of messages.
So, if you publish your email on the net, don't complain about spam, it's too late.
Men are born ignorant, not stupid; they are made stupid by education. Bertrand Russel
To some extent the huge volume of spam is a result of the increasing abilities of spam filters. To see why this is, suppose you're a spammer who needs to send one email to every person on your list in order to get enough responses to make a profit. If there were no spam filters in the world, you'd need to send one and only one message to each person. If there were spam filters but they only caught 50% of the spam, you would need to send two copies of the message on average to each person. If the spam filters were 95% effective, you need to send 20 copies. As the filter efficiency goes toward 100% the amount of spam you need to send to remain profitable goes to infinity. Since filters are typically climbing into the upper ninety percent range, we're just really starting to enter the interesting part of the spam curve.
I work for a company whose primary role is filtering spam from client servers. They point their MX records at us, and we send their good mail on to them on the other side.
We see approximately 60-70% spam, maybe 3% virii.
Thing is, they are probably counting internal e-mails, mailing lists, and whatnot as not spam.
I personally get about 300 messages a day at my business e-mail account, very few are spam, mainly because 250 of those are e-mail lists that I subscribe to.
(well, yeah, that too, but i really am talking spam here... =))
I have 4 email accounts I keep. One is strictly work related, and is provided by my employer. One is my personal account, provided by a third party (i.e. not my well known ISP). One is a hotmail account that i use for mailing lists. And the other is a yahoo account that is used solely for spam, online forms, etc. And you know what? The only account that gets any serious spam is the spam account. My hotmail account - it receives maybe 1 or 2 a spam messages day, scattered between hundreds of other mailing list emails. My personal account - 1 or 2 pieces a week, max. My work account gets none, altho my bosses's similar account gets flooded. I don't use external software to filter any of my accounts. However, I am very very anal about where my email addresses go, and I think my lack of spam is partly due to that, and partly luck. Who knows. I'm not complaining.
I'm one of those people who would be more pissed off at having to change my primary email address than my phone number. In a smilar vein, i *think* i'd rather give up my phone than email with broadband. Sad and pathetic I know, but it's the way the my world works.
I wish 30% of the mail coming into my organization was spam. As it is, we get between 60-70k messages per day that are bogus. That's for an organization with 10 people. There are MAYBE 400 legit emails per day.
[sig] 10 + 10 = 100 [/sig]
Just instincts on this one, but I would have to disagree with the prediction that it will get much worse. Mabye a little worse in the short run but this problem has hit such a boiling point that a great deal of attention is now on the problem. With all of the effort from software companies and lawmakers and even ISPs that once kept themselves out of the loop I have every confidence that one of these groups if not a combination of them will succeed in curbing spam and greatly reducing it.
(If at first you don't succeed, do it different next time!)
I believe for my company, SPAM represents over 80% of all the emails we receive.
Detected spams/total emails from our logfiles:
last week: 6317/7496 (short week because logfile rotation was a day late)
1 week ago: 7469/8956 (long week because of delayed logfile rotation)
2 weeks ago: 5984/7293
These numers do not include the emails that we reject besed on sender IP address, bad recipient, etc.
The real "Libtards" are the Libertarians!
With a name like Diptheria P. Cardboard, I'm thinking this person isn't getting too many dates.
That's too bad. Most of the names I get in the once-a-day "This is probably spam" list would make great pornstar names (possibly deliberately?):
Michael Payne
Lenny Champion
Carmen Dove
Katie Dickerson
Bethany Kyle
Evangelina Horne
Andre Holiday
Bradley Bravo
Linda Love
Dylan Pike
Toronto-area transit rider? Rate your ride.
3/17/2004, Gartner Group estimated the number was more like 60-75% for their clients. Don't forget the mail sent to dead domains. We have an old domain we're now using to test spam solutions that's 98-100% spam (2% or so newsletters), and the volume is out of control.
the major advances in civilization are processes which all but wreck the societies in which they occur - A.N. White
Since I've started using Incredimail. Also been using Lycos SideSearch for my web queries - great program!
You'd think it'd be easy to do something like retest every permutation of the subject line with one character deleted, since most of what I get is Vi(agra Via_gra v|iagra etc. Or maybe "Hmm, if v i a g r and a have appeared in the line within the last twelve characters in that specific order, it's spam." How many cases are there that have twelve characters containing v i a g r a in order that could possibly be legitimate?
My main gripe over on my mail.com account is they don't filter sidenafil citrate, which is generic viagra. How many legitimate mails have the word sidenafil in them?
Introducing the new Occam Fusion! Now with sqrt(-1) fewer blades!
This message contains no text. Surprisingly, all the contents of the message has fit into the subject line. Clicking at a subject line with (n/t) for 'no text' brought you to read this incoherent drivel. Thank you for participating.
If programs would be read like poetry, most programmers would be Vogons.
We have good filtering methods available now. I find bayes to be very effective, for example.
What I can't get over is the nagging feeling that this whole anti-spam effort is rather unstructured. I'd certainly like to chain together multiple filters, combine their results, and control all of this from within my client, but combining filters like this at the moment involves lashing them together with a script, but then what client plugins there are only control a single filter and probably wouldn't work with a funky scripted filter combination.
Lets define some standard CORBA interfaces for various types of mail filter, and a simple method of connecting them so that mail classifications and corrections can be properly distributed..
I DON`T L-I-K-E Sp'AM 2
For a limited to get our new SP-AM Fill-ter for F-R-E-E at http://www.microsoft.com/office/outlook2003/.
It's guenenteed to let SP-AM and other VIRUSE in qhile still mangeling your normal email.
Moneyed corporations, non-working 'poor' and criminal prisoners are turning productive citizens into tax-slaves.
And it's only going to get worse.
The company I work for switched to white-listing a while ago and it's been great. I've thought about doing the same thing at home.
crazy dynamite monkey
We are using a new product called GWGuardian that we spotted at Brainshare. On average I was recieving somewhere in the range of 1500+ SPAM messages a week. With the GWG I have had 1 Spam mail make it into my inbox. Have to love it.
I doubt it'll keep spammers at bay forever, so I really should start looking into some more spammer hostile things I can do to my mail server. Worst case, I can always shut the damned thing down. I was ready to do that anyway. If the service is useless to me (Because filtering spam takes so long that I don't have time for anything else) why should I bother running it?
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
One third of e-mail is spam? It's more like one third of all e-mail is legitimate. Here are my stats for the last 10 days from one of my servers:
24-hr period, Accepted e-mail, RBL rejects:
----------
Apr 10, 4589, 16876
Apr 11, 4837, 15997
Apr 12, 9393, 17438
Apr 13, 8569, 15755
Apr 14, 8583, 15996
Apr 15, 8211, 18496
Apr 16, 6293, 19224
Apr 17, 3685, 18054
Apr 18, 3769, 17929
Apr 19, 7372, 17939
Based on these figures, ~ 67% of SMTP transactions are SPAM. This means AT LEAST 67% of the e-mail is bogus. But this involves RBL blacklisting of connections to the SMTP server, so when you take into account a single SMTP connection typically delivers 1-10+ spams, the figure gets astronomically high in terms of spam-to-legit mail ratio. In addition to this, about 10-20% of spam messages minimum get past the RBLs so in reality, based on our server traffic, it's closer to 85% of all mail traffic is spam.
It's NECESSARY to filter email for me, and for an increasingly large percentage of Email users out there.
If nothing else, please do your bit and use SPF records so we can tell if a mail legitimately from one of your domains.
--Mike--
While the economics of email favor spam, spam will flourish. It's as simple as that.
To get rid of spam, we need to change the economics of email.
However, most systems proposed are too simple in that they serve to make a lot of the legitimate purposes of email too expensive, Maillists being a primary one, as well as mail from new potential customers.
Essentially, we can arrange email into a grid of Expected or Unexpected vs Desired or Undesired. We need a way to freely receive all Desired mail whether it is Expected or not, while making it expensive for mail that is both Unexpected and Undesired.
To address this, I believe a system where the promise of payment is encoded into the delivery may solve the problem. Note that the promise of payment doesn't mean that payment will be necessarily be required. However, having the promise encoded into the email does require that it be possible to place a charge on that email by the recipient. This would require verification at intermediate servers that the mail came from a known system that allows payment to be made before relaying it on.
Legit users send out so few emails that they could easily send out mails with promise of payment encoded, companies would not require the payment be made (as what a great way to lose a potential customer) so the status quo is preserved, and friends who they send mail to similarly would not bother requiring payment. Of course, if payment is required (you get into a fight with your friend) it should be a small enough amount (sub-dollar range) that it is not an extreme hardship even then.. provided you're only getting charged for one or two.
Mail-lists could be sent without the promise of payment, but since they are typically subscribed to directly, it becomes very easy to implement a white-listing solution for all the lists you're on.
Spam could not be sent using promise of payment -- if it was, the costs would quickly dwarf the profits since it is only the very low cost of email that makes spamming possible. Anybody receiving the spam would simply click the "Require Payment" button or some such, and the spammers credit card would be automatically charged the amount. Assuming only 25% of the recipients are actually able and willing to require payment, since the typical spam run sends out hundreds of thousands of email, the charges mount significantly quickly. Yet if spam was forced to not promise payment, since all legitimate email is using promise of payment, it becomes very easy to whitelist the spam out of existance.
Essentially, the promise of payment system allows unexpected but desired mail to proceed as normal, while unexpected undesired mail incurs a fee. Expected mail can use the standard email system with whitelists, or still use the promise system with no difficulties.
That Jesus Christ guy is getting some terrible lag... it took him 3 days to respawn! -NJ CoolBreeze
Yeah right! AT LEAST 80% of my email is spam.
Derek Greene
If you read a little further he gives an update due to a couple more substitutions. Variations of how to spell Viagra are now up in the sextillions (kind of titillating isn't it?):
1,300,925,111,156,286,160,896 ways to spell Viagra
i agree, spam seems overwhelming, but it's really easy to filter a big chunk of it out and just click the delete key on the few that make it into your inbox.
what I hate about email is that 2/3 of it is real stuff from customers, friends, co-workers, managers, spouses, on-line acquantainces, family, REAL PEOPLE who expect a thoughtful, complete reply to their message. Even for a simple yes/no reply this can take 20-50 times longer to deal with than just deleting a spam message. The actual cost of labor to deal with email is far higher when the message has actual content, and that's what IT managers should really be worried about.
As far as I'm concerned (given the amount of spams from ADSL users and the inability of major ISP to block obvious SPAM) major ISPs don't give a damn about SPAM.
They go on about how if you click 'spam' when you receive a 'spam' then they get a message blah blah blah usual IT support canned email comes back.
They show concern in their Press - the problem is they want to charge a premium for anti-SPAM technology.
I get spam from KNOWN spammers and spam sites with absolutely fricken(US)/fuggin(UK) obvious spam contents which my Mozilla quite happily bins but stupid multi-billion dollar/pound ISP with all the technology in the world at its grasp i.e. BT (who invented the Internet if you remember from their law suits) and Yahoo! (who invented the use of silly diacritics in brands) have trouble classifying bleeding obvious spam emails.
Oh and they also mark legitimate commercial email addressed to me directly to me sourced from a domain that has the SMTP server as 'Bulk' along with the other 200 per day of spams, thus causing me to have to scroll through via their crappy Web interface, hundreds of discusting email subjects trying to find a missing email.
BT and Yahoo! email is a joke.
I've said that I'll be charging them 10 EURO for every email from site.voila.fr that I get. Don't know if that'll do much but Yahoo! has to talk to Wanadoo and lose those crappy porno-dialer merchants who host on site.voila.fr and then spam the shite out of the rest of us.
I have to assume IDC based its studies on mail filtering reports and technologies using servers that at some point, started deferring SMTP traffic and didn't actually compile complete stats on spam. There's NO WAY the spam-to-legit ratio is 33%. It's more like 85%, especially for any boxes hosting e-mail addresses which may be on file with domain records.
That study is flat-out inaccurate. When they use those lame content-based filtering systems, their mail system slows down so much, they cannot handle all the inbound connections so they never really know how much SMTP traffic they actually get. Spammers hit their lame servers, get deferred, and don't come back. I guess this might be one reason why you might want to use MS Exchange: it's so slow it can't actually process all the spam sent to it, and then you get incomplete figures on mail traffic and spam.
IDC estimates that each worker would spend an average of 10 minutes a day dealing with spam.
That seems a bit low to me. Maybe with content-based filtering in effect. But they should also ask IT managers how much time is wasted per-employee looking for legitimate messages that have been held up by the inbound mail filtering/flagging systems that erroneously trap legitimate mail. I bet that figure is much higher.
RBLs work. Content-based filtering doesn't. This whole study is basically a shill for promoting more ineffective "strip-searching" of e-mail content as a "solution" [sic] to the spam problem.
fp?
I've used Yahoo Mail forever and I must second this person's comment. Yahoo has done a VERY effective job of filtering spam. It's certainly not 100% but it's pretty darn close.
Like him, I have pretty much stopped checking my bulk mail folder because it is VERY rare that I get a false-positive.
Hotmail, by contrast, still sucks. I use it for my "commerce" account and even though I have the spam filter enabled, I really do no notice a change at all. About every 5-10 days, I have go into the account and delete all.
One way to reduce this ratio would be to send more legitimate email. C'mon, get sendin'!
:)
Two further points.
1) Lots of comments are talking about how much spam they _receive_, this article was talking abut how much spam is _sent_. Naturally since spam is sent in huge numbers from few originators but most people don't send any spam, there's a greater ratio of spam received per user than sent.
2) Many comments say spam is easy to block, but then talk about blocking spam to only a small population or just a few accounts. The genuine email expected by one company that e.g. supplies bathroom fittings will be easier to avoid blocking than a huge diverse population e.g. a University, where people work in many different areas on lots of different projects.
When spam blocking, avoiding false positives (blocking genuine email) is key.
Environmentalism is the new Victorianism. Everyone ties on a green corset and pretends we're virtuous.
I run the email server cluster for a Major Corporate Network.
Over the past three years, they've gone from 400k to 600k real emails.
In the same time, they've gone from 500k to 6m total mails.
*90% SPAM*, and it takes over two dozen servers to filter it all.
Eventually, I switched to Spambouncer, which can send auto-replies (I assume other filters can do that, too). SpamBouncer classifies mail in three buckets: Legitimate, Bulk (likely but not necessarily bad) and Spam. Now I delete both, Bulk and Spam, but every Bulk message triggers a response, informing the sender that the message didn't get read (and instructions on how to bypass the filter).
I know, this is against the old rule "never respond to spam, Spamers will pick it up and use it for spamming". But at this point, it feels it's too late anyway (and I use a special email address for this). At least I have the peace of mind that a legitimate recipient knows I didn't see the mail.
SpamBouncer had the capability to simulate a bounce (in the hope to fool Spammers into thinking that the email got disabled). But that feature got removed.
Under capitalism man exploits man. Under communism it's the other way around.
So, I'm thinking - those people who actually respond to spam? We should host an awards show for them - called "Too dumb to Live". We give them a chance to give their speeches and thank their whatevers, and then, when they leave the stage to go to the "press interviews", we can just dispatch them in some nice, efficient manner.
We should ALL do something to make the world a better place to live, ya know...
eom
Everyone is taking the wrong attitude. We need to LISTEN to the spammers and do what they say. That way we'll end up with a nation of large breasted, large d**k, people, having sedated sex while watching everyone else on the net having sedated sex.
feloneous
IANAL, but I've seen actors play them on TV
I really do think they means One-Third of Mail NOT Spam. I've read a dozen reports in the past year that said that half of all the email was spam. I know it's not decreasing. 2.5 years ago half of the email coming into a provider I contract with was getting rejected as spam. Now that number is even higher. 1/3 my foot. 3/4 is more like it.
http://www.c-command.com/spamsieve/
:-) The corpus window was originally intended to be just for my own use in testing and debugging the program, but it turned out that a lot of users liked it.
http://www.pa ulgraham.com/spam.html
http://daringfireball.net/2003/09/interview_mich ae l_tsai (partial text below)
Gruber: Where else have you drawn from, tactically?
Tsai: The new math in SpamSieve is due to Gary Robinson. It was refined by Tim Peters, and later by me. John Graham-Cumming wrote an influential document that describes a lot of tricks that spammers use to obscure their messages. The SpamBayes and POPFile projects have generated a lot of ideas about how to tokenize messages, and some of them show up in SpamSieve. Whitelists and blocklists are an old idea, of course, but I don't know of any filters that automatically train them the way SpamSieve does.
Gruber: Part of what makes Bayesian spam filtering so interesting is that the concept is so simple. How would you describe it, for the non-programmer/non-mathematician?
Tsai: It looks at examples of your spam messages and good messages and figures out how often each word occurs in each type of message. When you get a new message, it looks at where the words in that message previously occurred. In other words, does the new message look more like your spam or more like your regular mail?
Gruber: Part of what makes SpamSieve -- or at least version 2.0 -- so effective is that it also looks at where the words occur. E.g., the word "money" in the Subject is counted separately from "money" in the message body. And also that these "words" aren't just words in the English language sense, but rather any of the distinct tokens of an email message.
For example, according to my SpamSieve corpus, the token "U:remove" has occurred 1009 times in spam messages, but not once in a good message. I presume this means "remove" in a URL?
Tsai: Yes. Spam messages tend to include links to CGIs and tracking images, so that's where a lot of the really spammy tokens come from.
Gruber: Given those stats, that's obviously a strong indication that a message containing a URL with the word "remove" is spam. Compare and contrast with the word "remove" in the body of a message (and not in a URL), which has occurred 927 times in my spam, but also 75 times in my good messages. Probable spam, but far from tell-tale.
It's kind of fun to look through the corpus, no?
Tsai: I think so.
Gruber: SpamSieve's whitelist and blocklist work very well. How do you defend against spammers who use forged headers to spoof the From: address such that spam appears to come from someone you know?
Tsai: Well, first of all you can decide whether you want to use the whitelist and blocklist. If you get a lot of forged spams "from" your friends, maybe you don't want to. But otherwise, the whitelist works in an optimistic fashion. It trusts an address on the whitelist until you get a spoofed spam from that address, and then the address is disabled and it falls back on using the Bayesian classifier for messages from that address. I have about 400 addresses on my whitelist right now, and about 25 of them are disabled for that reason; the others are trusted until proven otherwise. So a spammer can fool it at most once per address.
Gruber: Version 2.0 is a significant improvement over SpamSieve 1.x. What kind of accuracy were you getting with SpamSieve 1.3.1, and what are you getting with 2.0?
Tsai: I was getting between 97 and 98% with 1.3.1. That's probably a few percent higher than most users were seeing. With 2.0 I'm getting 99.5% and up. Nearly all the false negatives are virus e-mails with senders that are in my address book. So, because of the way I've set the preferences, those will always get through.
The accuracy matches what I've been hearing from the beta testers, so I think 2.0 will be a big improvement for the general user b
help out.
Maybe total, I guess, but for me, so far today I've got 58 spam, and 25 real mail. That's about 2/3 spam.
If you're lucky enough to get a valid email address, feed it in to your other spam (using their handy verify^H^H^H^H^H^Hunsubscribe link). Also useful for abuse/postmasters who do nothing.
Seriously though, nothing will happen as long as China (and a few other countries) don't care. A spammer recently picked up my cable address (which I don't use), and hits me 2-3 times a day. I've traced it back to china, contacted the appropriate admins, and received a "abuse mailbox full" bounce.
I use Macs to up my productivity, so up yours Microsoft!
If you count unwanted email traffic from relatives, friends, etc, that feel that everything that hits their inbox surely is a "must-see" by you, then I bet the percentage would nearly double!
David
Seriously- if you think about it, spam may be our last hope for privacy on the net. The more legal measures we put against spammers, the more freedom we lose ourselves. So why not just accept spam as a fact of life and find some useful purpose for it, like camoflage for stego. I know there's several stego programs out there that disguise their transmissions as spam- if we get rid of the spam, no more camoflage. Don't get me wrong, I don't like getting ads for pr0n at work any more than anyone else, but I think there are other ways of dealing with it- without legally screwing ourselves in the end. (pun intended)
DISCLAIMER: This post was not checked for speling and grammar- if you complain- you're a whiner
Start shooting spammers.
"No problem. I have the capacity to do infinite work so long as you don't mind that my quality approaches zero."-Dilbert
Can mailservers automatically reject emails with forged headers? (to the point where it never even shows up in user inboxes)
Men believe what they want. - Caesar
Wow is that all? id have said around 90% is spam, unless you count those for-every-thing-else-theres-mastercard emails then its more like 97% :P
This comment does not represent the views or opinions of the user.
Personal experience aside I know that number can't be anywhere close to right. We do watch all the email that comes into our company and fully 75% or more of it, literally, is spam.
That's up from 40% at this time last year.
>You'd think it'd be easy to do something like retest every permutation of the subject line with one character deleted, since most of what I get is Vi(agra Via_gra v|iagra etc.
It probably isn't that hard but then most of us are too lazy to actually write a program that does it.
Comment removed based on user account deletion
But what percentage of those 14,544 emails are critically important to you?
Who knows? One of those 0.66% of misclassified email might be the ONE email that matters most.
Conformity is the jailer of freedom and enemy of growth. -JFK
and time to have a regulated opt in system of mutual emailers who all agree up front to not spam, never send un requested commercial email, and to voluntarily never run insecure systems that act as an open relay, and etc. No email *ever* gets into the new paid system from outside the system, there's a definite boundary there somehow. If your machine is detected as a relay, you get 24 hours (whatever, some practical time period) to fix it or out ya go, back to the unregulated email system like it is now. It will cost some cash to join and per month or per year to be in the closed system, and ISPs could take the lead there, offer both kinds of email addys. If you run your own server and you are your own host, same deal, you have got to pay to get inside the cool guys email system,and every email address you want to operate with is registered,and each one costs folding money. If you spam or spew forth trojans and viruses, tough noogies, you get the boot, and you CAN'T get back in, not 2 or 3 strikes, one strike and you are out. Not perfect, but the best I can come up with.
Technically I don't know if this could happen, I am not hip enough on the making of funny headers and whatnot, but seems to me something along these lines should be possible, and I am guessing if you gave people the option of opting in at a nominal fee per year, 20 bucks maybe,whatever, it would be a huge hit and widely used. Like who WOULDN'T want a virtually spam free email address for such a fee? You could still, of course, have your "ordinary" open to the world email addy and protocol as it is now if you wanted to, especially useful during a transition period, but I think email addresses should be treated the same as domain registrations are now. It's the ease of creation and lack of any major expense that makes it so hard to control.
We live with the hassle of domain names and registration, it's not perfect but it's a LOT better than any alternative,and we could do the same with email addys. Make each individual email address cost actual folding money,it has to be a high enough fee to make it prohibitive for spammers and virus exchangers to not want to do it, that would tip over the costs so far into the negative that spam wouldn't pay much if anything any longer. I would love such a system, and would care less about receiving email from outside such a system.
Fortunately, there are SpamAssassin rules to catch those. I was getting a TON of those, with my name in the subject. Finally found a rule that killed them.
Join the spamassassin-talk lists.. rules galore!
When I installed tmda as a last-ditch effort to keep it going. So far it's worked pretty well -- had about 4 spams get though in the past 6 months or so.
That's only one measure of "working well". Have you kept count of the following:
- The number of spams your system has sent in response to forged "From" addresses?
- The number of real emails you've missed, because the sender doesn't feel like jumping through the hoops you've put in place?
Duncan Murdoch
So.... who's laughing now?
Your e-mail management time costs: 2 hours per day
MY e-mail management time costs: 2 minutes per week
Yes, I recieve a LOT of e-mail. I just don't get
any spam. Of course, I PAY for my porn OFFLINE , unlike
the majority of you spam (recieving) folks.
1/3 is getting worse? I thought 50% or more of all email was spam. I'd think getting it down to 1/3 would be a vast improvement.
You see? You see? Your stupid minds! Stupid! Stupid!
Does anyopne know when SpamAssassin will come out with a version that handles Bayesian Poisoning? (That's when you get about 20 random words added at the end of the spam to trick the filter into thinking it's a real message). I've gotten to the point where I've reduced my Spam-Level filter to 1 (!) just to keep the spam at bay. I've also tried to configure some temp rules to undo the affects of Bayesian poisoning, but it doesn't seem to work all the time.
I'm now up to almost 80 or 90 spam a day. Should I nuke my learned rules and get SpamAssassin to start again?
10% of all folks don't realize that 57% of all statistics are made up 87% of the time in about 55% of all cases.
That's right. All your base.
Our job posting account has a spam ratio of 92.64% according to dspam. Our email server also uses realtime blacklists that blocks about 60% of all incoming emails.
Here is my solution to spam, utilizing a combination of SpamAssassin and Sieve scripting on a FastMail account.
:contains "X-Spam-hits" "MIME_HTML_ONLY" { :contains "subject" "Infected file rejected", header :contains "X-Spam-hits" "FVGT_S_MULTI_OBFU_3", header :contains "X-Spam-hits" "NIGERIAN_BODY", header :contains "X-Spam-hits" "RM_sl_Parens") { :contains ["X-Spam-known-sender"] "yes" { :value "ge" :comparator "i;ascii-numeric" ["X-Spam-score"] ["10"] { :value "ge" :comparator "i;ascii-numeric" ["X-Spam-score"] ["4"] { :over 249K { :contains "X-Spam-known-sender" "yes" {
First, I set my account to scan all incoming e-mail for viruses and trojans. Any e-mail with an infected attachment is automatically deleted. Secondly, I set SpamAssassin to mark any spam with the score 4.1 or higher and move it to a "Junk" folder. Any spam with the score 10 or higher that is sent from anyone who doesn't match my address book is automatically deleted. Any e-mail that is HTML only is rejected and sent back to the sender. Since SpamAssassin doesn't scan e-mail above 249 Kb in size, I have it set to automatically let any e-mail above that size into my Inbox, since it's *most likely* not spam. Then, any e-mail that doesn't meet any or all of the above criteria, but doesn't match any address in my address book, is filtered into a "Gray List" folder, which is periodically reviewed every two-three days or so. Only e-mails that don't meet any or all of the above with e-mail addresses that match my address book are let into my Inbox.
It's a rather complicated system, but it works. For anyone else that uses FastMail (it most likely won't work anywhere else due to FastMail's unique headers), here's my Sieve script -
require ["envelope", "fileinto", "reject", "vacation", "regex", "relational", "comparator-i;ascii-numeric"];
if header
reject "Message bounced by server content filter";
stop;
}
if anyof( header
discard;
stop;
}
if not header
if header
discard;
stop;
}
if header
fileinto "INBOX.Junk";
stop;
}
}
if size
fileinto "Inbox";
} elsif not header
fileinto "INBOX.Gray List";
}
"We are all in the gutter, but some of us are looking at the stars." - Oscar Wilde
That shows where the spammers are located, not from where the spam is sent. Sure, Alan Ralsky lives in the U.S., but he happily contracts with ISPs in other countries to send the spam.
I work for one of the enterprise spam filtering services, and while it may be true that only 1/3 of the mail sent in the US is spam (I don't buy it, and the article doesn't state the methodology by which they derived that figure), I can tell you that the percentage of mail sent to *businesses* is way, way over 50% spam. I'm sure our competitors would all say the same. I guess that's what the spammers mean by "targeted email" :-/
- The number of spams your system has sent in response to forged "From" addresses?
This is unfortunate, but it is also expected behavior of a C/R system, so it is unfair to say that sending these mails constitutes not working well. I've received emails from mailing list software about my subscription status, due to a spammer sending a forged email to a list server. Does this mean we should disable all automated systems? What if I want to reply to the spammer to tell him to quit emailing me, and instead this reply goes to some innocent individual who had his address forged? I say the innocent individual just needs to get some filters too.
- The number of real emails you've missed, because the sender doesn't feel like jumping through the hoops you've put in place?
When I installed TMDA, I watched the "pending" folder for 6 months, to ensure that everybody confirmed their mail. And they do. Authentication is to be expected these days. You have to confirm mailing list subscriptions. You have to be granted authorization on IM (Jabber). Why should email between individuals be any different?
Was your new boss telling you how to "Make $$$ Online At Home"?
That's true, but I now have a bigger penis.
Si tacuisses philosophus mansisses. If you had kept quiet, you would have remained a philosopher.
This [comment] advocates a
( ) technical ( ) legislative (x) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
( ) Spammers can easily use it to harvest email addresses
(x) Mailing lists and other legitimate email uses would be affected
(x) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(x) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
(x) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
(x) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(x) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
(x) Public reluctance to accept weird new forms of money
(x) Huge existing software investment in SMTP
(x) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(x) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
(x) Extreme profitability of spam
(x) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook
and the following philosophical objections may also apply:
( ) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
(x) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
(x) Countermeasures must work if phased in gradually
(x) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
(x) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!
I receive 3 times as much spam as I do actual email from lists and friends.
Ya gotta wonder who does these studies, and what are they actually using for determining SPAM from GOOD.
It seems they used the same faulty techniques used in filtering SPAM, which means they missed the majority of the spam.
Na, try again.
A third? Get real! The half dozen accounts I use/monitor/administer are running about 90%. Of course that's the average. Two are at zero and two are at 99%. Guess who DIDN'T surf any p0rn?
I figured to bounce them all for a while (with MailWasherPro) - nuttin better for opting out than telling them the address is not valid, right? Forget it. Six months of that and there's no discernable effect. I don't keep numbers but it seems to be about the same.
I'm looking forward to reading the article in this weeks InfoWorld (just arrived). The headline even says one of the proposals is from "beyond the grave" (i.e. an old proposal from Postel.)
If 1/3 of all the email that is SENT is spam then how is it that over 60% of the mail received is spam as reported by BrightMail a few months ago?? Does it have babies as it goes through the routers?? If so is Spam processed out of rabbit meat?
1=2 I'm confused?
Not really a criticism, just a comment, mostly because we're seeing more and more of this type of story in the press where a lot of people are getting the wrong impression about eMail in general.
Although it's certainly of interest from an IT professional's perspective, I'm starting to get a little annoyed at how this is reported. We seem to have a format that simply gets repeated with each new study.
Statements like "32% of all eMail" reflect the volume of messages but we are really comparing (for a crude but illustrative example) 10,000 identical messages versus 20,000 individually crafted, wanted messages, for a total of 20,001 unique transmissions versus the sysadmin's problem of making room for 30,000 messages.
As this gets reported in the press, we're seeing more people who consider this good evidence to consider restricting the use of or abandoning eMail altogether. I know that the squeaky stat gets the headlines, but hopefully people have some perspective of what's really being described by the percentage.
eMail is very useful, for some indispensable. Spam is a problem (OK, it's a huge problem), but lest try to keep things in perspective. If you run across users in panic-mode, calm them down a bit.
Nobody mentionned the magnificent Spamgourmet.com ?
I love this service.
You can create as many disposable email addresses as you want.
Now you can even send mail, and those little critters won't be able to spam you on these addresses.
Hint, hint !: Create a single, specific address for each address you give, and you will be able to see who sold/gave your email.
And it's free (as in beer).
Your message was here?
Perhaps, with a flood of spam,
I deleted it.
As the user of a very old email address, spam is now running at about 95% of all incomming email. I find the 33% too low to believe.
There was an unknown error in the submission.
The average percentage of spam here over the past 24 hours was 99.83%. That's an average of 92.65 spams every 5 minutes and 0.16 non-spam messages every five minutes. Internal mail is not included.
----
All of whose base are belong to the what-now?
"since virtually all From and Reply-to lines in spam are faked, for every 100 spams you receive, you send 90 or so emails to innocent bystanders that don't want your bounce message."
SPF ( http://spf.pobox.com ) addresses this. If the "innocent bystanders" have SPF rules set up for their domain and don't allow unauthenticated relaying from authorized servers, then you don't need to generate a challenge for joe jobs (stealing someone else's email address).
"The number of spams your system has sent in response to forged "From" addresses?"
I have posted this elsewhere, but it's worth repeating: this is what SPF ( http://spf.pobox.com ) is meant to address. Forged From addresses from SPF enabled domains do not need to generate challenges.
Joe job reflected spam is not limited to challenge/response systems. It also arises when some of the recipient addresses generate bounces. This requires a fix to joe jobs, not an abandonment of automated responses and bounces (both of which serve valid purposes when used as intended).
The domains that i have direct access too have a much higher percentage.
2 are rather publicly known, the other 2 are not known outside a few..
Between the 4, I would say more like 75% of email traffic is spam.
This doesn't even touch on the daily attacks on the 2 that are 'public'.
---- Booth was a patriot ----
OK - I've been thinking - and I am sure many of - of have thought on similiar lines
- You add who may send you an email.
- Anyone else is Blocked.
- But if you send an email to anyone - that person is added automatically to your "allow list".
Wait there is more:* Knocks *
When anyone emails you - you will get a "knock" but just ONCE (and never again?).
You may then email that person asking him to resend the email which was automatically deleted(or stored?)
Anyway by emailing that person you immediately allow him to your list.
I dont know I have this bad feeling there is some bad flaw here somewhere
The calculations that show in dollar terms the costs of spam remind me of The Mythical Man-Month. Is it really possible of quantify the cost of spam? is it worth doing? What is clear is that it is pretty much universally despised. Do we need economic arguments before we do something about it?
ferretous
Spam levels must get worse, and will, before anything changes. Email will become useless (if it hasn't already) before. I imagine that by that time though, we may be using another form of communication (Instant Messaging?) as our primary means instead.
I know I'm about to beat a dead horse here, but...
SMTP must evolve or be eliminated.
We cannot keep duct-taping a protocol that is clearly not suitable for the internet of today and hope for any real success.
You sound just like my manager. Just because you personally don't want to talk to someone in Australasia doesn't mean other people your server is serving mail for don't. Whole Class A's and countries is far too coarse. The internet isn't just the United States!
Backups are for wimps. Real men post their data in comments and have slashdot mirror it
internet email has become absolutely worthless now due to spam, IM programs have become the better tool for personal communications, that and private messages in forums. Restricted corporate email is the only truely usable email to use now due to their ability to filter everyone but the companies own domain and those in a list.
I get about 6 emails a day at my personal (not work) email address. I get about 25-30 SPAM's per day at the same address.
That's about 5/6's - more than double this 1/3 figure.
I would probably hang myself if it wasn't for Mozilla's excellent SPAM filter.
I'd rather be a conservative nutjob than a liberal with no nuts and no job.
Total emails trained: Spam: 1811 Ham: 3728
When I installed TMDA, I watched the "pending" folder for 6 months, to ensure that everybody confirmed their mail. And they do.
That makes it sound like it's not practical for someone who gets a lot of mail.
I have a number of public email addresses for various roles. I get about 500 spams/viruses/garbage challenges/etc. per day. If I were to install a challenge response system, presumably I'd want to keep my "pending" box for a day or two before I deleted it: so then every day I'd be manually checking through 500-1000 messages, looking for that needle in a haystack that corresponds to a real message that hasn't been confirmed.
I honestly don't think I could stand to do that.
What I do instead is the following: I use Spamassassin as a content filter to classify incoming mail. Anything over a certain score gets held for a few days and then deleted unseen. Things that are at an intermediate score get put in a folder for manual checking. Low scores are treated as clean.
With this system I need to manually check around 20-30 messages per day (mostly automatic response crap from systems like yours, or virus checkers), and a few spams get through. I've probably filtered real email at some time, but I've never heard of complaints from senders, and I've never noticed my intermediate scoring messages to contain anything that I'd really want to keep.
Authentication is to be expected these days. You have to confirm mailing list subscriptions. You have to be granted authorization on IM (Jabber). Why should email between individuals be any different?
Those other systems have been abused: forged subscriptions to mailing lists used to be a common way for kiddies to flood each other's mailboxes.
On the other hand, it's far more likely that an email "from" me to you is faked than real, so why do you offload the filtering burden from your own system to mine? You're adding to my pollution because you're too inconsiderate to deal with your incoming mail yourself.
When I get a challenge from a system like yours, usually I don't see it (since your systems are fooled so often, my Bayesian filter is trained to treat your challenges as spam). If I do see it, I generally only send the confirmation if I really, really want to contact the person. If I were thinking about buying something from you, your "kiss my ass" message would likely make me change my mind.
SPF will probably be helpful eventually, but it's going to take a while, and it's going to cause a lot of trouble for people like me.
I have email addresses on about 5 different domains, registered with 4 different registrars. One domain does SPF; two are under my control, but I don't see anything on the registrar's pages about SPF, and the last two are academic domains, which means whether they do it or not depends on who is handling DNS this week.
So I'll have trouble using 4 of my domains for sending, because they're not SPF registered. I'll have trouble setting up SPF on the two domains I control, and who knows when the academic domains will get it.
I'll also have trouble because my academic addresses forward mail offsite. My reading of the SPF information says that forwarding is not supported. So I'll have to change that.
I'll have trouble setting up SPF, because the SPF page is poorly written, and lots of owners of small domains like me won't know how to go through their "wizard" to figure out the best setup. (For example, it asks me "Do you want to just approve any host whose name ends in my.domain?" What's that supposed to mean?)
In the meantime, I get dozens to hundreds of bounces coming to my mailboxes.
120 good
8 virus
==>95% spam...
Oh well, what the hell...
Forwarding: it's not so much that it's not supported, as that it has to work differently.
For most domains, you should run all your email through an SMTP server. You would just set SPF for that mail server. If the same mail server is used for both sending and receiving mail, you can just set it to the MX record.
So no ISP should be in the "filter" or "block" anything business.
Spammers are not important, people and companies that pay spammers to spam are.
The only final answer to "spam" is to go after the source of the money that pays for it. And I mean the direct source - the place spam tries to get you to go and spend money at. Not the idiots that buy from them, for they also are not important. Go after those that hire spammers.
In the interim use: "fetchmail -> qmail(with pop3 & smtpd)+qmailqueue + qmail-scanner(st/hcc) -> clamav -> spamassassin w/(razor, pyzor,dcc) -> tmda, and kmail with tmda-ofmipd, on Mandrake 9.2", or the equivalent.
TMDA is great, but you need to drop about 66% of the spam prior to the C/R system in order not to create more spam then you started with - in other words be a good net citizen.
I've been using this set up for over a year with no false positives and no (and I mean NO) spam.
I set up a default form on my website for anyone that could not reach me by email (in case of a problem with the filter/block/drop/TMDA method). It has never been used by anyone, so at this point if they can't reach me, they have nothing to say that I want to hear anyway.
Get ISPs out of the role of Big Brother. The only thing that can come of that is disaster.
NewToNix
I'd say that at least 30% of the physical mail I receive on a typical day is junk mail, which is just the real world version of spam. On some days, it's a lot more than 30% junk.
An interesting point about physical junk mail, by the way, is that it costs money to produce and it costs money to send. And yet, continue to get the same crap day after day. There are a lot of people out there who think that the key to stopping spam is going to be charging the sender for sending mail. But real world experience shows us that it just ain't so... physical mail costs a lot more to produce and send than anyone has proposed charging for e-mail, and we still get plenty of junk mail.
I think the real key is going to be something akin to the national do-not-call list. In fact, it could be an extension of it. You could register an address (street or e-mail) and say that you choose not to receive unsolicited commercial mail. That, combined with better regulations requring accurate sender information, could really help.
That list only tells us the distribution of spamming individuals throughout countries, it says nothing about how much spam any of them send. One or two of these spammers may send 99.99% of all the spam sent in the world, and they may be located in Bora Bora for all we know. Inferring that the ratio of spam sent from any one country is the same as the ratio of spammers living there is logically incorrect.
Good site - thanks for the work!
:-)
However, my nitpicking disorder prevents me from overlooking your frequent use of "asterix" whenever you supposedly mean "asterisk".
Or are spammers really putting comic book characters bewteen the letters now?
Baumi
...is 40 million spammers devising schemes to beat your spamfiltering 'tests', now you've published them on Slashdot for all the world to see.
Thanks a lot mate!
---- scrm
There are lots of problems with the objectivity and quality of this self-described "landmark study". First, it is not an independent scientific or academic study but a commercially prepared document that is for sale. In itself that does not invalidate its conclusions, but add the fact that IDC does consulting work in this area and the objectivity of the study becomes a factor. There are clearly financial benefits for IDC in the paper's conclusions. I have no evidence that it is not completely objective, but there are definately some clues about its poor technical quality.
I have not read the study since I am not about to pay good money for the privilege, but from IDC's own press release the most obvious problem is that the writers don't seem to be able to differentiate between spam "sent" and spam "received". This is basic stuff. The writers state that 32% of email sent in North America is spam, but what they really mean is 32% of email received in North America is spam. [They only know "received" spam because all their data came from questionaires sent to companies that receive spam]. That is a big difference, particularly since a sizable percentage of spam sent is either filtered out before receipt or is addressed to non-working email addresses. I suspect that the amount of spam sent is many times greater than the amount of spam received and is significanly greater than indicated in this so-called study.
Their incorrect assumption that email received in North America came from North America is another indication of the poor quality of this self-described "landmark study". I am no spam expert, but even I know my email can come from anywhere. This is a basic error that does not speak well for the competency of the report's writers.
My personal mail account stats for the preceding 3 days:
970 total messages
6 of which real emails
964 spam.
My SpamAssassin proxy needs a tweak or an upgrade, it only correctly tagged 750 of the spams.
I'm a good-natured sort, but this pisses me off. If I ever meet a spammer I'll fucking kill his ass dead with a 2x4.
Funny ...
I just run Perl's regexp syntax validator on the headline, and if any part of it goes through, it's spam.
Because Slashdot wasn't when I submitted my site as a newsworthy article some time ago.
In a nutshell, my program, CF13 uses a number of simple, non-mathematic, pattern-matching tests to make it virtually impossible to get English language spam past it. These tests do not require the overhead associated with Bayesian Filtering and its ilk.
I think the key feature to it is to treat as spam all email from unapproved senders that contain more than 'spaces' and alphabetic charaters.
This simple but powerful feature makes it IMPOSSIBLE to conveniently spell email addresses, URLs, postal addresses, prices, and phone numbers. These items are neccessary for e-commerce to take place. Without them, e-commerce is IMPOSSIBLE or at least extremely difficult to conduct. It also treats as spam email containing 'non-ASCII' characters. I have gotten quite a few such emails at another email address I use infrequently--all spam (sales pitches in foreign languages).
As an added benefit, CF13 makes it 100% IMPOSSIBLE to accidentally run malware sent by email provided a particular registry setting has not been compromised. It does this by treating all email and file attachments as 'text files' that can be scanned for malware and handeled safely. Thus, one's PC CANNOT be compromised by a malicious malware HTML webpage or worm/virus/trojan email file attachment.
It also detects 'mailbombing' and handles it a manner that makes it easy to clean up afterwards.
It is probably best to fight spam at the SMTP server level but I have heard it is best to fight spam at the end user level. Both approaches have their advantages and disadvantages so this issue appears to me to be a toss-up for the time being....
I have a sad, sad, boring life.
Why should I argue rationally with someone being irrational? I'll just mock them instead.
End of problem. Payment made. Spam sent. Life goes on..