Slashdot Mirror
A Look At Google's Email Spam Prevention
CNet has a story about the security measures Google employs to protect their email systems and fight the never-ending war on spam. Their Postini team, acquired two years ago, has a variety of monitoring tools and automated response systems to find and block undesirable messages. Quoting:
"The system scores each message on numerous combinations of criteria, assigning a weight to each and then comparing the score to those in a database of several hundred thousand message types that have been flagged as good or bad from Postini honey pots and customer spam reports. ... To block fresh spam attacks not covered by existing heuristic technologies and viruses not covered by existing signature databases Postini relies on proprietary Zero-Hour technology to identify new outbreaks that show up in the traffic patterns and quarantine them for later rescanning. Customers can also create and build out their own white lists of message senders they trust and blacklist others they don't trust. It takes an average of 150 milliseconds for a message to be scanned by the antivirus engines that Postini licenses from McAfee and Authentium.
I now get a couple of shed loads less spam. I used to check the apam directory for false positives. Don't bother doing that either.
:-)
Go gmail
If I had an Ass, I'd call it Fanny Bottom, then I could slap my Ass; Fanny Bottom, on the Arse.
My previous ISP switched me over to Postini with no advance notice (we got a cheery note from marketing after the deed was done). Blocked half the spam and half the ham. They told us how to disable the filtering "features" but it turned out that all the filtering could not be turned off.
I'm not with that ISP any more.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
and now, where I am supposed to to get my weekly viagra supplies?
...has a quick look and goes back to catching up with news on the MailScanner mailing list.
AT&ROFLMAO
but what I really want to tell you is that I've inherited a great deal of money and I need someone to help me transfer it to the US. I live in Nigeria. You all seem to be great gentleman, so I will pay appropiately.
Contact me.
part of gmails phishing filter seems to do this
if(hyperlink in email ends in .exe)
{
isphishing = true.
}
Even if this is an email from someone in your whitelist and is merely quoting text from your own message you sent them. .exe in it to be marked this way :(
And there seems to be NO way to prevent a message with
DRM-free indie games for the PC and Mac: Positech Games
They can filter out the obvious spam mail, but some spammers are so clever and so well hung - because they've taken some DrMaxMan to acquire an enlarged sexual wand with which you can perform better and be bigger for f.r.e.e - that they can actually embed their spam offers inside real messages in such a way as to be completely undetectable by filters.
We go to cinema. Join us.
Waiting for a telephone call.
Club you asked about.
Check this song.
---
Is it worth a try?
Stop ruining yourself.
From Gaston Woodard to me,
Buy unexpensive, best price pharmaceutical products online.
---
And here's my favorite, Aloha by Josie:
loss enable smug filth!
joy smug. stable smug egress smug? telly comity argue jocose?
entail haggle. abbess sleigh dalle filler. loss quid egress.
ennui smug put. scrap stable haggle. focal terse.
furore pirn spur uptake? tower alert dagger tower! pinto abbess.
tother diver tower solar! jocose solar lower juicy. proem common pant.
enable today whack juicy! winy bane juicy. jocose sleigh drill uptake.
hern haggle khan abbess? enable common pant egress! sinewy ennui.
focal robin tower potto. paid jocose legal hunch. parish whack loss paid?
tother brooch tower lower! metope tendon. scrap boh.
Comment removed based on user account deletion
So, the product is still great. Tech support has gone downhill though. Anyone who has tried to deal with Google tech support for anything will know how it feels..
Never email donotemail@WeAreSpammers.com
This is great for business mail too... small company where I work was literally BURIED with spam until we moved to gmail. Since their mail addresses were "in the open" on our website for years, some of them get 200+ spams a day. Now, if 1 in 1000 passes, it's a bad day. Also, in my private inbox, I had an VERY old mail address still redirected to gmail address... turned out that was the source of 1/2 spams (100+ / day). But those were filtered too without problem. So far so good... not a single false detection for ham. Nothing but praise so far. Disclaimer: I do not work for gmail. I am the genuine satisfied customer with smile on my face, from "after" picture, as seen on TV!
Its brilliant. End of story.
At least 75% of my spam is addressed as though it was sent from *my* gmail account. Of course, it's easy to set up a filter to reject all such spam, but then I lose the ability to send reminder messages to myself. Seems like it would be extraordinarily simple for google to outright reject messages that claim to be sent from their servers that in fact were not. I sure wish they would!
It takes an average of 150 milliseconds for a message to be scanned by the antivirus engines that Postini licenses from McAfee
The filtering works for me. But I know people where the filter catches 400 spams a day and 5 hams, making it totally useless.
All I know is gmail's spam filter is amazing, mabye 1 gets through a month but thats it.
LAGOS, NIGERIA.
ATTENTION: SLASHDOT USER
DEAR SIR,
CONFIDENTIAL BUSINESS PROPOSAL
HAVING CONSULTED WITH MY COLLEAGUES AND BASED ON THE INFORMATION GATHERED FROM THE NIGERIAN CHAMBERS OF COMMERCE AND INDUSTRY, I HAVE THE PRIVILEGE TO REQUEST FOR YOUR ASSISTANCE TO TRANSFER THE SUM OF $47,500,000.00 (FORTY SEVEN MILLION, FIVE HUNDRED THOUSAND UNITED STATES DOLLARS) INTO YOUR ACCOUNTS. THE ABOVE SUM RESULTED FROM AN OVER-INVOICED CONTRACT, EXECUTED COMMISSIONED AND PAID FOR ABOUT FIVE YEARS (5) AGO BY A FOREIGN CONTRACTOR. THIS ACTION WAS HOWEVER INTENTIONAL AND SINCE THEN THE FUND HAS BEEN IN A SUSPENSE ACCOUNT AT THE CENTRAL BANK OF NIGERIA APEX BANK.
WE ARE NOW READY TO TRANSFER THE FUND OVERSEAS AND THAT IS WHERE YOU COME IN. IT IS IMPORTANT TO INFORM YOU THAT AS CIVIL SERVANTS, WE ARE FORBIDDEN TO OPERATE A FOREIGN ACCOUNT; THAT IS WHY WE REQUIRE YOUR ASSISTANCE. THE TOTAL SUM WILL BE SHARED AS FOLLOWS: 70% FOR US, 25% FOR YOU AND 5% FOR LOCAL AND INTERNATIONAL EXPENSES INCIDENT TO THE TRANSFER.
THE TRANSFER IS RISK FREE ON BOTH SIDES. I AM AN ACCOUNTANT WITH THE NIGERIAN NATIONAL PETROLEUM CORPORATION (NNPC). IF YOU FIND THIS PROPOSAL ACCEPTABLE, WE SHALL REQUIRE THE FOLLOWING DOCUMENTS:
(A) YOUR BANKER'S NAME, TELEPHONE, ACCOUNT AND FAX NUMBERS.
(B) YOUR PRIVATE TELEPHONE AND FAX NUMBERS -- FOR CONFIDENTIALITY AND EASY COMMUNICATION.
(C) YOUR LETTER-HEADED PAPER STAMPED AND SIGNED.
ALTERNATIVELY WE WILL FURNISH YOU WITH THE TEXT OF WHAT TO TYPE INTO YOUR LETTER-HEADED PAPER, ALONG WITH A BREAKDOWN EXPLAINING, COMPREHENSIVELY WHAT WE REQUIRE OF YOU. THE BUSINESS WILL TAKE US THIRTY (30) WORKING DAYS TO ACCOMPLISH.
PLEASE REPLY URGENTLY.
BEST REGARDS
So by using gmail, am I indirectly making money for McAfee?
...because it's actually not working - Gmail spam filter recently became very ineffective - i have to classify about 5-10 Viagra spams daily. (Google, have you heard of it? geez!) then it occurred to me that a while ago Gmail captcha was cracked, so I imagine spammers send themselves hundreds of spams only to classify them as "non-spam". - as a consequence, spams are now slipping through the crowd-sourced filter because the crowd is infiltrated. c'mon google this can't possibly that hard to fix!
I started getting much more unfiltered spam. Lots of it looks like this (a real example) : Subject : acceptant accelerometer abysmal abusive accession accolade So, no website, no valid return address. Just random words. I'm wondering if either there's a bug in the spam generator (I get others that start with a line of similarly random words, but then continue for a page or so and are followed by an ad), or if they're trying to confuse spam filters.
There's another variant that looks more like english text with a number of errors in spelling so only a few of the words are real.
In my humble and largely anecdotal experience, Postini works well. We send out e-mail that can often be flagged as SPAM when we perform penetration testing, and Postini seems to be the toughest to get around. We see in-house devices such as IronMain, and outsourced services such as MXLogic and FrontBridge/hosted Exchange, but Postini seems to do the best at stopping illegitimate messages. The company I work for uses this it as well, and logging into my Postini inbox I see a lot of spam but no false positives. I think it's a pretty good solution if you don't want to handle SPAM in-house.
Don't know about anyone else... I've been with gmail since it really *was* a beta, and it has been pretty good about not letting spam through. Past couple of months though and I have been getting three to four a day through. Are the spammers getting better or is the filter getting worse?
I used to have 20,000+ in my spam folder every day for years. Recently it dropped to the low 400's.
But because there's much less spam, I actually check the spam folder quite often to see if there are false positives, and I almost always find a few. Makes me wonder how much mail I missed all this time?
I use SSL/TLS encryption on my SMTP traffic using STARTTLS. The reason is a long story but it has to do with my work in infosec. So I run my own mail server with STARTTLS configured. After having the same problem with their anal spam blocks I too had to set up a special mail route for anything to gmail to go via my ISPs mail relay.
"...computerized systems monitor 3 billion messages per day"
/(60*60*24) = 5200 messages being processed at any given time.
"It takes an average of 150 milliseconds for a message to be scanned by the antivirus engines..."
A little arithmetic: 3E9 * 0.150
I take it that there must be more than 5200 processors at work -- on average.
When gmail was "invitation only" I opened an email account. I never used it for anything. Never gave it out, never signed up for anything with it, never sent a single email.
I've logged in to it four times, and I deleted something like 2000 spam messages.
I'll continue to not use it, thanks.
Now apply this technology to Google Groups.
Yeah, I know it's usenet, but they could apply it to their web interface (see comp.lang.c++ for a sample of what it has to deal with).
Fascism starts when the efficiency of the government becomes more important than the rights of the people.
What I find telling is how my SPAM volume rises and falls according to the American holidays. Whenever the Yanks have a holiday, SPAM drops to a trickle.
That to me is a clear indication that most SPAM originates in the US even though it mostly gets relayed through Asian proxies.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
I counter your anecdote with my anecdote! No, seriously - not to be an a$$ or anything, but I haven't gotten a single spam in GMail in over two years. There is none in the inbox, and none in the spam folder (label) either. I'm not sure why you are getting them, but it is clearly not everyone who is so afflicted (thankfully!). I'm not sure if it has something to do with accounts on different back end systems or what, but mine hasn't gotten any spam in one heck of a long time.
Sorry but my laptop can do it faster when using something like CRM114 or DSPAM.
When ever I see those wild claims how good and accurate a commercial service or filter is, then I get reminded on the excellent text written in 2005 by Jonathan A. Zdziarski called Justifying Statistical Filtering.
Postini might be good but I am not letting them decide what spam is and what not. Users have their own opinion and something so static as Postini can not adapt fast/good enough to my needs. And the same goes for the other services like MXLogic, SpamSpy, MessageLabs, Barracuda, IronPort and all the others out there.
And why paying money when I can have better for free?
because I get so much more spam than I do to my 4, older ISP accounts. Almost from day 1 (about 4 years ago) on my gmail account, the spammers had my address figured out, and it is not what I would consider a very obvious one with some numbers mixed in to make a non-dictionary/non-normal name.
I run a moderate sized community and last month alone we banned over 50 throw away gmail spammerbots. That might seem to be a small number, but we're currently blocking .cn and .ru, and most other free email providers. Gmail addresses account for over 95% of our spam problem.
Be nice if they did something about that.
Spam and scams originating from Gmail has been so bad lately that several clients of mine have actually requested that I block gmail entirely. I have been tempted to do so with my home account as well since its rendered craigslist all but unusable. When do they plan to address that...but then what could they really do??
I have had two gmail accounts for a couple years now. One of them has my name on it (in the form of: "firstname.lastname@gmail.com") and the other is a nick (not the same as my /. one) that I often use in forums/games. Curiosuly enough, neither of these accounts gets any spam at all. And by this I don't mean that the spam filters are effective because there is no to be filtered. I can understand that my name based account doesn't get spam, after all I rarely give it out to anyone except people I know in person and very important sites (mostly web-stores that require my full name anyways) that I trust. However, I use my nick based email on nearly all forums and sites that require an email address during registration and despite that, I only get mail from those sites. No unwanted viagra adds or anything. Now I know from earlier comments and stories such as this that spam is a huge problem to many people. So am I just incredibly lucky? I honestly don't know.
The only thing I have consciously done to avoid being spammed is that I have never typed my adress directly to any forum post/site. In fact, most forums allow you to hide the email address, and even the sites that option of sending email to other users usually require registration to see the adress and have methods such as CAPTCHAS in place to prevent bots from getting in to harvest the addresses. If somebody asks for my email I'll just send it to them via private message or similar method instead of leaving it "in public view".
Like I said, I have no idea if I'm just lucky or something, but spam has never been a problem for me.
"It is the business of the future to be dangerous" -Alfred North Whitehead
Damn it, you owe me a new keyboard!
XD XD XD Hehehehehehehehehee...
Keep in mind folks, Gmail's Spam filtering is seperate from Postini.
From the article:
"Google's Gmail antispam efforts are separate from those of Postini, which Google acquired two years ago, although it follows similar computerized operations and the teams have started to integrate the processes."
I've had email at an ISP that uses Postini, and I have email at Gmail. IMHO, Gmail > Postini.
i never use the account except as a backup, and have never used it as a throwaway address (i use the awesome spamgourmet for that) but it has a full page or two of spam whenever i visit it. my daily mail goes through http://www.junkemailfilter.com/spam/ THAT is the bomb, GMail can't touch it.
look sig is kool
Contrary to what another poster who replied to you has to say, I agree that Postini has gone down hill within the last month or so. The software is missing virii attached as normal attachments to emails. Virii that the Symantec anti-virus on our in house Exchange server is catching. I never in a million years thought I would see the day when Symantec would be doing a better job than Google.
My folder have an average of 3,800. What about yours? http://i44.tinypic.com/2j4vd76.jpg
I find it very strange that my Gmail account received so much spam long before I ever started actively using it. It's not like me e-mail address is made up of one or two words. I cannot for the life of me understand how anyone would possibly guess my e-mail address (two letters plus an uncommon word). I'm guessing someone got a hold of their user list. Anyhow, their spam filter is fairly accurate.
signature pending slashdot approval
...because it's actually not working - Gmail spam filter recently became very ineffective - i have to classify about 5-10 Viagra spams daily. (Google, have you heard of it? geez!)
I wouldn't call it "totally ineffective" but it's not totally effective either. For months I was getting "I saw your profile on (random, probably made up site) and would like to meet you" type spam several times a day. These aren't coming though any more but it looks like it's because they're not being sent, not that the spam filter has finally learned to classify them.
I get about 100 spams a day in my gmail account. All of them are obvious, bad spelling, oddly capitalised letters, promising wondrous things. There are maybe 10 of those 100 that the filter doesn't catch. On the odd day it will miss maybe 50. I do think the filter has become less effective over the last few months.
You probably misspelled your mail server's "user agent" string as postfux ;-)
Use one email account to respond to emails, never subscribe to online magazines with this. Change the first email account ever three months ..
Gotta say, I've been using Gmail for years. I still get around 1500 spam messages a month caught by Gmail. I *maybe* get 1 spam message through a month and I have never had a false positive. I really can't complain :)
I suspect they are useing a rolling hash to recognise and chomp up a message in several blocks. They are then able to tag a block as spam, especially if people 'vote it down as spam'. That is why you still see a lot of spam if you check your gmail all the time, but none if you only check it now and then (other people allready voted it away). This also explains how their threading works, and how it is that they can offer so much space (in threads they only have to save the headers and the changes in the posts).
They seem to do a great job catching spam. I curious as to why they still bother "sending" the majority of the reported spam? After 5000 people report a spam email as such, why does the spam mail still need to be sent to my spam folder?
Learn About Outsourcing. http://www.pioutsource.com
Hell, they might not even be in the right forest, for that matter. Google should know well enough that spam is an economic problem, not a software one. They can write all the fancy filters they want, they will never win the war that way.
They have the resources, they should fight the war the right way - by going after the people who sponsor spam. They are electronically reading our gmail email, they can see the headers. They know where the spam comes from, and when. They know what domains are being spamvertised, and they can determine who owns those domains. They should be going after the registrars, the ISPs, and the owners of the mail relays. Only when spam becomes too expensive to be a viable business model for the spammers will it go away. Until then we will only continue to play spam filter whack-a-mole.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
So you need to encrypt the first leg but not the others?
Nerd rage is the funniest rage.
Gmail's biggest advantage is sheer number of users, not the actual technology. Their filtering would be pretty effective if all they did was learn from their users hitting the spam button. If you get a spam into your inbox, chances are that hundreds or maybe thousands of other gmail users read that message before you and marked it as spam. After a certain number of these manual filtering events, Gmail can simply blow it out of all other mailboxes.
Postini's service is 'nice' but not great. SpamAssassin still blows it away. Why on earth do I receive 'viagra' sales emails from my co-workers? Oh right because user@domain.com can email user@domain.com even with SPF.
...is misleading. New summary:
Bayesian filtering.
NEXT PLEASE.
You feel sleepy. Close your eyes. The opinions stated above are yours. You cannot imagine why you ever felt otherwise.
...because it's actually not working - Gmail spam filter recently became very ineffective - i have to classify about 5-10 Viagra spams daily. (Google, have you heard of it? geez!) then it occurred to me that a while ago Gmail captcha was cracked, so I imagine spammers send themselves hundreds of spams only to classify them as "non-spam". - as a consequence, spams are now slipping through the crowd-sourced filter because the crowd is infiltrated. c'mon google this can't possibly that hard to fix!
Actually I think they already have. I noticed the same thing only I was receiving a far greater volume. I think I suddenly went to a couple of hundred emails per day, some getting as far as my spam folder, some getting in to my inbox. Now just this weekend I noticed that this has now ceased and the number in my spam folder is working its way back down as they are deleted.
I dont read
...are better off doing their own solution using a combination of sendmail, mimedefang, spamassassin, and greylisting. If you are big enough to 'need' postini, you likely have a staff that can do it better themselves using open tools and tuning that solution to your particular environment. But nowadays, nobody wants to hire competent staff, it seems.