Gmail Spam Filter Testing

An anonymous reader writes "What can you do with 1000MB of e-mail space on your Gmail account? One guy, by the name of Aaron Pratt ( prattboy@gmail.com ), has decided to test the spam filters of Google's Gmail service by having his Gmail account blasted with every kind of spam imaginable. He is testing to see how well Gmail's spam filters can sort out the spam from legitamate email (yes, he does get personal emails from people). As of May 25th, he was at about 30% of his Gmail account's 1GB capacity. You can track his progress on his website, http://gmail.prattboy.net (Google cache of this site: cache: gmail.prattboy.net). Here is also an article talking about Aaron's efforts from webpronews.com"

Comments

first spam?

[miketang16]

psh.. i've done this to my friends before.. they didn't need to make a website to ask for it...

The Filter is great!

[umrgregg]

Apparently, Google's spam filter even filters messages that aren't there. From the website:

3778 messages were received, totaling 213 MB.
3917 were spam, and Gmail correctly identified 41.9% of these messages.

Fantastic

Re:The Filter is great!

[Anonymous Coward]

No, thats just a classic threaded code bug:

They just forgot the mutex surrounding the two snprintfs... so this user probably got 139 messages in the time it takes to execute snprintf, all spam.

Which is.... about right.

Re:The Filter is great!

[aismail3]

When I add up the figures from May 13 to 19, I get that 4869 messages were received. 4717 of those were spam, and 1820 were marked, so Gmail's success rate was 38.6%.

One of the best things Google/GMail could do

[Anonymous Coward]

Is use the GMail data to operate a checksum blacklist. Obviously, if thousands (or millions) of their users are getting the exact same email, it's probably spam.

Re:One of the best things Google/GMail could do

[lockefire]

Actually, I get a whole lot of emails with the random words and nothing else. I haven't quite caught on to the advertising strategy in that.

Re:One of the best things Google/GMail could do

[Cruciform]

I've been getting them as well.
The only reason I could think of someone sending those around is to bog up Bayesian filters with random crap, possibly lowering their effectiveness.

Any spammmers/spam-experts feel like enlightening us? :)

Re:One of the best things Google/GMail could do

[Halo1]

Most of the time, these messages contain both a text/plain section with only random words, and then a text/html part with the real payload. If you use mutt or so, you most likely only see the text/plain stuff. Another trick is using just a text/html section with random text, but also with an image that contains the real payload.

Re:One of the best things Google/GMail could do

[jefe7777]

>> You think they bother?

heh heh...abdolutely.

100 known good addresses are worth 10,000 "who the fuck knows" addressess.

>>It's cheaper to just send mail to everyone

no it's not.

let's pretend you are a spammer, and you want to send out spam.

If you target 1 billion questionable addresses, each time a client has a new campaign, then that's 1 billion pieces you have to deliver. every time.

what if you have 1000 clients? that's 1000 billion deliveries.

do you see where this is going? if you don't KNOW WHAT A VALID EMAIL ADDRESS IS, YOU HAVE TO GUESS.

but what if the first time you send out just a "test" to those billion addresses, and then subtract the one's that bounce.

You are left with 50,000 known good addresses.

that's gold. You now have 1/20th of the load,and you are now serving your clients quicker, a helluva lot less load. you are only using an open relay for 1/20th of the time.

overall a smaller footprint by 1/20th.

you tell me. does it make sense to blindly blast out email?

Re:One of the best things Google/GMail could do

[letxa2000]

Spammer is trying to do two things: 1. break any Bayesian filter used on that mail server/inbox. Adding noise to the filter will allow more mail through as "questionable". This might still be tagged as spam, but not as readily as it would be without the added noise

Except that won't work, as anyone that understands Bayesian filtering will tell you. In the case of every message with "random words" I've checked recently, the random words actually increased the spam score of that message. Why? Because it seems the random words aren't so random and either the same spammer is using the same "random words" over and over or various spammers are using sets of the same words. Over time most of the "random words" they use actually become great indicators of spam since my real email doesn't typically contain the random words they use.

In one recent analysis, 10 random words were inserted by the spammer. He got lucky and 1 of those words actually had a very low score for my Bayesian corpus. Unfortunately (for him), the other 9 words had scores of 99.99%! His use of random words literally nuked any possibility of him getting through my filter.

Anyway, random words will not help spammers get through Bayesian filters. But it seems that many people (both spammers and non-spammers) think it will. But, hey, that's good for me: as long as "random words" is seen by spammers as a viable solution to Bayesian filters, my Bayesian filter will continue to work and will not have to deal with any innovative way to get around the filter (if any exists).

He gave out his e-mail address...

[Anonymous Coward]

... to the entire Slashdot community! Now he's going to be flooded with all sorts of spam and shit. LOL!

Oh... right. :)

whining?

[Gothmolly]

What's Google going to do to protect its users from mail bombs?

Now you're complaining that your free, 1GB-limit, access-from-anywhere email service could be mailbombed? Live with it. If Google "decides" anything more about our emails, we put on our tinfoil hats and scream. If we broadcast a bogus email address, obtained from gmail for clearly sinister purposes, and it gets mailbombed, we whine that Google doesn't "protect" us. Whats the story, or are we all just schizophrenic?

Don't want that "vulnerability"? Don't use Gmail!

Re:whining?

[supersnail]

I don't think its about protection just practicality. Google offers a SPAM filter the littel pratt tested it and found it wanting.

I think its more of a problem for Google than the end users. The whole Gmail "get a gigiabyte of memeory free" business model is predicated on most people using only a small fraction of that Gigibayte but felling good about the capacity being there. If I open up a gmail account, get p*ss*d of with the spam and go elsewhere without closing the account the 1G will fill up with spam in a couple of months, Google will end up storing terabytes of spam for cutomers who no longer use the service.

Re:whining?

[Pharmboy]

Now you're complaining...

That is his JOB, to point out shortcomings of the system. He is a tester, and he is doing it for FREE. Google doesn't want testers who get 3 emails a day, they want people to test the living shit out of the service and point out what is wrong with it. Everyone knows Google will try to fix all the bugs, so all the press, good or bad, is still good press.

If Google barfs when handling 999 messages in 4 minutes during testing, image when several million people have gmail accounts. Fortunately, now Google has an even to look at to see what the problem is. When you are trying to harden a system, YOU MUST BREAK IT OVER AND OVER AGAIN, to see where it is weak. This is what is happening.

My impression is that the tech's at Google are spending a significant amount of time saying "oh shit, never thought of that, cool." which is the ENTIRE REASON FOR TESTING. They can't think of every situation by themselves. This is also the entire concept behind "open software is more secure". Google's gmail is going to have bugs at this stage and lots of them, period. Google knows this, hell, everyone knows this (this is why its in testing, and not open to the public yet, duh)

It not whinning, its stating the facts, which Goggle obviously WANTS him to gather, as a TESTER. Seems to me that he is going beyond the call of duty to test their servers, since he is spending a fair amount of his own time.

Re:whining?

[Beryllium Sphere(tm)]

>The whole Gmail "get a gigiabyte of memeory free" business model is predicated on most people using only a small fraction of that Gigibayte

Why?

Google uses commodity IDE drives. Those retail for about fifty cents a gigabyte. Google's not paying retail.

I read a quote from a Googleperson that by the time the drive is installed in a system, powered, cooled, backed up and administered Google is paying two dollars for a gigabyte.

Good point about the problem of abandoned accounts, which won't bring Google any ad revenue. Wouldn't be surprised if they start euthanizing inactive accounts.

Not a fair test

[SWroclawski]

He's not counting all the mail that Google is rejecting and not even being allowed in for further classification.

I'll help

[L. VeGas]

Let's all send him an email and ask him how it's working out.

Re:gmail still beta

[waddgodd]

>isn't gmail still in 'beta' stages? if so, isn't a review of
>spam filtering techniques a little premature?

What part of Beta TEST escapes you here?

Re:Not that impressive

[Apiakun]

Don't forget that this is google's first foray into mail software, and it is still in beta. I have so far gotten very little spam in my gmail inbox.

Is this the AventureMail guy?

[magefile]

The guy who got booted off AventureMail (2GB free) for trying to test their spam filters? The story is on Kuro5hin, if anyone wants to see it.

My own gmail testing

[Twid]

I did some testing of my own. I forwarded a ton of spam from my personal account to my gmail account, just to see what would get through and what would be filtered. For me, gmail was really effective, but strangely, one Nigerian e-mail scam mail didn't get tagged.

It was from " Mr Jubril Udeh Manager of Credit and Accounts Department of North Atlantic Securities Sarls Lome-Togo Republic."

Now, the funny part is not that the mail made it through, but that google also decided to show me contextual ad's on that account. Currently, the ads are:
- Payroll Cards a Poor Substitute for Checking Account
- Tips for Tackling Check Fraud
- Sophos hoax description: Ethiopian airline letter
- FAP non-US Investment FAQs

In the past the mail has also shown me ads on how to open an off-shore bank account. I'm glad google is willing to help me with the $10.5 million dollars that I'm about to receive! :)

gmail spelling

[Anonymous Coward]

>legitamate

How about having Slashdot editors/Hemos test the gmail spell checker too?

0% Spam

[yuri]

Spam is unsolicited, so google should filter none of his mail.

This guy solicited it.

Lack of updates?

[Xiadix]

Did anybody else notice that his site hasn't been updated in almost a month (May 25)? Seems his project is no longer working. I wonder if Google booted him.

KevG

Cache?

[Freon115]

Do you really expect the Google servers to go down because of /.? ;)

Re:Cache?

[leo_llew]

Obviously not, they provided a link to the GOOGLE Cache ;)

Re:How to never get spam

[mumblestheclown]

Hi! And welcome to the Internet! We're glad to have you aboard.

Just to get you started, I'll give you a quick hint: virtually every internet discussion on spam includes some high and mighty moron that claims that by not giving out his email address, he never gets spam.

The problem is, that for every one of those, there are plenty more who follow the same precautions and yet get plenty of spam to those accounts for a variety of reasons. Clearly, your soution is not the answer to "how to never get spam."

A good rule for using the internet is to read a few discussions before you post. This way, you will be less likely to post something that makes you look naive. So sit back, relax, and enjoy a steaming hot cup of STFU while you read and learn!

Wow

[EaterOfDog]

His wang is going to be huge!

More focus on false positives.

[ron_ivi]

Reviews of spam filters always seem to focus on how much stuff they block.

The consequenses of blocking a non-spam email are so much worse (parent not hearing from kid. the customer that would have saved your startup.) than a spam getting in, I wish the spam filter reviews would focus on those.

New spin on the "word salad" strategy

[Scott Richter]

Except that won't work, as anyone that understands Bayesian filtering will tell you. In the case of every message with "random words" I've checked recently, the random words actually increased the spam score of that message. Why? Because it seems the random words aren't so random and either the same spammer is using the same "random words" over and over or various spammers are using sets of the same words. Over time most of the "random words" they use actually become great indicators of spam since my real email doesn't typically contain the random words they use.

Right, and my Thunderbird Bayesian filter catches all of those word salad approaches. But they've come up with a new one - what I call the "encyclopedia attack."

What they do is copy an encyclopedia entry and put it at the bottom of their spam. The thing is usually a few paragraphs long, so that textually it dominates the message. The subjects are fairly random, and are occasionally educational ;)

The problem is that the text of this doesn't trip the "too many strange words" flag that's used for word salads. My Thunderbird filter is really having trouble with these. Anyone else having trouble with these spams?