Filter-foiling Gibberish Becoming A Spam Staple

hcg50a writes "Wired has a story about the random words which have recently been appearing in spam. Antispam experts agreed that this isn't a brand-new technique, but said the addition of potentially filter-foiling gibberish is rapidly becoming a common component of spam."

Comments

Well...

[i_am_syco]

A lot of the time that "random gibberish" comes in the form of a story or something. Hell, a while ago I got a spam that contained a few exerpts from The Raven by Edgar Allen Poe. I got a laugh of that one.

Spamkiller doesn't care

[Frisky070802]

My Mcafee Spamkiller ignores the white noise, and simply nukes all the mail containing viagra, etc.

Re:Spamkiller doesn't care

[K-Man]

Let's see:

Gen3r@c v|agar@
Gener@c v|agar@
Generic v|agar@
Generic viagar@
Generic viagr@
Generic viagra

That's an edit distance of 5, pretty large, but still findable with a little approximate matching, especially if it's weighted, to recognize the similarity between @ and a, or i and |.

Most spam contains repeated phrases 40+ characters long. the mistake is to use word-counting techniques which ignore phraseology.

For instance, here are some phrases from spam, circa one year ago:

Please fill out the form below for more information
To unsubscribe
To remove your
in the Marshall Islands
Please allow 48-72 hours for removal
to this email with REMOVE in the
the Northern Ratak
the information
thousands of dollars
that you will
this list, please
this advertisement
this email in error
this message, you may email our
this transaction
of thousands of
of EnenKio and
of Eneen-Kio Atoll
of His Majesty
our mailing list
out 5,000 e-mails each for a
opportunity to make

Re:Spamkiller doesn't care

[letxa2000]

The encoding V*I*A*G*R*A would break out to the letters V I A G R and A.

V: 76.9% Spam score
I: 47.2% spam score
A: 68.8% spam score
G: 72.2% spam score
R: 72.2% spam score

On balance, if I get a message with the individual "words" of V, I, A, G, R, and A, that's going to be leaning towards spam.

That's the beauty of Bayesian. Anything the spammers do will eventually come back and bite them in the butt. Even some of the "random words" they are starting to use are getting high spam scores:

WHEREUPON: 99.9999%
NEOCONSERVATIVE: 99.9999%
LIBERAL: 74.3%
LIBERTY: 84.0%
MEGATON: 99.9999%
METHANE: 99.9999%

These are just a few of the "random words" I found in recent spams and, interestingly, the random words they are using are actually INCREASING their spam probability.

Statistically, it's a lost cause for the spammers, they just don't realize it yet.

Re:Spamkiller doesn't care

[letxa2000]

I get the same statistics as you with my SA install, most of it is given a BAYES_99 score. Unfortunately, many don't train their own filters, and this is rather effective against them.

True. Although an obvious caveat of using Bayesian to filter is that you HAVE to train it. In the anti-spam service I use (see tagline) it defaults to NOT using Bayesian. If you turn Bayesian on it specifically sends you an email reminding you that you MUST train it or things will actually get worse.

But you're right, a misused Bayesian filter might actually be worse than no Bayesian filter at all. But that's the case whether or not spammers insert random words.

There are ways to poison Bayes-filters that are better than this, and that may well be effective. If you sit down and think about it, I'm sure you can think of something too. I'm not going to write them, because it will be too easy for spammers to implement. Fortunately, spammers are stupid, and that buys us some time, but we still need more options.

Let's talk about them. We're not going to come up with anything that spammers can't come up with so I don't think we're going to make things any easier for them or give away the farm by discussing it publically.

I personally have thought about it and I'm unaware of how they could poison Bayesian statistics. I only see two approaches, theoretically. 1) Make your spam get a lower Bayesian score so it gets through. 2) Make non-spam get a higher Bayesian score so it gets caught as a false positive.

Approach #1: Short of going to the "spam of the future" predicted by Paul Graham, I don't see any way for spammers to really get a lower spam score.I've seen entire sections of the Constitution embedded in spam that still got a 98% spam score. The only way spammers are going to get a lower spam score is by doing things like using the names of my friends, using words related to topics I often discuss, etc. And that's just not possible. Like I said, they might get an occasional lucky shot but what gets through to me most probably won't get through to you. I just don't see any way for them to reliably get past a significant number of Bayesian filters.

Approach #2: Poison the Bayesian stats such that non-spam mail gets tagged as spam. I'm pretty convinced this isn't possible, either. Again, they'd have to heavily use words that are specifically non-spam for the receiver such that the spam rating for those words increases so high that it is considered spam. But if the words are heavily used in both spam (trying to poison the stats) and non-spam, it's going to float to a middle position, like the word "THE" which has a 53.2% chance of being spam (and that's only because 92% of my mail is spam so a neutral word is usually slightly over 50%). But neutral words are completely ignored by Bayesian--only the "most interesting" are considered, those that are 99% spam or 1%--THOSE are the words that define whether or not the message gets scored as spam or not. Plus if they knew which words to poison, those are the same words they could use to get their spam past the filter to start with... so poisoning the filters is pointless anyway.

I really don't see how they can get around it. I'd be interested in your views. If you really think it's dangerous to talk about it in public then let me know and I'll email you at your mangled address above. Is that your correct address?

It's not gibberish, it's steganography

[phr1]

They are sending sekrit instructions to al-spamda about where to hide the weaponz of mass distraction. Or who knows. Any government efforts to control steganography (like reported just yesterday ) better go after spammers first, or we have to wonder what they're really up to.

What I'd be interested in...

[dswensen]

...is knowing how successful this spam becomes. I get a lot of it, and I have to think that you'd have to be beyond merely dim or technically inept to take it seriously -- you'd have to be insane or have some sort of debilitating head injury. (Granted, that still may leave a lot of the Internet covered, but still).

Spammers seem to have a lot of success when they're emulating more legitimate sources like Ebay, Microsoft, etc., but I get spam now that can't even seem to decide what it's selling. The subject line says "get rid of mortgage payments" and the body is selling "V.I.A.G.01331.A." I'm not even sure what I'd be getting if I were dull enough to actually click on anything in the message. Heck, I'm not sure if even the SPAMMERS know.

I'd be interested to know if these spams are as successful as past efforts have been.

Not an effective technique

[Len]

This doesn't seem to be a very effective spam technique. It works pretty well at fooling my "bayesian" spam filter, but the spam messages have gibberish subject lines! Who's going to read a message titled "deprecatory parrot bizarre dessert"? (an actual example)

My Bayesian filter is slowing becoming a whitelist

[ObviousGuy]

There is so much crap flooding my inbox these days that the spam filter is slowly becoming a whitelist of my coworkers and a few external customers. Hardly anything else that comes in is worth the time to look at.

I know that whitelists aren't the answer, but then nothing short of immediate execution of spammers is.

The Grammar Filter

[Esteanil]

Let's see... There is translation software out there that has some basic understanding of grammar.
Should we add a grammar-filter to the list of things we look for it spam?
A large amount of incorrect grammar would increase the chances of the file being caught in the spam filter.
Of course, this would lock out most of AOL users from writing email... But is that really so bad? :P

The problem with this technique

[pclminion]

The problem with this technique for foiling spam filters is that Bayesian filters only examine words which occur in the dictionary of commonly used words. A Bayesian filter is individually trained on your personal mail. If the "red herring" words in the spam don't occur in your personal dictionary, they will be ignored by the filter and have no impact on its decision.

For example, take the word "Byzantine." This is a very non-spammish word. However, if you've never received a legitimate email containing the word "Byzantine," your Bayesian filter will not have it in its dictionary, and the word will be ineffective in "tricking" the filter. The red herring words only have an impact if they are relevent to your actual mail sample. Since everybody's email communication is different (some of us are programmers, some of us are literature majors, etc.), this is a real sledgehammer approach to defeating the filters -- and it's extremely ineffective.

This technique just proves that spammers don't understand the theoretical underpinnings of current Bayesian anti-spam methods. Otherwise, they'd be using much more common words as red herrings, instead of these extremely rare, and therefore insignificant, words.

I personally use a spam filter of my own design which is based on information-theoretic and neural network techniques. It kicks the shit out of spam, even the messages that include these stupid red herring words. The spammers once again prove that they are morons, incapable of understanding how anti-spam technology actually works.

Re:The problem with this technique

[YU Nicks NE Way]

Actually, the attack is more subtle than you think. The value of a random-words attack lies in the long-term damage it does to adaptive filters, not in how well or poorly it does with fixed filters.

When an adaptive filter sees a rare word in a spam, it is likely to assign that word high spamminess. Problem is, the next time you see that word is likely to be in a piece of ham, resulting in a false categorization of a piece of ham as spam. The user cost of such an assignment is very high, and so users will be forced to look at their junk mail...which is, after all, what the spammers want.

New use for Project Gutenberg

[KalvinB]

randomly grab a paragraph from a book and include it with the spam.

It would also help spammers to write better pitches. Use real words, actual English but put it in narrative real world sceneario format. So it reads like someone you know telling you how they use such and such a product.

"I went up the cabin last week with my girlfriend and tried out those new pills I heard about while I was there."

There's pretty much nothing in there that would be filtered. And then a slight plug of the product name with a link and you're done. It's also Marketing 101 that the less of an ad sounds like an ad the more effective it is.

But none of that thwarts my method which is to filter based on the URLs of links found in spams.

I get virtually no spam with a Mercury rule file that's all of 23KB and grows very slowly as spammers use new domains to host their product pages.

Ben

Different Techniques

[kalidasa]

The article doesn't do a good enough job of explaining the different techniques in use.

First, hash busters. Yes, spammers are loading a random jumble of meaningful words in meaningless sequences into their spam, usually in the plaintext message body of a message with HTML content (i.e., you get hash buster - html message with spam content - hash buster). So HTML-aware clients (the main clients targeted I'm sure are AOL and Outlook Express) show the spam message, but not the hash buster. I'm guessing that this is specifically targeting bayesian filtering tools at AOL (anyone know if AOL is using a bayesian filter?); it works by introducing words that would not be found in a spam corpus in greater numbers than those that would.

Second, noisy spelling, like v1@gr@. Obviously this is also intended to defeat regex-based filters like spamassassin. If you vary your cliches enough, and you introduce very strange, but easy-for-a-human-reader-to-recognize spelling variants, you make it much more difficult for filter writers to write effective regexes.

The real problem will be deliberate poisoning

[Jerf]

The real problem will be when the spammers finally figure out how to deliberately poison the Bayesian filters. So far they're using more-or-less random words, but that won't really work against Bayesian; it can tolerate that.

However, what constitutes "non-spam" is not as unique as most people think, as I've examined here. If they figure out how to deliberately put in hammy words, Bayesian will fall.

I feel OK posting this because I freely admit to this point I've overestimated them; I'm sure spammers have read that piece, and to date they have been too stupid to figure out what I said in plain English. But sooner or later one of them is going to figure out.

There's a strong core of "ham" that is "ham" for everybody, and sooner or later they're going to start abusing that.

And if I may forstall one objection... "But you don't understand Bayesian, it's [awesome for some reason and can't be beat ever, by anybody]" - I'll listen when you've actually written a program to examine filters yourself, OK? I understand it pretty damn well. It'll take more then bald assertions to convince me I'm wrong, I've done actual research, in the original sense of the word.

/usr/share/dict/words

[HeelToe]

I thought about this after seeing my inbox spam increase to about 80 a day (the box that contains what is filtered is usually 10 per hour - my adress has been valid for just short of 10 years).

Why not check the subject or first few lines of plain (not html) text and see if 80% of it is in /usr/share/dict/words? I thought about trying this out, but have been too busy to get off my ass and do it.

Slimier than slime . . .

[mjprobst]

I saw one just yesterday that contained a list of important key sentences and phrases from the literature of common charities and political activism organizations.

In other words, if your Bayesian filter accepts those, based on your past decisions, it will detect the spam. If you reject the spam, you reject these communications as well.

Good filtering practice would dictate that one reads the junk box carefully enough to find both false positives and negatives. But the sheer bulk of mail that ends up in the junk box makes this unfeasible for many.

I have started letting these particular kinds of spam through, manually categorizing them (many words of random strings, dictionary vocabulary attack, positive phrase attack) in the hopes that filtering technology will soon advance to the point where these can be used as inputs to a more intelligent system.

Of course overhauling the mail system is a prerequisite to solving any of this long-term. For once I don't mind D. J. Bernstein's Internet Mail 2000 proposals. Of course there are other proposed systems, none of which has enough momentum to start a slow steady change. The end result of any non-consensus system will be to fragment the worldwide network of Email into competing, noncompatible systems that need to communicate through some kind of loophole or gateway. Back to FIDO-net days.

I see this too

[rockwood]

I've been using "SpamBayes Outlook Plugin" since a previous /. article talked about it.

Agreeing with this article, over the past week or two I have seen excessive about of spam being missed by SpamBayes, even after marking them as spam for improved filter, they continue to hit the inbox whereas previous absolutely no spam made my outbox. Additionally, there may have only been 2 or 3 emails marked as possible spam when they were not. And zero items mark as definite spam that were not.

SpamBayes has worked great previously, but now even it is falling short.

I feel as the spammers manipulate the conents/context of the spam, it will eventually become impossible to determine the difference without physically looking at 500+ email daily.
My primary use of email is business and not personal, therefore I cannot risk missing a client email, payment, question, etc... I've also see a progression of clients having MY emails deleted or caught in spam filters due to the business aspect and requests for payments. I feel this is primarily due to the comparison of too-often-common-phrases that a spam email and a business email contain. Such things as Click here to submit payment, or Buy these Products, Overdue etc... Even though all clients I email are only clients that contact me. I never cold-email anyone.

More spammer are using this random text as the only text in the subject and body, and using an image as the content of their email, which makes scanning even more complicated, if not impossible.

Being on the net prior to what is is today (going on 20 years), I often wonder how much control the spam actually has over the net in several aspects

• If spam were to disappear, will overhead costs decrease that greatly in order for ISP's to pass along higher saving to the consumer?
• If Spam were to disappear completely, how much faster would the Internet be?

Has anyone ever done a study to determine how much effect spam has on degrading the net, and what would it be like if all spam was gone tomorrow?

Re:I don't get it, really

[McDutchie]

Why bother to market to them at all?

In addition to living in their own criminally delusional world, spammers often don't spam for themselves but work for others. They get paid by their, er, client for each message sent, it doesn't matter to them whether it's wanted or not.

Plus, there's always that .001% of suckers to keep the biz going if the cost of sending is close to zero.

What I don't understand

[Trejkaz]

What I don't understand about this type of spam is that often it doesn't contain any actual advertisement, just three or four lines of random words, and the end of the email right there.

I don't get it. If you're not selling a product, what is the spam for?

Mind you since TMDA, I haven't been seeing any spam anyway.

Re:What I don't understand

[berzerke]

[What I don't understand about this type of spam is that often it doesn't contain any actual advertisement, just three or four lines of random words, and the end of the email right there.] Actually I was viewing the source of the whole email, not the text part.

I too see this sometimes. You're not crazy (at least with regards to this). I've looked at the full source, but still can't figure out what the goal is. My best guess is either they are fishing for bounces (ok, these are bad addresses; the ones that don't bounce may be good addresses), or the spamming software has a problem (bug or is misconfigured).

Re:What I don't understand

[Brian Ristuccia]

I hope to hell they're fishing for non-bouncing addresses, because at the moment any email which SpamAssassin says is spam, I bounce.

Don't ever do that, all spam has forged headers. You're just making life hard on someone who had their address sold.

Returning suspected spam might have a small adverse effect on the legitimate holders of forged addresses, but silently deleting suspectred spam adversely affects everyone by causing misclassified messages to be silently lost. The practice of bouncing spam doesn't increase collateral damage, it prevents it. Automated processes must cause mail to either reach its destination or be returned to its purported sender. Otherwise legitimate mail will get silently lost. That's collateral damage.

This balance of burdens is fair too. Fake bounces are much easier to filter than ordinary spam. Even if the bouncing MTA engages in the unfortunate practice of sending bounces that don't contain the original message you can still filter all fake bounces with 100% reliability. Simply send each of your outgoing messages with a unique tagged, timestamped envelope sender address. Bounces which arrive at other addresses are always in response to forgeries and can be safely discarded.

A method for removing spam from your life.

[crazyphilman]

It's old fashioned, and some of you will probably make fun of me for using it, but hey, I'm old school. FYI, here's my method:

1. Create manual spam filters (NOT beyesian filters) in your inbox called "Friends and Family", "Work", "Services", "logfiles", and any others you find you need. Each category applies to a broad type of email address you'll receive email from. Then create a subdirectory in your inbox for each of these filters (named the same way, naturally).

2. For each filter, build a list of people who are allowed to email you. For example, your ISP, your bank, and your phone company would probably be added to services. Just add the email address they send their messages from to the list.

3. For each filter, have the filter move messages matching the filter (From equals ) to the correct subdirectory for the filter. Then stop processing for that message, so it doesn't get interpereted by other filters. Think of this as an analogy for ipfilter or ipfw in your firewall setup -- only you're filtering emails instead of packets.

4. Finally, DELETE EVERYTHING ELSE in the very last filter.

You USE this approach by doing a quick scan of the deleted items folder to see if anything is interesting. If not, just clean out those deleted items. It's a one step operation, much easier than selectively deleting a hundred emails one at a time.

Then, you scan each of the folders you set up, IF the folder has picked up an email, focusing only on your REAL email.

This approach has saved me a HUGE amount of work lately. My life is a whole lot easier, and it's way easier than trying to train a Beyesian filter. If I don't know you, you can't get too much of my attention.

It's all about being on the list, sort of like getting into a nightclub... ;)

Simple trick that is semi-efficient

[tomstdenis]

Just block the domain name/ip of the hosted images. Most spams I get come from random IPs but usually have common IP/domain name for the hosted images e.g.

hostz300001.com/ads/viagra.jpg

Or whatever. I've cut down from 50 spams to about 3 or so a day by doing that.

I bet a bayesian filter would work nicer but unfortunately I'm too lazy to mod the mail setup [that isn't mine] to get one installed..

Tom

Re:Grammar Check and Spell Check...

[mrpuffypants]

The solution to randomness is to spell check and grammar check incoming e-mail

Apparently you've never gotten emails from either a:

1) 14-year old girl
2) Gamer
3) UNIX sysadmin describing a sendmail .cf file

Yikes.

Re:why not filter out 1337 sp3@k?

[the_mad_poster]

1337 speak isn't a big deal. It's definitely filterable.

I've begun seeing chunks of text appearing in messages that are like legitimate mini-messages in and of themselves. Sort of like a counter weight. I don't think the aim is to pound Spam through the filters now, because what's happening is spam is getting slightly lower ratings each time while legitimate messages are getting slightly higher ratings.

In other words, the spam probably won't ever be legitimate, but it's making me lower my threshold for what is spam more and more. Eventually, I'll get to the point where some legit messages will cross over into being labeled as spam and spam will go through legit because the thresholds will be so close together as to practically overlap. It's also killing my ability to keep a spam trap that I can use to quickly train filters.

Whether this scene will actually play out and the "plot" will be succesful or not remains to be seen, however.

Re:why not filter out 1337 sp3@k?

[letxa2000]

You're completely right. I love it that spammers try to conceal their mail with weird combinations of words.

Examples from my corpus:

VIAGRA: 99.797%
V!AGRA: 99.9999%
AGRA: 99.9999% (from things like VI.AGRA)
IAGRA: 99.9999%

PORN: 98.573%
P0RN: 99.9999%
PR0N: 99.9999%

Plus, the trick is looking for things that give away spam that aren't just words. I call them "characteristics." For example:

Various pharmacy related terms: 99.9999%
HTML using % escape sequences: 98.789%
Http:// references that don't use www: 85.538%
=?ISO- in Subject: 99.9999%
Suspicious domains (BIZ, BR, PRO, etc.): 99.174%
1 "Adult Term": 70.8%
2 "Adult Terms": 85.7%
5+ "Adult Terms": 99.9999%
5+ HTML Comments: 92.0%
10+ HTML Comments: 98.3%
30+ HTML Comments: 99.9999%

In short, there are so many aspects of a message you can analyze and make "Characteristics" that my Bayesian filter can often make a decision entirely based on the characteristics without even looking at some of the terms used within the message. But if the characteristics aren't damning enough, the content virtually always is.

Re:gibberish...

[Mr Z]

Actually, I avoid deleting my spam. I have an archive now of over 270MB of spam that I can use for a training set for whatever filter I might intend to deploy.

That archive has more than just spam, mind you. It also has all the virus/worm email I've received over the years as well, such as the "Internet Email System" informing me of an undeliverable message, or "Microsoft Corporation" providing me a convenient, easy to click "December 2003 Internet Update" or whatever.

*sigh*

--Joe

Gibberish, or code?

[cr0sh]

I, too, have noticed these seemingly random words that seemed to have nothing to do with the main text of the spam. I have also noticed the "gibberish words". One of my thoughts was that it was for defeating or bypassing bayesian filters - and likely, that is the case. But my thoughts turned to another possible use...

What if spam and the spammers software - was actually being used by a third party in a surepticious manner to send/receive messages? Kinda like plaintext stego. Maybe the software used by spammers is backdoored by this third party - he sends instructions to the machine(s), maybe via a virus or something simpler, the spammers send their messages, but "unknown" to them the spams have this garbage at the end. The spammer doesn't really care, maybe he bitches at whatever passes as tech support for the spam software. Most people who recieve the spam see the stuff as garbage, or filter busters. But a certain group of the third party's friends - they have special email software that downloads these spams, and strips the garbage out, decodes it, and reassembles it into the real message. Maybe each spam only contains the equivalent of a couple of characters after decoding (maybe the garbage is actually packets telling order in the sequence, and other info to reconstruct the message) - but over a week or so, an entire message could be sent...

What is the possibility of that? Occam's Razor suggests otherwise, and filter busters are probably what the stuff is - but...what if...?

Re:gibberish...

[1u3hr]

Someone keeps buying from spam.

Not necessarily. I'm sure most of those people (had to backspace over a few epithets) who spam Make Money Fast either lose money or get into legal trouble. But the damage is done (to me) before they learn that it won't make money. I think the driving force is selling spam services to gullible clients like these. (Not including the industrious Nigerians who seem to take a more personalised DIY approach.) Even if someone DID want penis-enlarging cream, I think by now they'd have a source of supply, that market must be pretty saturated by now.

Threshold? Bah!

[glpierce]

I'm worried about spammers realizing that they can effectively negate the usefulness of filters without breaking a sweat (spammers, please don't read the following). If they switched from super-short fake messages to mock-real messages (a paragraph or two long, a legit-sounding subject, etc.) and they all sent out millions a day, everyone would be forced to turn off their filters. There would be no effective to distinguish those fake messages from real messages for most people (without a whitelist/blacklist system, which does more harm than good for most).

In such a situation, email would grind to a halt. Anyone who kept trying to train their filters would just end up blocking most legit emails, and those who don't train for it or turn off would be flooded with real and fake messages they can't distinguish between. The messages would even be profitable, so long as your "friend" included a link to some "cool website" that happens to sell [fill in spam product here]. Go ahead and train your filter to block emails containing URLs. Hah! Maybe if you don't have a job, friends, or buy things over the internet you can, but for most it's just not going to work.