Slashdot Mirror
Using AI for Spam Filtering (w/ Source Code)
jarhead4067 writes "Article snippet: "Up until recently, most researchers in the fight against spam have failed to classify it as an artificial living organism, hindering the development of effective tools and techniques to kill it. While this classification may sound strange, consider the following..." A novel approach to filtering spam, and hey, there's free source included."
There are too many people accessing the Web site at this time.
SHE does throw dice.
BINGO!
... after we get an AI to counter the Slashdot effect.
"Enough of this wretched, whining monkey life." -- Marcus Aurelius, _Meditations_, Book 9, 37
That one went down after 3 replies :(
No luck for mirrors?
I think we can keep recursing like this until someone returns 1
Google cache
I won't believe spam is a living organism till I see Marty Stouffer do a special, complete with comedy 'boing' noises and 'aint that cute' music as we watch a mother Spam care for her young.
How about a nice game of chess?
I dislike spam, in the same way only more than I dislike all the billboards along the highways. They get in the way of what I really want to see, and essentially make me feel inadequate. Billboards make me feel poor, because I can't afford a new home, or a meal at that expensive restaurant. Spam makes me worry that my penis is too small, my breasts are too small, I'm too fat, I don't send enough money to Nigeria. That said, it's illegal to saw down billboards, but it's not illegal to filter spam so I don't have to see it. The article is slashdotted, so I can't read it, but I think we already have good (free and open, no less) spam filtering available. I use Spam Assassin on my server, plus my mail client has a spam filter for double protection. Both have been learning more and more what constitutes spam, and it's rare that I even see spam anymore. If everyone would use these filters, spam would no longer be as profitable.
The emperor is naked.
And the AI says....
The page cannot be displayed
There are too many people accessing the Web site at this time.
Please try the following:
Click the Refresh button, or try again later.
Open the www.generation5.org home page, and then look for links to the information you want.
HTTP 403.9 - Access Forbidden: Too many users are connected
Internet Information Services
"Quoting famous computer scientists out of context is the root of all evil (or at least most of it) in programming." - K
1 time out of 3 I can access it.
"living organism ... and techniques to kill it"
Next thing we know, we will have Animal Rights Activists in Washington, D.C. protesting our "spam traps"
who | grep -i blond | date cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
Sounds pretty cool, but i doubt there will ever be a way to completely get rid of spa unless the governments pass laws, and an international body is set up to prosecute spammers. Making the risk too high for them to bother doing it.
.
> most researchers in the fight against spam have failed to classify it as an artificial living organism
Who would have thought Skynet has its origins in spam?
Who talks like this? Really.
The owls are not what they seem
Isn't Bayesian filtering system used in, Eg, Mozilla Mail classified as an AI?
I mean - hello, humans create it.
We're not up against a new being - it's the same type of beings that create scripts for the hell of it that wreak havoc on computer networks because 1) "We can" or 2) "To show them their weaknesses".
It was a very interesting read for sure - the genetic marker bit was quite interesting. Admittedly though I got about 2/3rds the way through it and lost interest.
Blame the spammers I say. ^_^
The site says that There are too many people accessing the Web site at this time. and that I should contact Microsoft Support.
Maybe if we hit reload more often the site will become accessible again ;-)
This is really a testament of strength of yet another MS product.
On a more serious note, anybody has a mirror?
Code is Speech. No to Censorship.
From the article: Spam has become the first great plague of the 21st century.
Yeah, spam sucks, but isn't calling it a plague going a bit overboard(and didn't it start before the 20th century?)
Maybe AIDS is a worse plague. I know, it didn't get started in the 21st century, but infection rates have started to climb in the 21st century, esp. in Asia, after showing signs of falling...
Spam has become the first great plague of the 21st century. Over 60% of all e-mails are spam, costing U.S. corporations more than $10 billion annually, on top of the productivity lost from scanning through e-mail and deleting spam. Along with this, an estimated 5% of spam campaigns are a pure and outright scam, with the remaining majority pitching products that are dubious at best. It used to be parents had to worry about their kids surfing and finding pornographic websites, now we have to worry more about our kids opening an e-mail client and finding a pornographic spam message. Spam must be stopped before it cripples the infrastructure of the internet and drives users away from one of the greatest forms of communication, E-mail.
Can Laws Defeat Spam? No. This has to be one of the greatest misconceptions of users. The internet is just that, an "INTERnational NETwork" that cannot be governed by one country's laws. Spammers can exist anywhere on the internet, meaning they can sling their wares from anywhere in the world, making the laws of one country completely irrelevant. Also, the decentralized, self-organizing design of the internet makes it nearly impossible to regulate by external means. It would be easier to regulate the weather than to regulate the internet.
Spam as a Living Organism
Up until recently, most researchers in the fight against spam have failed to classify it as an artificial living organism, hindering the development of effective tools and techniques to kill it. While this classification may sound strange, consider the following:
Through the fight against spam, spam has demonstrated an uncanny ability to adapt to the conditions of its environment, namely the internet. When one barrier against a strain of spam is put up, another, resistant strain appears. This is similar to how bacteria builds immunity against antibiotics, the strains that are not immune will die, while the ones that are immune take over and become the dominant, drug resistant strain. This leads to the belief that spam will not die until the barriers of its environment evolve faster than it does.
The internet is a complex chain of systems that all rely on each for the other's survival. Without an internet protocol, a web browser couldn't exist. Without web servers, the web wouldn't exist. Without ... (you get the picture). This chain of systems can be likened to an eco-system, with spam existing at a parasitic level of species within this system. It consumes resources (bandwidth, servers, time) in its attempt to reach its primary host: us. Once spam reaches its target, its sole purpose is to solicit its "food" from us, primarily money. If it is effective, that strain of spam lives and continues to propagate, otherwise it will die. Can the internet eco-system be modified so spam can't feed?
Just like any organism, spam contains certain traits that uniquely identify it. This can be a combination of words, information inside the header of the e-mail, the format of the message (HTML, plain text, rtf), the message encoding (base64), does it contain image links, the number of links, does it contain hidden text, so on and so forth. Up until recently, spam filters have primarily focused on just one of these traits, the wording of the e-mail. Spam, being an organism, evolved so this marker was hidden within its code, making it difficult at best to filter. It did this by including random, non-spam words in hidden areas of the e-mail, by modifying words like Viagra with V1@gr@, sending spam as image links, and by encoding the message in a format that filters could not read. The good news is this "gene" is still present, and can be unlocked by identifying the defensive genes wi
wot no sig
This is really a testament of strength of yet another MS product.
No, more likely it's some guy trying to use Windows 2000 Pro as a webserver. It has a ten connection limit; you're supposed to use a server version of windows for live webservers. I've never seen that error from a server version of Windows.
The quote at the top of the page is pretty damn funny; "Tricksy spammers, they'll stop at nothing to get my precious."
I have to ask; if you're going to classify spam as an organism, would you not also have to classify email as an organism? So if spam is predatory in nature, then regular email is not?
And so what if we do this? What guarantee do we have that spammers won't evolve past any thwarting mechanism developed? My thoughts are that you have to keep slowing it down, to the point where only the most experienced spammers can get past the armor. Make it so tough to start spamming that people can't just simply pick it up as a hobby. I can remember back in the early nineties when it was relatively easy to spam and there were quite a few people doing it. But nowadays it's not that easy to just start spamming. There's no guarantees that the email will get through, so while many people try to spam, many fail at it. There's no payoff, there's less people taking it up.
But at some point, we will have to deal with the moneybags supporting spam. Maybe legislation could make funding spam illegal? Fine the little old ladies who give their money to spammers, and you'll see many of that just stop. Cut off the revenue generated by spam, and you cut off the spam itself.
The dangers of knowledge trigger emotional distress in human beings.
Just what I want, my computer learning from porn ads.
The first half of the article is total bullshit whereas most of the second half consists of wellknown facts about spam and about neural networks.
I wonder if neural a network will work as well as a bayesian network. My intuition says no, but I guess it's worth a try.
Also, it ties up email servers meaning yours can take a little longer. I once got a spam message 2 weeks after it was sent, so what happened to legit email is a mystery.
I think for the damage it does both to servers (slowdown) and to people (moneydown), it could be called a plague
Get paid to search..It's geniune and
Your web server can also be classified as an artificial living organism. But I ain't so sure about that living part anymore...
There must be a theory that explains why /. hordes hang out at a site carrying news for nerds they presumably want to know, but that becomes inaccessible simply by the sheer numbers of slashdotters.
"Academicians are more likely to share each other's toothbrush than each other's nomenclature."
Cohen
How exactly is this news ? It seems that the author of the neural network idea didn't do his homework - e.g. DSPAM includes neural network as an experimental classifier already. And compared to the proposed C# solution, DSPAM is a widely used and mature product already.
Regards, Jan
We already saw a plagiarized article green-lighted, and now this? Cmdr Taco, Slashdot was a brilliant idea of yours, and I love your site -- but that's because I have reasonably high expectations for it.
First, the submitter of this article has he email address jarhead4067@hotmail.com -- and so does the article's author.
Second, what is presented is not a genetic algorithm. The characteristics of the email to be considered to discover if the email is spam are finite and hard-core -- and even the threshold some characteristics must reach to qualify as spam are hard-core:
A genetic algorithm is one in which the goal is hard-core, different means of reaching that goal are generated, and the characteristics of the most successful are used to generate the next "generation"; this is repeated until the goal is reached.
But in this model, each "chromosome" contains statistics about one email. The heart of this model is to train a neural network with known emails ("chromosomes") and then tests unknown emails ("chromosomes") against the network.
Neural networks have a checkered history in Artificial Intelligence research. A (very much simplified) model of biologic neurons, neural networks were for a time seen as a great hope for Artificial Intelligence. A neural network basically starts out with an array of input nodes and an array of output nodes, with each input node connected to each output. Each input corresponds to some characteristic of the items the network is trained with: for classifying animals, the inputs would be characteristic of animals, e.g., "furry", "bipedal", "feathered"; each output a classification, e.g., "mammal", "bird", "human".
To train the network, the input nodes are set to the characteristics of an item, and then the strength of the connection of those inputs to the correct outputs is increased (or that of other connections is decreased -- it's the same thing). With enough training, it's possible to isolate the salient characteristics from the ambiguous one sin a mechanistic way.
This is useful, but it was soon discovered that these simple neural networks, for certain sets of inputs, failed, because of overlapping categories: both birds and humans are bipedal, but only humans are also mammals. In a single layer neural network, the connection strength between input "bipedal" and output "mammal" would fluctuate, unable to describe humans or birds well. These problems can be alleviated by adding additional "hidden" layers of nodes between input and outputs, and by allowing "back-propagation" from output or hidden nodes to layers "previous" to them.
But even with these enhancements, it's been conclusively shown that some problems are intractable for neural networks. In any case, neural networks are no new thing.
Of course I have no idea if classifying spam is intractable or not, but I have to question whether using a neural network reliably can outperform Bayesian (or quasi-Bayesian) filtering. My guess is that since Bayesian filtering can judge email by the occurrence of single tokens ("words"), and not just "chromosome" statistics, and given that this "new" method also uses Bayesian filtering to generate one of those "chromosome" statistics anyway (and for only the most difficult to characterize emails to boot), this method itself probably mostly relies on its Bayesian sub-component.
So I'm a bit at a loss to see why this method is in any way revolutionary or even particularly interesting, or why it was green-lighted for Slashdot. Of course, I only gave the linke
Opinions on the Twiddler2 hand-held keyboard?
from SpamAssassin? It takes a bunch of rules, applies them, and uses a neural net to classify the message. Seems to me SpamAssassin does the same thing, only is more mature and extensible and uses a genetic algorithm rather than a back-propagation neural net.
and you must have typing that message from the web terminal installed in the vestry.
That it no only recognizes spam but also Slashdot visitors.
2. Spam lives within an eco-system, and we're its food
Yummy! Slashdotters for breakfast!
Neural networks, are basically many interconnected neurons, which are mathematical functions performing a weighted sum of the inputs and delivering an output through a non-linear transfer function. They are used in a research oroject I have worked on to determine data patterns in 'electronic noses'.
Link for info here
Personally, i think this article is very full of buzzwords, but using ANNs to pick out spam is very possible and would return an excellent catch rate, especially if the data collected from being 'trained' could be collected and refined for making better networks.
I'm not sure how 'up for it' home end-users would be, it may take a lof of hardware to run (we use a shit load of Xserves, but maybe thats overkill).
It could be great for actual mail servers, like gmail for example.
The entire concept is quite ridiculous.
The guy proposes picking nine well-known indicators of spam, ones that could be (and often are) implemented in rule-based spam checkers, then proposes we use a neural network to evaluate a message based these metrics.
Problems:
1) If you detected spam indicators, this is indicative of spam, no? The whole "fancy" bit of this technique is thus needless.
2) These indicators are not inherent to spam, just represent most current bypassing / obfuscation techniques. If you filter them out, they'll evolve. There is nothing that makes his spam filter follow the arms race.
is it really little old ladies who keep spammers going, or is it porn surfers? I don't get much spam anymore, from hardly ever using email or giving out my email addy and moz's built in filter, but from what I remember the spam I used to get was way more porno,porno with anatomy enhancements, porno and and viagra, then it fell down the list, toner carts, home mortgage refinance, cheap semi legal drugs, etc.
Anyway, I tend to like my basic idea. Regular email is a default setup of accept everything,then struggle to try to filter out the junk. I prefer a universal whitelisting/blacklisting only, have a default setup with email clients to ban all email,that's the default blacklisting, and only let in who you want on an addy by addy basis, that's the whitelisting. If spam wasn't the predominant email, you wouldn't need to do it that way, but it is, so just recognizing that fact means you should go to a default blacklisting. what we are trying to do now is 180 degrtees backwards from what logic dictates. With spam and bayseian schemes, you are struggling and trying to lock the barn door after the horse gets out,that didn't work well way back then, ain't gonna work well now.
You get around "first contact" issues in this system I propose with businesses and mail lists who need access to "new" people regularly by using a webform (cheap/fast/works good enough) or the telephone (for very important transactions) as the first contact.
Feedback from individual spamfilters is used in realtime to throttle back bandwidth on high-probablility spam generators. Not cut 'em off, just slow them down to the point where it'll take weeks to send out their evil payloads. Aggregated spam statistics are used to update individual filters. A reputation system develops to weight individual spamfilter results.
. . . it's Sunday morning. Get off slashdot, so I'll stop getting this damn
HTTP 403.9 - Access Forbidden: Too many users are connected
Problem: I can't access the article.
Solution: It's Sunday morning. Why don't all of you geeks get off slashdot, and get a life.
(Of course, this doesn't apply to *me*:
I *had* a life once, but demonstrated that I'm incapable of maintaining it.)
What I note is missing is how to deal with the spammers attacking the network using it's own techniques against itself. For example, flipping the ham/spam caches so that "good" mail is classified as spam and spam email is classified as good mail.
Without know EXACTLY who is participating in your network, there is no way to guard against this... and once you solve the problem of knowing exactly who is participating, then why not just use that as your uber whitelist?
I think about the only good thing I can say about this article is, at least he's not out killing puppies.
Religion is a gateway psychosis. -- Dave Foley
"Up until recently, most researchers in the fight against spam have failed to classify it as an artificial living organism, hindering the development of effective tools and techniques to kill it."
That's because spam isn't an artificial living organism. It's a bunch of emails trying to sell stuff to fools. If you think that it's a useful analogy to draw, then by all means do it, but don't blame everybody else for hindering anti-spam development simply because they didn't consider it and you think it's so cool.
Moreover, the proposed idea of using a central server to coordinate and select rules doesn't work, because everybody gets the same rule sets sent to them and the spammers work out how to bypass them. Bypass one, bypass them all.
I've given up on Spam filtering and concentrating my efforts on Ham filtering.
Basically the present thinking is based on attempting to filter spam out - I would argue that given the amount of variables involved, it it a method doomed to failure. Current methods also assume that the incoming mail is mostly valid, and are attempting to remove the undesirable parts - spam.
What I am having success with is turning this on it's head and assuming that the bulk of incoming mail is bad, and filtering in messages that I want.
The way I am doing this is to use my address book as a whitelist - if an incoming message originates from someone in my address book, then it's delivered into the inbox. If not, then they are moved into a "not in address book" sub folder. Anything my ISP spam assassin based filtering marks, is sent into the "Spam" folder. Doing it this way means that I am only notified of incoming mail that is confirmed from someone in my address book. Periodically I check the other folders (obviously).
We have come to the point I think where the number of variables involved makes filtering in a less intensive process than attempting to deal with the myriad of underhanded techniques that spammers use. By limiting the mail I want to people in my address book, I make it so that spammers are the ones having to deal with the variables as they would have to guess addresses in my address book. If lots of people started filtering like this when we would see spammers using known bulk mail addresses (such as the address iTunes receipts are mailed from) however we can simply alter the filter to include the originating IP / mailer and so on.
Think of it like fishing - you wouldn't attempt to control an entire ocean and remove the water to leave the fish - you accept that the water is there and develop techniques to get the fish out.
post-docs, that's who. ;-)
(see bottom paragraph)
I'm still groggy with the earliness of the hour, so I'll bite here and assume that you're being serious.
The answer is simple: Don't allow your self-image to be formed by other people, particularly low-lifes such as spammers. Seriously, do you give two hoots what a spammer thinks of you? Particularly when this is:
Unfortunately, most people aren't so successful with filters, particularly if they cannot tolerate any false positives at all. Even for those who don't have their own mailservers, every decent ISP nowadays offers server-side filtering -- but it is far from perfect, and I doubt you'll find too many people who claim that filtering even comes close to eliminating the spam problem.
Microsoft Windows is, fittingly, the official Desktop OS of Olig
(http://slashdot.org/~October_30th/journal/)
SAYS"
consider the following...
Who talks like this? Really. "
BUT AT THE SITE LINKED ABOVE, JOURNALS"
All things considered, a very pleasant visit.
As usual, my only regret is that I couldn't speak the local language beyond a few simple words I picked up in the first few days. Having to resort to English - which is not my native tongue anyway - always annoys/embarrases me to no end when traveling.
Isn't it about time for star trekish universal translator gadgets already (and let's not forget about all those flying cars we were promised decades ago)... "
YOU ARE ALL MOUTH.
COME TO THINK OF IT, THAT ONLY MEANS YOU FIT RIGHT IN 'ROUND HERE.
I just got a SPAM 2 days before it was sent...
Subject: rattlesnake 180 piroshki
From: "Randell Workman" <mrqahlst@yahoo.com>
Date: Tue, July 13, 2004 4:49 am
--- Have you seen MURL?
How does Al have time to read all that spam? And if that wasn't bad enough, now he's been slashdotted.
For starters, he things Internet is short for "INTERnational NETwork" as opposed to a NETwork between entities (vs. network within an entity: intranet).
Then, his criteria:
Is the format of the e-mail HTML?
This is not a bad criterion.
Is the e-mail formatted in valid HTML?
Have you ever seen a commercial program (esp. word, used by Outlook) generate good, 100% valid HTML?
Is the e-mail encoding base64?
No argument here. Unless base64 could be confused with Unicode - don't think so, but not sure.
Does the e-mail contain image links?
Does the e-mail contain "hidden" text that the user cannot see?
Heck, yeah, block it.
Does this e-mail have a large number of recipients?
Most of the spam I get has less than 5 recipients, and a lot of my mail is from a listserv with more than 5 recips.
What's the ratio of links to words in this e-mail?
I generally see only one or two links in my spam. Although I do see zero links in most of my ham.
What's the ratio of misspelled words to words in this e-mail?
Dear lord, no. This is a worthless criterion. Maybe if you looked for a ratio of non-letters (@, |, etc) to letters, but not spelling.
What's the Bayesian spam probability of this e-mail?
WTF does this have to do with AI?
Basically, he's stated the obvious, then made some really idiotic assumptions. Plus a shitload of spelling and grammar errors.
From a life form analogy perspective Spam is not evolutionary, it's more an example of intelligent design.
The problem with the proposed method of detecting spam is that spam changes often. It is mutated to get by Spam Assassin, Brightmail and Spam Bayes. This is just another attempt to get ahead of the spammer on the treadmill.
You need to change the tokenizer regularily, you need to handle invisible ink, etc. etc.
This solution has the added difficulty of the training of the neural network -- how long does that take? Something like Spam Bayes starts recognizing new spam after a few messages of the new type.
1. While the author proposes some marvelous cure based on treating spam as an organism, he just lists traits that any spam filter can use, and which most probably do, though he would suggest that most don't. I fail to see how the artificial-life observation improves spam non-spam determination from the list of traits he proposes filtering on.
2. The article reads like a sales pitch for the author's spam filter.
3. If 2 is true, and it is a sales pitch, then you have the irony of a very effect form of spam that makes it past the slashdot editors.
It's ALIVE!!!!
Letter To Iran
Spam comes from people. People are organisms. People adapt so spam adapts. There is complexity but it follows from the source, there is no emergent behavior.
Have they tried Penicillin?
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
If I were to sum up this approach, it would be SpamAssassin with a multi-layer neural network. I should mention that I maintain the tool that SpamAssassin is useing to train its single-layer neural network for version 3.0, so I can honestly say that have a fair amount of experience in this area.
I'm not too keen on Evans' use of the biological metaphors. I think that they only confuse the issue of what he is doing. I will use the standard terminology, features, from here on out.
What he is doing is finding a nonlinear decision surface between two classes using a universal function approximator. I will explain this in layman's terms.
Imagine a sheet of paper filled with multi-coloured dots where these dots are arranged in clusters and each cluster contains mostly the same number of dots. Starting with a simple example, imagine two clusters of dots, one blue and one red. Assume that you can draw a line that separates the two clusters. That line is called the decision surface. You would say that any new dot that would appear on one side of the line will be called red and the other blue. Any blue dot that appears on the red side of the line would be misclassified as red. This is referred to as a linearly separable problem.
Now, imagine a more complex arrangement of clusters where you can't draw a straight line to separate the red from the blue, but you can separate them using a curved line. This is called a nonlinearly separable problem.
Artificial neural networks are very good for representing these decision surfaces. They are constructed of one or more perceptrons. A perceptron uses an activation function and a transfer function to take a set of inputs and produce a single output. The most popular form of neuron uses a linear activation function and a sigmoid transfer function. The linear activation function is the sum of a set of weighted inputs, i.e. f(X) = sum w_i *x_i. The logarithmic sigmoid transfer function is g(x) = 1/(1+exp(-x)). The output of the perceptron for any given input is O(X) = g(f(x)).
These perceptrons can be chained together in many different ways. One popular method is the multi-layer perceptron, where a set of neurons in the hidden layer process the inputs and pass on their outputs to the output layer where the final output is formed. I don't have a source for you, but it has been proven that, given a large enough hidden layer, the multi-layer perceptron is a universal function approximator.
As long as all of the transfer functions are differentiable, you can train a neural network using error backpropagation by gradient descent. I will leave it as an exercise to the reader to learn how it works, but I assure you that it is very simple. Machine Learning by Tom Mitchell has a good section on the subject, as does Fundamentals of Computational Neuroscience by Thomas Trappenberg.
Evans has identified a large set of features of e-mails, some of whom on their own convey little or no information about whether an e-mail is spam. He trains the neural network to recognize the combinations of these features which can lead towards the conclusion that a message is or is not spam. While his approach is a good idea, I would hesitate to call it novel. Massey, Thomure, Budrevich and Long did a very similar experiment [3] where they used a multi-layer neural network with SpamAssassin.
While his approach is good, there are some downsides for widespread deployment that need to be addressed first. With a large feature set like he is using, you will probably need a lot of training data to find a good fit with a multi-layer perceptron. To train the single layer neural network for SpamAssassin 3.0, I'm using 160000 messages.
Also, as his own arguments show, spam adapts to spam filter technology. Most of the features that he presents in his whitepaper can be easily fooled by a spammer. They can deliberately manipulate these features to evade the spam filter b
On a Sunday at 15:00 UTC, what time-zone would you speculate is occupied by the plurality (if not outright majority) of actively-browsing slashdotters?
;-)
"Besides, what's wrong with swinging by slashdot when you stagger home at 4am? (assuming you're alone, that is.)"
I think a lot of people would say that if:
(a) you stagger home at 4am Sunday,
(b) you're alone, and
(c) you then swing by slashdot *despite* the troubling evidence of (b) that your time is misused,
then you don't "have a life".
Also it was not clear to me the connection with biology.. that is, it seems that genetic analysis tools might be very useful, and the ideas about how spam acts like an organism and has "genes" is great. But, it was not clear that this has anything to do with the programming strategy.
For example, the use of a perceptron might be a great idea but to someone not trained in them it is hard to see how a multilayer perceptron would be especially good. Also it is not clear that this is what is used in real world genetic analyses. (For example it would have been interesting if genetic databases and bioinformatics tools like BLAST were mentioned). Also the Chromosome object does not obviously have anything to do with a real chromosome; it is confusing and made me wonder if there was something I was missing, or was it named that way to sound "cool"? Also it was not clear to me if any of the dynamics of genetic transcription and whether gene crossover, mutation, and selection have anything to do with this project.
Also I am curious about the choice of programming language. Being a perl fanatic I wonder why that is not being used, and of course perl is great at text, and pattern matching, and the important parts of many modules are invariably in C or C++ already, etc.. But also perl is a language of choice for bioinformatics, and there are a number of existing modules for example BioPerl which wraps other programs and Boulder which is an interesting format that could be used to pipe spam genes to other people's filters. Now I don't know if existing bioinformatics tools could be applicable but certainly these are things that ought to come to mind.. and what these tools do is not trivial, and if genes are a valid metaphor for spam components then there is a potential for existing code to be used too. That is something that would be cool.
There are also documented, easy to extend perl modules related to using genetic algorithms or for rolling your own analysis modules, I'm thinking of Genetics and AI::Genetic.
Finally I note the use of the term Corpus. This is really interesting, and suggests the author is into computational linguistics which also represents a massive amount of existing, nontrivial pattern resolution code.
So I'd like to know more about the relationship of both computational biology and computational linguistics to spam. For example, one big part is going to be how to identify genes, or whether you need a generator of pattern matchers that will be able to identify the existence of a gene.
Also there is a short bit about stopping spam by making it literally not pay to spam. I'd like to hear more about how that might be linkable to the biological metaphor.
I don't mean to detract from the work represented by this article, not at all. But I would like to know more about how the system analyzes and exploits the realities of biological dynamics to make a superior antispam tool. For example it would appear that some "genes" might be postulated for links to websites or even mail servers (the vectors of the disease). And some linguistics tools might even help link references to product types as genes.
Finally, and this is just brainstorming really not criticism, I was bothered by the development of a 0 to 1 probability of ham or spam. This is to me the biggest problem with automated filters. I know it can be done, since my antispam method consists of hitt
Internet stands for "Interconnected Networks".
Meme of the day: I browse "Disable Sigs: Checked". So should you.
I'd like to see more posts like this one on /.
First off, identify the characteristics of the spammer's mail servers. In my experience, they are usually zombies or open relays that I don't have any legitimate contact with anyway. So.....
Seed the spammer's databases with a bogus address. That's easy to do. Just post what looks like a legitimate address in places that spammers are likely to scan.
Then, any email going to that bogus address is broken down and the originating address is put in a blacklist for your FIREWALL. Any connections from those sites are not even acknowledged.
Unless your mail server has previously sent email to that address. (this will take care of spammers sending from Hotmail or AOL or some such.
You'd also need a method of "learning" addresses that had not sent email to your bogus address. But that should be easy and similar to "SA-Learn" in SpamAssassin.
There. You only block sites that have sent you spam in the past but it should take care of over 80% of the spam (in my case).
Eventually, the spammers will congrugate on major sites that you have legitimate contact with. In which case, those sites can implement throttles to restrict the outbound email.
A nice side benefit is that if someone at one of the open relays DOES try to send you legitimate email (and follows up with a phone call on why you haven't responded), then you can explain that they have an open relay and are spamming the world and they can fix the problem. Everyone is happy!
Yet Another Way For Automatically Pressing Delete.
Of couse, this won't solve the bandwidth/ressource theft problem...
No, the only solution to kill spam is to immediately nullroute any network that keeps any spammer (or spamming zombie) for more than 2 hours.
I agree that most of his tests are useless. Not to mention, they are easily passed by pasting a few passages from any legitimate source at the end of the message. That will throw off the percentage estimates.
Any tests that are run on the CONTENT of the message will eventually be bypassed as spam gets designed to pass those tests.
I believe that focusing on the SERVERS that send the spam is the only workable approach. Identify which servers send the spam and have your firewall drop those connections.
Kind of like a black/white list, but only with respect to connections.
If you send email to a site, it goes on the white list. Since most companies send email to fewer sites than they receive spam from, incoming connections can be checked against the white list and authorized if on it, before checking on the black list to be dropped.
It will take some processing power, but I don't see any way to handle it. Content analysis will, eventually, fail.
This also solves a large chunk of the "false positive/negative" issue. Since the email never hits your regular filters (SpamAssassin), then it will not get incorrectly flagged.
If it is a legitimate email coming from a blocked host, the SENDER's system will inform the SENDER that there is a problem with connecting. Then the SENDER gets on the phone. Which should also result in the SENDER fixing their zombie/open relay.
(b) you're alone, and ;-)
(c) you then swing by slashdot *despite* the troubling evidence of (b) that your time is misused,
then you don't "have a life".
So if I hit the town on Saturday night and don't manage to pull I've no life??? Do you bat 1.000? Do tell how.
... using freecache, instead of using somthing for what it's not susposed to be used for, and doesn't give a very good view of the site to begin with (images, etc, even though they show up because it can still get them from the original site in this case). Why aren't we linking to freecache in our stories? Maybe slashdot could use somthing that would strip the URL in all links, and always use freecache unless a flag was set to specifically not to??
Sig: I stole this sig.
mfh (56): What guarantee do we have that spammers won't evolve past any thwarting mechanism developed?
SpamByte: Game Over, Spammers/Computer Crackers...
This post describes two programs I coded that will eliminate lots of spam and malware for Window's systems if used widely. Both programs, when used together properly, make it effectively impossible for a user to receive spam or malware by email.
Baysian filters are bypassed just like any other. I'd bet most of us here have tried some form of adaptive filtering with varying results.
He's right in one key respect though -- spam is cheap to send, but spam DESTINATIONS (the links they try to get you to go to) are relatively expensive. You can't registered a hundred thousand domains a day. While its cheap to get one or two, massive domain registration is an expensive proposition. That's currently, IMO, the best way to catch spam once you've gone through the bonehead catch of faked headers.
Personally, I do two stages: First, I catch the obvious stuff -- it says its from AOL.COM but didn't come from their published servers. duh.
Then, I take those "known spams" and search for the call to action link -- what url are they trying to send me to. Take the primary part of that (the domain, plus a little more) and make a list of "probable spam destinations".
I do the same thing with known good mail (mail from people I have sent mail to).
Now have I have good baysian fodder -- actual destination lists both good and bad.
Making a baysian list out of those results in a fairly accurate secondary filter.
Email inbound to me now goes through three checks:
1) have I sent you mail before (whitelist)
2) is this obvious bonehead spam
3) how many links in the message are to the same place as the ones in the bonehead spam?
This works to stop 98% of the 400+ spams a day that get sent at me with a very very low false positive ratio.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
Bayesian is just one of the features that's captured, and then the rest is plugged into a neural net, which is AI. Read all the article...
I think that there is an undiscussed danger in AI picking through my e-mail: it might have different tastes than I do.
For example, maybe the AI mail filter will decide that it really likes those e-mails about Viagra or penis enlarging. Perhaps it will get jealous of my significant other and decide to delete her e-mails.
AI sounds cool but what we really need is spider-sense spam filtering!
maybe because freecache is only for large (as in several megabytes) files?
HAND.
If spam is a living organism and we want to control it, it's not enough to have a filter that passively nibbles at what swims nearby. Write something that invades spammer's servers, makes charges with all of their credit card numbers and then e-mails a final "spam" with an outlook express-based viral copy of itself before formatting the hard drive. Let it adapt to that!
So clearly we shouldn't use a whitelist alone, as the article suggests, it should only be one phase of filtering.
Okay, but then if you are using other methods that are effective at filtering out spam from potentially forged email addresses, wouldn't those methods be just as effective at filtering out emails that weren't on the whitelist as those that were? In other words, isn't the whitelist completely superfluous? To further cap it off, this makes it increasingly difficult for people that you do not necessarily expect to contact you for possibly legitimate, or at the very least, non-spam, reasons. I've had someone I used to know some 20 years ago contact me because they recognized an anecdote that I was relating right here on slashdot. If I had been using a whitelist, that email would have been classified as junk mail and I most likely never would have even noticed it, let alone read it, and I was actually quite happy to hear from the person that I hadn't seen in years.
Also, there is the very real possibility that an otherwise legitimate emailer's computer may have been compromised by a virus/trojan to send out spam. It's not particularly fair to suddenly block these people because there's no practical way for them to notify you to unblock them if they should happen to be able to purge the trojan from their system.
The problem with spam is that the sender addy can be faked, and I can see no legitimate reason to do this other than to prevent your email address from being harvested by a spammer. I do not predict an end to email spam until the email protocol is completely redone (which may never happen).
NOTE: The sample code for this application is in C#. C# was chosen over C++ so beginners could better see the structures of the process, and C# was chosen over Java because of the inherent performance advantages of .NET.
What morons. what total losers.
I don't know about that... everyone I know gets spam, but nobody I know gets AIDS!
(it's a joke; laugh!)
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
Oh.... so thats why this MS product is so strong... who wouldve thought... :)
NO SIG
Its hard to take someone seriously when they think that "Internet" is short for "International Network".
its "Inter-connected Network" dumbass.
Go after spammers' customers. If they have to pay $10,000 for every spam sent on their behalf, they'll soon stop,
Fuck the spammers. They are merely supplying in response a demand.
Dry up the demand by an internationally (I know of NO govm't who'd turn down money,) backed law making it illegal to have spam sent on your behalf.
The response to spam is NOT going to be technical.
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
Can we put it in cans and eat it?
"So if I hit the town on Saturday night and don't manage to pull I've no life?"
not at all what i meant (or said).
merely saying (amicably), if one seriously wants to be in a relationship, one's time could be better spent than partying until 4AM AND following up by hitting slashdot; e.g., get to sleep earlier to make time to spend Sunday at the park, museum, etc.
anyhow, it was semi-facetious (and apparently poorly worded).
Bayesian filtering is very simply the probability that a word will appear in one context or another. Once you've done this for a huge selection of words you select a few thousand and put them in a dictionary.
There are other techniques that go much further than just checking the "score" of a message based on what keywords show up in it. There are some techniques that try to parse the message for it's grammatical structure and the "intent" of the message. These are much more accurate techniques that what is essentially glorified keyword filtering (Bayesian).
Someone is WRONG on the Internet!
Oh how I wish people would differentiate between GA(genetic alogirthms) and ANN(artificial neural networks)!!!!!!!!!!! GA is chromosomes, crossover and all that jazz, ANN is neurons, back-propogation et al. Combining the 2 is like the unified theory of everything (talk to mr. Hawking if you don't believe me)!!! </rant mode> To my mind, the article makes an interesting point though :-
Catogorizing features of an email which can be passed to a classifier.
However, looking at it from a human standpoint (one which spam is aimed at), should the only feature of email we are interested in be that of the actual text which appears in our client????
Too often spam classifiers get clouded with hidden text, number of links etc.
Do you make your classification of spam in those terms????
Me neither!
Perhaps a spam classifier which looks at the context of a message might be more successful!!!!
Just a point :-D
Here.
Thankyou for working on spam assassin.
You dumb souless bastard. Ever hear of 7AM mass. Guess not, you godless fool.
Spam is good, spam is nice, we can't get rid of it, that way is doomed to failure. That is the wrong problem to attack.
The correct way to attack it, is to attach a price to spam that is directed at people that don't want it. Thankfully, this is easy enough. If the cable Internet companies, the DSL companies and AOL/Hotmail/Yahoo got together and just built a database of IP addresses that send spam to them, and charge $5 for each instance of spam sent for that entry to be removed from the database, and hard IP filtered all traffic from every IP in the database, in seconds, the spam problem would be solved. Since the filtering would be done immediately, there would be a very low possibility for more than a few such emails to be sent from any address that spams. Because there could only be a few entries in the database, a thoughtful organization that accidentally gets added, can, in real time, pay their `fine' for spam, and be educated by it, total cost $5-$1000. Before they hit the `pay now' button, they would _probably_ learn to take care of the _problem_. If not, they can learn again, and again, only limited by the depth of their pockets. Since spammers can't pay out $5 for every spam messsage they send, they could not continue sending email to people that don't want it. In time, measured in days, every one would know if they can send email to the rest of the planet or not, and if they can't and they want to, they can then switch from the spamful company to one that is responsible.
Why must this work, because with a 5 tillion dollar a year spam load, someone would either have to pay 5 trillion a year, or the problem would be gone, I'd wager, the problem of spam would be gone. By spreading the costs around to those organizations that engage in sending spam, we distribute the solution to those organizations. Inovative companies that solve the problem, get rewarded, those that don't, well, they can't send email.
In all companies sent an equal amount of spam, they could cross settle for a net 0 cost, no matter the innocent spam load. If one of them allowed an abnormally high spam load, they would face rising costs. They have only two choices, pay the fines, or stop sending email. Hosting companies would have to classes of IP addresses for lease, those that can't send email, and those that can. The ones that can, may be more expensive, and may entail the hosting provider paying fines and charging the customer to keep the address clean. If the customer can't pay, they, they quickly migrate the user to the can't send email address, and pay the remaining fines on the good IP address, take the hit, and raise prices to keep the clean IP address clean. The hosting providers that don't care, quickly will find they will accululate IP addresses that can't send email, in time, none of their address will be able to send email, except for those of long term, good customers, their address will be clean and remain that way.
The database would be open and free for all to use. It would be best if large numbers of organizations agreed in advance to use the solution, and to contribute to the database. The reason to contribute, $5 for every entry contributed, we reward people, organizations that get spam, think of it as a pay to read. If you read 100 spams a day, you make $500 a day in spam, at that rate, many could quit their day jobs.
To counter forgeries, we charge $1000 for each faked entry. At this rate, few would dare fake large numbers of such entries, and the rare few that tried, would soon find themsolves shut out of contributing to the database.
How many times does somehow have to remind a slashdotter about this shortcoming of freecache?
Yet somehow, after being mentioned in an article ONCE, freecache is the darling INCORRECT answer for every slashdotting-related problem?
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Wow, I never thought this would generate this much discussion or attention! To everyone that's giving me good feedback (Henry Stern especially), thank you very much as it is appreciated... now for the detractors and script kiddies.
Ok, first off, I was not de-railing existing spam filters, so you guys that took it that I was personally attacking your spam filter, get over yourselves. If you don't have a problem with spam, then congratulations, you get a freakin' cookie. Since most users do, I was trying to create a forum for discussion on new techniques and ideologies for addressing it, not lameass quotes like "this guy's an idiot". And, most of those comments were because people didn't read all the article or didn't understand it. I guess that's what happens when you mix script kiddies with democracy.
Next, if other people have tried and tested this method, great, I wasn't trying to steal their work. This is basically a guy, a garage, and a computer type of project, and I don't have access to research done in deep academic circles. If this overlaps with other work, my deepest apologies to those individuals.
Now, for you people that thought I was trying to say this was the next best thing to canned bread, give me a break! If you read the conclusion, I basically state that no one simple algorithm can adequately address spam (this one included!). That it would take a multi-pronged attack, and that we should start treating spam the same way we treat evolving organisms. That was the point of the article. For the algorithm, I was just trying to present a starting point, not an end.
Whew, now that I've got that off my chest, to the other problems with the article.
Henry, you were absolutely right, the training set does need to be randomly shuffled each time per epoch. My fault on that one guys.
Also, I do apologize about the metaphors, I should have used the proper ANN terminology. I used the metaphors to re-enforce my point about the similarities between this and biology.
Now, I am not quite sure why it takes so many training cycles to train the network in SpamAssasin, but since I'm using standard BP, it is quite fast, and does not require an extensive training set because of the generalization traits of a MLP.
I do agree about the ability of spammers to exploit the error inherent within the hidden layer of a MLP. But, I would think it would be extremely difficult to do this, especially if multiple distributions are created of the trained MLP, each with different structures within the hidden layer and with different initial weights. Then, they would only be able to take advantage of certain traits for a small percentage of filters. Also, I would argue that training should also happen on the users PC, as a background task, say weekly. This allows the network to adapt to new types of spam, and when initialized with random weights, it should make the error within the hidden layer random among differnt installations, effectively killing that exploit, as each machine will converge differently.
Now, regarding spam messages adapting, I am sure their are many more exploits they can implore, but as of this point, they are running out of options, atleast "structual" options. If we can work together to identify the exploitable structures, then we should be able to design a filter that can adapt to the different features within these structures.
That's all I have, thanks again to the folks who gave me constructive feedback on the article. To those who's remarks were out of complete ignorance, **** off.
Thanks SlashDot!
Shawn Evans
It'd be bitching if we (or someone) could set up a sort of website or service whereby suspected spam links could be collected and analyzed for trends.
Perhaps webhosts could be identified as being problematic... and contacted. Or maybe it might lead one to a compromised ISP or residential net.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Gee, how original....
"But even with these enhancements, it's been conclusively shown that some problems are intractable for neural networks. In any case, neural networks are no new thing."
Not so. Maybe you're still thinking about extremely simple neural nets, because no such proof of intractability exists for larger more complex networks.
Here's proof: Neural Networks can emulate a Universal Turing Machine. Since they can also be emulated by a UTM their limitations are no greater or less than those of any UTM. One citation if this isn't obviously true.
This is exactly why Marvin Minsky has been accused of slandering neural nets unfairly, and hindering AI research. In his book _Perceptrons_ he demonstrated a simple problem that a trivial (one or two layers with no feedback) NN can't solve. A lot of scientists wrote off Neural Nets just as you have, because a toy was the only tool used. Never mind the fact that an only slightly more complex NN can solve such a problem easily. I find it telling that for a human to solve the same problem, one has to construct a strategy to do it. Not the sort of thing I'd assume any extremely simple machine could do. These days Minsky complains that AI isn't trying to build human brains. He's a brilliant man, but in some cases (as with many famous people) his chutzpah occasionally outstrips his judgement. I only wish that great scientists were immune to this.
Lots of less qualified people complain that neural nets aren't useful because they have some unpleasant experience with them. They have no idea of the variety of neural nets. It's like using a Playstation and complaining that computers are not useful.
As for spam filtering with AI, unless you have the narrow definition of AI, the Bayesian techniques of SpamAssassin are AI, as is the Latent Semantic Analysis done by OSX mail.app for spam filtering. LSA, while computationally expensive on a PC, is regarded as equivalent to a particular type of 3 layer neural net, (see Kohonen self-organizing maps.)
One thing you have right. Neural nets are "no new thing." They're as old as biological brains. Novelty is not a criterion for usefulness.
Assembly is the reverse of disassembly.
Ahhh, thats interesting, thx. that sheds some light on ms's business strategies:
Step One: Sell user Overpriced OS advertising "so you can host your own Web site on the Internet"
;-)
Step Two: Profit!
Step Three: Post story from user's internet page to slashdot.
Step Four: User upgrades to server version.
Step Five: More Profit!
Code is Speech. No to Censorship.
:-) I hadn't seen that. On the next page, though, they own up to what you're actually getting:
It could also filter out 133t speak.
I had to love the comment about ratio of misspelled to corectly spelled words. As one of the worst spellers in the world I fear for my future emails. I would also worry about highly technical emails getting flagged. Spell checkers think any word they do not know is misspelled.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Pronounced as in "goto crutch already".
How Freudian.
The spammers, by necessity must disguise the text to elude simple Outlook filters. By doing this, they have introduced more accurate, yet harder to detect, indicators that the message is spam.
By understanding the tricks used by spammer, and using "real intelligence" to detect the tricks, it becomes possible to accurately detect spam without relying on training and other burdensome processes.
Blue Squirrel's anti-spam products detect spam using the text as well as detecting the tricks used to disguise the text.
Almost all of the methods for detecting spam are included.
- Whitelist (by e-mail)
- Blacklist (by e-mail)
- Blacklist - RBL (by IP)
- Dictionary - detects tricks used to throw off Bayesian analyzers
- Bayesian Analyzer - Trainable
- Challenge/Response for false positives
- EMail Stamps - Give spammers the option to pay
- Bouncer - Trick spammers into taking you off their list
- Good Words - Make sure you get messages you are interested in
- Anti-Virus Detection and removal
- Script removal
- Dangerous attachment removal
- Detection and removal of Web Bugs
- HTML "loudness" detection
- SPF / MS Caller ID
- Reply possible detection
There is more, and new techniques are added as they are shown to be helpful. Weighting of each of the techniques is simple. Administrators can keep control centralized, or give each user control over their own account and its settings.There is an SDK available upon request for plug-in analysis.
Nothing is hidden. Every message gets a report. A web interface lets users see the quarantined messages and a detailed report on why it was not allowed through.
It learns the valid users. Or, the program links into LDAP or Active Directory, or RADIUS (for ISPs), or allows the import of users and automatic generation of passwords.
It may be the most complete anti-spam system developed to date. It does not rely on one technique or method, but rather combines them all, and then allows for new techniques.