Slashdot Mirror
Paul Graham on Fighting Spam
Ramakrishnan M writes "Paul Graham, the Lisp Guru is back with a great technique to fight spam. It is based on trust matric, and he claims, only 5 out of 1000 spams got leaked out of this system with 0 false positives. Worth looking at."
I propose we define spam as unsolicited automated email. This definition thus includes some email that many legal definitions of spam don't. Legal definitions of spam, influenced presumably by lobbyists, tend to exclude mail sent by companies that have an "existing relationship" with the recipient.
This needs to happen, just because I buy a book from a company doesn't mean I want their stupid monthly mailing list.
This seems very similar to Spamassassin, which alot of us are using with great success.
I got an email last night about this! Also, it asked me to help out his Nigerian cousin...
of course! it sounds so obvious now.
jeez, that alone would cut down on spam, cross reference that with my trusted address book, and I'll probably be ably to filter all spam.
I have that feeling you get when you've been stuck with a problem, and some guy looks at the code for about 2 seconds and finds a problem.
The Kruger Dunning explains most post on
(Yeah, yeah, I know...)
But if you do, check out Cloudmark's SpamNet. I've been quite please with it's ability to stop spam, and it gets better the more people that use it.
1) Lisp...ever since i ran into scheme, I have _loved_ the concept of lisp based languages. A nice Hoo-ha to anyone who says there are no practical applications of lisp based languages. (except haskell...which personally, i think sucks! if one of our own professors hadn't invented it, it would be dead by now) 2) _0_ false positives. I'm perfectly happy to settle with "some small number of spams getting through" given there are NO false positives. Early on in the article he states that he realizes this is a critical problem, and from the start keeps no false positives as a goal. It is far better to have no false positives then to have 100% no-spam rate with that in mind... 3) the statistical word analysis is really interesting..."describe" is innocent. unfortunately....what happens when a few smart spammers get their hands on this analysis *sigh*
When in doubt, parenthesize. At the very least it will let some poor schmuck bounce on the % key in vi. (Larry Wall)
Create an E-Mail address called, say, spam@example.net.
Put a link to it on your website, but tell people not to use it for anything, E.G.
<a href="mailto:spam@example.net">Spam trap - don't use me</a>
Then, it'll get harvested along with all the others on your site. That mail box will fill up with spam, and nothing else.
What good is that? Well, you've got a ready-made list of messages to filter *out* of your other mail boxes!
So, just write a script that checks each inbound E-Mail against the spam list. If it matches, you *know* it's either:
1. Spam
or
2. An E-Mail that somebody has also sent to the "Don't use me" address.
In either case, you don't want to read it, so it gets auto-deleted. Nice.
Oh, I think I'll patent this, and not tell any of you about the royalty I'm going to charge in 15 years time. Hahahahahahaha!!!
Oh, by the way, first post, first post... NOT!
that means CmdrTaco reduces his spam intake to around 500/day.
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
Sadly once the spammer knows this method's being used, he'll start chucking in obscure (but valid) words... ah well, maybe at least spanm will start getting interesting to read, assuming the spammer tries to use the word in context.
"Buy my superlatively efficacious mail list."
Maybe not...
Tom
Oh arse
"if you outlaw spam, the only people with spam are outlaws..." er something.
...and I use the term 'business' loosely.
anyways, what I was going to say is ok, US outlaws spam. now what? sue korea as a whole? how about china? nigera?
laws don't mean shit.
you need to go after the people making MONEY off spam, not the spammers. Most of them are US "businesses".
Looking for Book Reviews? Check out Literary Escapism.
BUT, now, the best spam filters out there already use statistical properties. Spamassassin does this, for example, and it works *extremely* well. Before I found Spamassassin, I had a huge procmial recipe that used it's scoring mechanism to do basically the same thing -- but of course spamassassin does it better, so I switched :)
From the article:
.97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability. And Bayes' Rule, equally unambiguous, says that an email containing both words would, in the (unlikely) absence of any other evidence, have a 99.97% chance of being a spam.
/dev/null immediately without as much as a second glance... :-)
Based on my corpus, "sex" indicates a
Hmm.... take an average adult geek and yes, an email mentioning sex or sexy can go to
On the other hand if you run the statistics on email of an average horny teenager, the probabilities might get a bit different.
Kaa
Kaa's Law: In any sufficiently large group of people most are idiots.
"But the real advantage of the Bayesian approach, of course, is that you know what you're measuring. Feature-recognizing filters like SpamAssassin assign a spam "score" to email. The Bayesian approach assigns an actual probability. The problem with a "score" is that no one knows what it means. The user doesn't know what it means, but worse still, neither does the developer of the filter. How many points should an email get for having the word "sex" in it? A probability can of course be mistaken, but there is little ambiguity about what it means, or how evidence should be combined to calculate it. Based on my corpus, "sex" indicates a .97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability. And Bayes' Rule, equally unambiguous, says that an email containing both words would, in the (unlikely) absence of any other evidence, have a 99.97% chance of being a spam."
Tom.
Oh arse
Here's how: the spam should be written as a 'multipart/alternative' with an html version of the spam as the primary alternate. The text version contains an innocuous message intended to pass the statistical spam filter. The spam message is entirely contained as an /image/ within the html. The text of the spam becomes invisible to the reader but not to the poor schmuck who gets the email.
I'm guessing here that the inclusion of a single image tag in the html is unlikely to trigger the spam filter, and supplying a wealth of evidence that the email is 'not' spam in the unseen alternate text will let the letter through.
Spam is wrong, but so's murder. That doesn't stop it from happening.
We should pursue legal avenues for stopping spam, but that doesn't mean we shouldn't try to block it in the meantime! The article sounds like a phenomenal way of blocking spam.
what his spam filter would make of his article?
-- MartinG To mail me: echo kewyjlcxyzvjfxbqwh | tr bcefhjklqvwxyz
Tom.
Oh arse
Spam, like these things, is going to be extremely difficult to enforce. Laws or no laws, filters will be necessary.
Comment removed based on user account deletion
This is the brilliant part, and crucial to the endeavour, and so easy to implement!
It appears all the nay-sayers here haven't even read the article (no surprise). With as little code as needed to implement this it should be a must in the next mozilla mail/pine etc. code base.
only infrmatn esentil to understandn mst b tranmitd
Having had the same email address since '93, I receive close to 1000 spams per day to my personal account (which is also aliased from root/postmaster/webmaster).
/dev/null.
I've tried everything under the planet to reduce the amount that I see in my mailbox; SpamAssassin being one of the best so far. But even that lets through quite a bit (around 10%).
So I decided to attack it from a different angle. I wrote a series of perl-scripts that I plunked into my procmail file.
The scripts work by checking the address of the sender each time a message is received. That address is looked up in a database. If it exists in the db, and it's marked as "authorized", it's just passed into my mailbox.
If it's marked as denied,
If it's never been seen before, an authentication message is sent to the sender asking them to reply to it to authorize themselves. If that authmessage is bounced back, a db entry is made as "denied".
If it's replied to in a normal fashion, that email is marked as "authorized" and any queued up mail from that person is pushed out.
The concept is that spam will almost never have a valid reply-to; so it will bounce and be marked as denied.
Even if the email doesn't bounce, no spammer alive will reply to it; so after 30 days, that email is marked as "denied".
Since I've set this up (for myself and my 10-year-old son who receives porn in his box (grrr!!!!)), it has worked flawlessly. The "real" email is unharmed, while the spam is stopped.
Oh, and I have a web-based control page so that users can manually add email addresses (for lists and such).
This week, for the first time in YEARS, I don't have spam in my mailbox anymore.
Hurray!
No if I can only stop those damned dictionary-based scanning of my servers, I'll be set. Thank the gods that I don't have metered service.
Given that much of my spam is not only /from/ Korea, but /in/ Korean, a considerable amount likely comes from Korean businesses.
/all/ telcommunications from Korea until they develop some responsible marketing laws and enforce them (with, say, a 90-day notice in advance).
As for what to do? One heavy-handed bit of leverage would be to block
Only the dead have seen the end of war.
I guess you never wish to converse with a blind person, or someone who's restricted to a text only medium then?
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. (Einstein)
He isn't fighting spam, he is filtering it. There is a difference. Filtering still costs in bandwidth. Fighting it would eliminate the source and free up the gigabytes of bandwidth lost for this marketing purpose.
Filtering is fine for now, but ultimately it must be fought and defeated.
--------
It's OK to be social, just don't tell anyone about it.
Using Graham's system, write a message that will get a very high mark. The highest mark will win.
The message has to be understandable English. Please post your entry as a reply to this message.
Trollem mirabilem hanc subnotationis exigiutas non caperet
Great... now that they know, they'll spam me with gifs and jpeg.
All humans are mortal. Socrates is a human. Socrates is dead.
I think that spam is a necassary evil that can be easily controlled. If we make a law to simply ban spam then we might be banning other things like mail lists. I personally recieve NO SPAM in my main account and less than one piece a day in my "junk mail account." That's inluding things that the spam filter catches. All people have to do is to be careful with their e-mail addresses. Spam is not a problem for people who use a modicum of common sense
Let me tell you, the longer you've been online the more likely you are to get this shite. Remember, it only takes ONE posting of your mail address to a newsgroup (which in my case could have been years ago) and that's it. Then of course you end up on one of these "1 BILION fresh email addresses for $100" lists and you're dead meat.
Matt
Can I use that feature for my own (commercial
or open source) mail client development?
Although to be honest, I don't understand how the algorithm works. However I'm sure some enterprising soul can probably work it out and code something (hell I will if someone can explain it in decent mathematical terms).
All we need then is a repository of spam mail and non-spam mail to "teach it".
Whatcha reckon?
Avantslash - View Slashdot cleanly on your mobile phone.
I'm continually amazed at the people who are beating their heads up against a very simple problem. The answer is not statistics, it is not heuristics, it is not AI, it is not procmail.
The answer is verification...aka whitelists. Check out TMDA, tmda.sourceforge.net. This program assumes you don't want mail from anybody whom you haven't explicitly allowed, or who has verified that they are a real person and not a spammer.
Verification is simple, and some people will point out that it could be defeated by a spammer. But, the economics of spam do not make it feasible for a spammer to attempt to defeat TMDA.
TMDA is similar to making your phone number private. You only get phone calls from people you have given your number to, and you never get telemarketers.
TMDA user since December 2001. Spam messages that tried to get in, 12,133, spam messages that got in 3, false positives, 0. Time I've spent tweaking and modifying the program since installation, 0 minutes.
This is a very bad idea. What about companies such as Hyundai that have Korean and American (and many other countries) divisions? Or, what about my friends from Korea trying to e-mail their family back home - should they be hurt because some companies in their home country do bad things (and/or it's government doesn't have/enforce laws to stop them)? Name a country that doesn't another country/ies thinking that they need to 'change how they do things over there.'
"Giving money and power to government is like giving whiskey and car keys to teenage boys" P. J. O'Rourke
a nice idea to filter spam ...another one to fight it.
1. the MTA's (mail transport agents like sendmail etc) establish trust relationships between themselves or manually. They also maintain a users safelist (i.e. addressboook + list of addresses user wants to recv mail from)
2. All email over the trusted links and from addresses in the safelist are delivered unfiltered.
3. For each email sent over an untrusted link
a. Perform MD5 over message body.
b. Ask neighbouring trusted agents if they have received an email whose MD5 is given.
c. If no. of positives are greather than a threshold, reject as spam.
You could develop a corpus of spam over a long period of time, and look for shifts in the data. What this paper describes is distinguishing between a spam-corpus and a legit-corpus, but you could also compare a spam-1999 corpus to a spam-2002 corpus, and see if the spammers are up to anything new.
Not that it would be useful, but it might be kind of cool to try it out and see.
It's not wasting time, I'm educating myself.
However to test such an idea I need a repository of spam mail - something I don't have. Hotmail junk is no good, it's just the same old adverts regurgitated over and over again.
Does anyone have anything like the 4000 junk emails that this guy has? If so, please could you pop me an email to org dot ewtoo at silver as I'd really appreciate it!
Avantslash - View Slashdot cleanly on your mobile phone.
From the article:
.97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability...an email containing both words would have a 99.97% chance of being a spam.
In the spam filtering business, false positives are your biggest worry...Based on my corpus, "sex" indicates a
False positives could be a HUGE problem in this case...imagine the agony if you missed this email from your wife: "I'm feeling REALLY sexy today - meet me at the motel off 12th street at noon for some lunch-hour sex!"
In this case, the damage to others /is/ the point, just as that's the same logic behind the Usenet Death Penalty. Hurt others (in the case of a UDP, the customers of the ISP who send perfectly legitimate email) whom the authorities do care about so that they change their policies...
It's not particularly nice, or even remotely fair, but something like that might work. A large-scale boycott by major ISPs might do the trick.
Only the dead have seen the end of war.
The theory (as I understand it) is that there are enough "legit words" in the "Sexy email to your gf" (i.e. her/your name/nickname, her/your email addy etc) that they'd cancel out the "bad words"
The big shift in thinking from looking for phrases vs scoring each and every word in an email is that the rest of the email is just as saving/damning as the stuff that filters look for.
Ok, I read the article but quickly, and at the end of it I wasn't sure how he ultimately told the system that an individual e-mail was spam or that it was legitimate, so it would know into which bin to toss those words...is that a manual process?
I set up a homebrew whitelist (which still shows me the potential spam) I'm pretty happy with. I'm trying to figure out if I should keep in the subject based whitelisting or not...some spammers use my typical "hey" or "hi" subjects now...and it's the part of the system that grows the most. I'm just worried I'll send out mail to someone and they'll reply with a different e-mail address...maybe I should expire subjects?
Hmmm.
SO YOU'RE GOING TO DIE: The Comic for Dealing with Death
I wonder what Bernard Shifman would make of this article?
What is our 'CS Consultant' up to these days?
Are you local? There's nothing for you here!
it looks great, and i will try it for my account that i use eudora or outlook for...however, i use a hotmail address for my main account (so it can travel wherever with me), and their custom filtering system sucks (if i may say so)...the only things they let you filter on are subject, From Name, From Addr, & To or CC lines...no option to filter on message content, which is where this would be useful...oh well, i guess that's what i get for using hotmail...i should get a real e-mail account...
"Facts are meaningless. You could use facts to prove anything that's even remotely true." - Homer Simpson
Making spam illegal would probably cut down on people buying email lists and starting to spam in their free time because it seems like a great way to make some money. It might even cut down on the "legitimate businessmen" types here who do it professionally. It's going to have no effect internationally, however, and there's really not much you can do about it.
There's an interesting point about this in the article, however, when graham says:
I would agree with this - it seems to me that for a lot of "crimes of this nature, drugs being the best example, the solution is not criminalization but regulation. People aren't going to stop dealing or using drugs, nor is it something as serious (like murder) that it's worth it to put them in jail anyway. If drugs were regulated, however, most of the problems could be easily reduced. Enforce strict controls to prevent cutting, ban advertisement, and tie sellers to treatment programs to help get people off of drugs. As long as there's no incentive for people to buy them illegally (ie, their being much cheaper or, as it is now, the only supply), people will buy them from regulated sellers.
Similarly if you regulate spam and make people attach footers you'll be less likely to drive people overseas to spam while also making it much easier to filter out.
Of course, there's still not much you can do about the Koreans, other than trying to get their government to do the same thing.
Besides, do you really want to encourage the government to effectively prohibit certain kinds of non-victimizing (non-kiddie porn) speech online?
It was with the help of spam that with just a simple herbal supplement I was able to add three inches to my penis (an increase of over 20%). I had assumed it was just a scam, and nobody was more suprised than me that it worked.
Well, except my wife.
Phallic Symbols in LOTR
Good method. I work with Bayesian technics often and I had thought of the same thing but for a different purpose: automatic classification of emails. When you receive an email, your mail reader would propose a list of potential folders into which you might want to put your email after (or before) having read it. And the best thing is that is learns with time and it gets better. And as this article shows, this method can also automatically filter emails. Now if I have time to get involved in the Evolution project or kmail, ...
I'll do it for cheesy poofs.
Feel free to review the work at http://research.microsoft.com/~horvitz/junkfilter. htm
They came up with similar processes to both filter and to categorize. Bayesian analysis is a very flexible, and while Paul Graham is not the first to try this, his work looks very exciting.
I had nothing to do with any of this work; just a fan of Bayesian research.
Michael
I spent about six months writing software that looked for individual spam features before I tried the statistical approach...[cut]...Based on my corpus, "sex" indicates a .97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability.
ofcourse these probabilities may vary from person to person.
Based on my corpus, "sex" indicates a .97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability. And Bayes' Rule, equally unambiguous, says that an email containing both words would, in the (unlikely) absence of any other evidence, have a 99.97% chance of being a spam.
Obviously, the author just isn't sexy.
m00.
Look in UseNet. The group news.admin.net-abuse.sightings is where people post their spams. Enjoy!
No replies made to AC posts. Please log in.
1. Create layout of spam
2. Take a screenshot
3. Convert to low res PNG or JPG
4. Mail the JPG to 100,000 annoyed geeks
5. ???
6. Profit
This message brought to you by the Council of People Who Are Sick of Seeing More People.
"Technique which utilizes a probabilistic classifier to detect "junk" e-mail by automatically updating a training and re-training the classifier based on the updated training set"
The full details of the patent can be seen here.
Patent Link
I'm surprised you guys don't check at the patent office first before you get all excited about a new idea. Doh!
The bikini - security through obscurity since 1943
The latest trick from spammers is sending out HTML e-mails with their ads. Not a problem by itself, but by embedding the entire spam ad as a single GIF or JPEG image, there's no text for the spambot to filter out. It's easy to trap false positives with this, too, since a family member or friend might want to send out photos without necessarily attaching text as well. Boom, statistical analysis is instantly useless, and we have to go back to the old tricks -- filtering out known spam e-mail and domain sources.
I actually had to close down my hotmail account; the spam would exceed the 2MB within 24 hours after being cleaned (and that's with the wonderful MS spam filter set on "high.")
BTW, these days I'm getting individual spams that are 170 KB in size. Talk about rude...
Phallic Symbols in LOTR
Did you happen to read the article? He discusses this at length. He makes a strong argument that his system is actually pretty robust, since to get around it consistantly the spam has to look just like your real email, which is pretty darn hard for them to do.
In a lot of ways this problem is like cheating in games. As long as you're the only one who knows the exploit, you can be pretty sure that it's not going to get fixed, though you'll still get kicked off every server you play on. Similarly, with his method a spammer might be able to find a particular phrasing that's likely to get through, though his messages will still be deleted on arrival. But even if he does, if he starts sending you too many emails or starts selling his technique the filter will adapt with the spam and start filtering it out.
I think rather than lump stuff into "spam" and "non-spam", it should assign a ranking number and preferrably display a color to represent ranking.
If you are tired, then you can ignore or defer the grey areas.
Also, if the display list displayed the first X characters from the content, then one can often check without reading the whole thing. (Perhaps filter out non-indicative words like "the" and "and" to make it more compact to display.)
I don't think there is one magic technique because spammers will work around it if it gets popular. Thus, a combination of machine and human working together will be more effective IMO.
Table-ized A.I.
Actually, I'd recommend a combination between a nasty spam filter that kills off close to anything that might conceivably be spam and white-lists of senders who are automatically cleared. Your friends mails can get through, but woe betide the remote aquaintance or casual relation who mails you anything about sex... on the other hand you might be better off without that anyway.
>> based on the assumption that it is written in English
There's no reason to think that Spanish, German, or even Chinese spam doesn't follow the same statistical word frequency rules.
>> following the simple steps outlined in the URL above
What if you are subscribed to mailing lists, or have mail bots that send you useful messages (like "your server is down")? The usual answer is "just configure those in advance" but that's a pain and not very robust. My hosting company was bought out and their automated server status messages just started to come from a new domain. If I had this kind of filter I would have missed them.
What I want to know is:
Would this also work with email virus? I think it would since the virus would also have a defined patern to it and the program would pick it up after the first one.
Could this be made part of the STMP protocol or built into the backbone layer of the network? Again, I no major reason why it couldn't.
Problems that I have with it are:
Since each word is treated as a token and everything else is not, I'm sure that spammer would quickly figure out that a spam like this just might work:
<HTML>
<BODY>
Enlarge <!-- elephant --> penis [etc..]
</BODY>
</HTML>
which would show the message but hide the balancing words, so it could be possible to change the delta into your favor.
Does anyone else have thoughts on how this might be broken?
III.IIVIVIXIIVIVIIIVVIIIIXVIIIXIIIIIIIIVIIIIVVIII
SMTP is designed broken because it:
1) Allows senders to be faked.
2) Is slow.
3) Requires bounces for broken messages.
4) Allows loops.
5) Cross-subscription to mailing lists, complicated mailing list management.
6) MIME.
7) Add your gripe here.
See http://cr.yp.to/im2000.html
Laws will never stop spammers. The damages are very hard to prove, especially when the judge/jury don't realize that their ISP filters their mail for 95+% of the spam already. Most people just don't GET it. And most spammers are sending the spam from another country, running a fly-by-night operation, so prosecution is nearly impossible.
Filters are helpful, but they still require huge resources to receive the e-mail and process it. And as stated in the article, the risk of a false positive is often much worse than just receiving the spam.
There are already only a few mail relays that are willing to send out spam, and virtually nobody accepts ANY mail from them. The spam going out is coming through illegally used mail servers. This shows what is to be the solution to the problem of spam: ISPs will only act to stop spam when the spammer is damaging their system.
Most spam gets deleted without the enclosed links getting clicked by at least 99%. The company hosting the web site just sees their customer getting some success with their business. They don't know why, and they really don't believe/care when someone e-mails them to say that the user spammed them from a mail relay in china. The user probably paid for a 2 gig/month of traffic, and they are well under quota.
It's time to change that. With a SETI@Home / Prime95 type application, we could easily DDoS a daily spammer off the net. Slashdot alone could easily field 10000 users willing to put their cable modems up to the task of pounding spammers accounts (and possibly the hosting ISP) off the net. Beat them down until the account appears to be deleted. Maybe then ISPs would hold users accountable for being spammers. Web hosting contracts might start including fines ($500+) for abusing the service, rather than just the scary risk of a cancelled account. All we have to do is beat them down before the few clueless morons come buying and make it worth their while.
Legal? Sure, I don't see why not. I can send a 10 http requests to the ISP in a second... I've never heard of a law that says I can't do that every second. As long as the computers involved are from willing users (sysadmins get permission in writing first), there is no 'hacking'. Every DDoS case I've ever heard of involved charges of 2k+ computers 'hacked', rather than the ensuing attack. Even if it is illegal, this is vigilantism that nobody (other than the hosting ISP) is going to complain about.
This reasoning is statistically invalid. It is only true if the chance of the word "sexy" appearing in a message is independent of the chance of the word "sex" appearing. In other words, only if knowing that the word "sex" appears tells you nothing about how likely the word "sexy" is to appear, can you reason as he is doing above. That's probably a very poor assumption in this case.
He is doing:
The correct formula is: where the last term means the probably of "sexy" given that "sex" appears.Maybe his approach is good enough for his purposes, but the statistical foundations are not correct.
Your filter's usefulness is inversely proportional to the number of people who use it, since it is trivial to bypass by a spammer who knows its details.
Can you imagine the day everyone uses this. You send mail to a public list and get back 2000 messages asking you to "authenticate" yourself.
This is a bad plan for working in the large.
Clippy/BOB/etc were based on Bayesian techniques, right? Does this mean M$ could soon build this into Exchange/Outlook?
dundunDUNNNN
[o]_O
By the time one can apply the filters, you have already received the spam. This is a load on your resources. In some cases your in-box may even fill up (yes, I've received 1000's of the same piece of spam in the same hour, exceeding the capacity of my allotted storage and effectively DOSing me from real e-mail) or you may exceed limitations from forwarding services.
The spammers don't really care. Or notice. Their goal is to hit millions of victims, knowing that some of them will respond. The response is all they care about. Filter your e-mail all you want, you were not going to respond to them anyway. All they care about is reaching the mark that doesn't know any better, and this filter doesn't do anything to stop that (unless it is applied automatically by ISP's, unlikely due to the fear of fales positives).
What might help is a two fold attack on what they want: responses from marks. I suggest the following:
A massive education campaign to educate the general Internet user to never respond to (or even read) strange messages that show up in your e-mail. Banner ads would seem a good place to start, it would be a public service if a good percentage of banners were replaced with ones that educated the Internet users who still make spam profitable. This might even have the long term effect of improving banner revenue: if banners compete with spam as a way to get out a message they have a lower value than if the public is taught to not buy from spam and even to aggressively resist doing business with a spammer. In the long run an antispam banner campaign could improve banner revenue for those who help fight spam. Ideally another great way to get the word out would be UCE, but that poses a moral dilemma....
The other thing that could effect the spammer is if the ads are not getting the desired results with the advertisers. What needs to happen here isn't filtering, it's massive negative response to the advertiser. No response don't hurt them, but making them respond themselves to unwanted responses is a more suitable way to respond to those who originate unwanted messages to use in the first place. These people need to get responses that waste their time and resources like they are wasting ours. Obviously those who supply 800 numbers are a prime target for this, while those who supply only postal addresses make it too costly to respond. I think such negative response campaigns need to be coordinated from major popular sites to be truly effective (not just from a few geeks who spend their day on an anti-spam website. Their efforts are much better applied by getting the spam sources in black holes and getting ISP's to block or filter spam). It sure would be nice to see the slashdot effect applied to spammers rather than the poor smuck who puts up a small but interesting website.
Interested in other's thoughts in this area.
I'm an American. I love this country and the freedoms that we used to have.
Just out of curiosity, what if a bunch of geeks set up servers to DOS-flood sites that spammed. (This would not be the return address, since those are usually phony, but the website that sells the goods being advertized.)
If such was possible, then Viagra.com would think twice about starting another spam compaign.
Table-ized A.I.
The big difference is that we can SEE where spam comes from. It's in our log files. Heroin is all underground. Music is not email. We also have laws against copyright infringment and yet the labels have not gone after traders. Believe me, if there were good tough, well-written NATIONAL (not state) laws on spam, I would go after EVERY SINGLE SPAMMER. It would be a GREAT source of additional income.
I don't know why you say that spam laws would be difficult to enforce. We have logs, the illegal mail (spam), and the target phone numbers / web sites (the spamvertized material.) It's pretty cut and dried. If the DOJ get's a chunk of the fine, and the spammie gets "restitution", it would be a self-funded program.
I have ZERO problems instituting a "usenet death penalty" type block on coutries that don't have tough laws on spam. I already do so personally on my servers for about 30 countries. Emailers in those countries get rejected with a pointer to a web page that tells them whats going on, and how to get "whitelisted" if they are legit.
I have no problems with having filters as well as laws, but we need the laws to reduce the bandwidth bills. Spam takes Massive bandwidth and a toll on server CPU. Not too long ago, spam took AT&T's email servers for worldnet down for 3 days. This kind of thing HAS to stop. Filters at the recieving end won't stop the bandwidth usage of spam. Without stopping spam at the source, the problem will just get worse.
Freedom of speech is not the freedom to tresspass on my computer equiptment, use my resources for me to listen to your advertising!
This is not a prohibition on your paying your moneyto spread your advertising. This is a prohibition on you spending my money to spread your advertising.
Commercial speech does have some constitutional protection, but not to the same level as non-commercial speech. But even with pure political speech, there is no requirement for me to pay for your speech.
As for hitting the delete key, at that point, you have already tied up at least 2 of my computers used my disk storage, my time, my bandwidth without paying for it.
If you want to spam, no problem, just pay me in advance.
Fight Spammers!
This is the best paragraph of the whole article:
So as spammers start using "c0ck" instead of "cock" to evade simple-minded spam filters based on individual words, Bayesian filters automatically notice. Indeed, "c0ck" is far more damning evidence than "cock", and Bayesian filters know precisely how much more.
The Bayesian filter. You can run, but you can't hide!
+1 Insightful, -1 Troll. What can I say, I'm an Insightful Troll.
In addition to filters being individually tuned, the system allows for "whitelists" - Any mail address on the user's whitelist automagically bypasses the filter.
The difference between this and other whitelist approaches is that "new" people who are sending you legit mail (Like Horny Teenager's latest BF/GF) will likely get through, as opposed to having to authenticate in some manner.
retrorocket.o not found, launch anyway?
I had my first spam before I received my e-mailed copy of the journal. It was "related" to the topic of the journal, and said something like "I sure agree with what you wrote about in the journal. What's your opinion about http://my.url ?" But the To: line was the clue. It included not only me@myrealaddress.com but also that of smart.guy@nospam.address.com (another poster in the journal.) It was very apparent that the author had simply harvested the HTML and dropped it into his address database.
It was only hours before I was getting offers for detoxifying myself, HGH, climax gel and all-free teen pr0n.
John
A few quick comments about this. Although powerful, such approaches suffer from being somewhat too 'black-box'. That is, you turn control over to the computer to make decisions based upon statistical recurrances. This leaves you very vulnerable to several problems.
For instance, the author remarks that he believes a bigger corpus of spam would help train filters. That's true, but misleading: it would help train filters that distinguish between his 'nonspam' corpus and his 'spam' corpus. In this case, he is surely increasing his true-positives.. his rejection of things that really are spam. But his false-positive rate is not helped at all, because his samples are so biased.
(Example: 10 spams get the word 'blunderbuss' but he has no regular email with that word. Therefore, any future email may be rejected because of the word 'blunderbuss', even though there is no basis to know whether the word CAN be used legitamately.)
If the system is done intelligently, this will simply mean that having a lopsided sample will do nothing (the resolving power will be dominated by the smaller of the two samples), but this may be counterintutive to some.
Another problem is that you don't know WHY choices are being made, and that's bad science. Ok, ok, so this isn't science, it's Spam prevention, but I like science.
---N
ASK is a system similar to yours with some tweaks: .sig, they are automatically whitelisted.
If you send someone email, and they reply to it, leaving in your
Mailing lists are handled automagically.
Check it out:
http://a-s-k.sf.net
The built in spam filters for Outlook and Hotmail are just so much less efficient than Spamassassin or Razor/SpamNET.
My recent experience shows about 90% of the spam I get can be detected by Spamassasin, 70% by SpamNET and about the same for Hotmail. The Outlook/Outlook Express filters are basically blacklists and catch maybe 40% if properly maintained.
It does sound very similar, so why haven't they been able to implement a Bayesian filter as successfully as the lisp guru?
What's the legality of a DDoS where each attacker is an individual person and not a "zombie"
I recall during the RIAA DoS discussion there were some methods of DoSing that were rather legit. (Slow HTTP request for instance - G, sleep 5, E, sleep 5, T, sleep 5, etc etc. Not a huge bandwidth hog but wreaks havoc with HTTP servers if enough people do it.)
retrorocket.o not found, launch anyway?
Well, my spamfile did (thanks procmail) and I submitted it to spamcop and noticed they have a freephone number.
Now I could ring up and it would cost THEM money, which is a little teensy bit of payback - but imagine posting that freephone number on a site somewhere where like minded people hang out. They could all ring up the number, cost the company money and tie up their staff by chatting to them about their product.
It might make them rethink their spamming.
no sig.
Paul is taking an interesting approach here, but he's not correct in saying that SpamAssassin doesn't use a statitstical approach. He has a bit of a point in noting that his system will generate a prediction probability which is more intuitive than SpamAssassin's scoring system in terms of determining how likely a message is to be spam, but there is also an attractive element to the simplified, non-math way that SA uses scores, which allows them to be more understandable to non-math people.
Seems like a number of the points which Paul makes in the article about spammers being defeatable, about the basic premise that they must get their message through in order to be successful, and that the war on spam is winnable are extensions from my interview with Salon a few months back, but his statistical approach fails to make use of one factor which I believe is critical (and which SpamAssassin attempts to exploit), which is that those commercial messages must convey a commercial message, in other words, they have to be a message, and have some sort of linguistic component which encourages the reader to do something. A purely statistical approach to spam filtering will lose the power of doing analysis of higher-order linguistic concepts.
SpamAssassin's approach is to use the universe's best known natural language processors (humans) to build rules which they believe can differentiate linguistic elements of spam vs nonspam messages, and then use the best optimization and statistical tools we have (currently only using decent tools, not the best tools) to determine how to score those rules against individual messages. The scoring system is very simplistic today, just being a simple sum of the scores of the various rules (though it's slightly nonlinear because of the properties of some of the rules, like the auto-whitelist). Future SpamAssassin development directions include extending the scoring system to be much more non-linear, including examining statistically the frequency of occurrence of combinations of rule triggers.
Automated rule-creation certainly has its place (for example, SpamAssassin's spam-phrase rule, or the auto-whitelist), but I truly believe that the ideal spam filtering system will always have to make the best use it can of human language processing skills. Using this combination of human/computer power, I believe that SpamAssassin can (and often does for many existing users) achieve better ROC performance than anything else.
I was going to post a message suggesting exactly this but you beat me to the punch.
:-D
Why doesn't my email app have this already???
Black holes are where the Matrix raised SIGFPE
Wow. It's described down to a level of detail that would make you think they've already written the Outlook add-in for it. I wonder why we haven't seen it yet?
Or, if you're not willing to sacrifice (or mess with) your MDA, check out ASK. It does about the same thing and works with sendmail, procmail, qmail, etc.
A-S-K
In short, because the morons that support spammers are not likely ever to bother with filters.
The one exception - Filtering with bounce messages. This will cause SOME spammers (not all) to take you off their lists. Since implementing fake bounce messages triggered for every identified spam (See spambouncer.org), my spam counts have halved, from 90+ spams/day to 30-35, and decreasing. Unfortunately, some spammers (azoogle.com) blatantly ignore bounces, and others have non-bounceable return paths. If more people bounce their spams back, those who DO have bounceable (but ignored) returns will have their bandwidth costs increase.
I think the ultimate solution is that the spammers themselves have to be fought. Legislation is one - If 1 in 100,000 people respond positively to spam mail and only 1 in 100,000,000 sue for $500-1000, spam quickly stops being profitable. Also, some form of "voluntary" DDoS of spammers would be nice. Not voluntary for the spammers, but for all those who are attacking. For example, download a small app that each day presents you with an article, that basically states, "Today's target is xxxx - They are targeted because yyyy" and the evidence is presented against them. User can now decide if they want to participate. To minimize legal risks, trickery such as an absurdly slow HTTP GET would be useful. (G, sleep 5, E, sleep 5, T, etc etc) - Doesn't increase bandwidth costs, but the server will probably be brought to its knees rather quickly from having to serve too many simultaneous connections. A client could easily spool up 40-50 such connections with minimal use of local resources, but the server would have to open up hundreds of thousands of simultaneous connections, causing the server to fork like crazy.
retrorocket.o not found, launch anyway?
Yup. SpamAssassin is pretty good at identifying spam. Only problem is that you have already incurred loos of bandwidth and CPU power. Yeah, not TOO big of deal for individual users, but magnify that by 1,000,000 or so if you are a big ISP or some other number if you are a business and it is STILL a real problem.
Filters are great for keeping spam out of your inbox but it doesn't solve ANY of the other problems associated with spam. While many people don't like the idea of spam laws, they would create a financial / criminal incentive to cut the volume. What if kidnapping and rape were not illegal? Wouldn't the problem be MUCH larger than it is today? The trend we see with spam is that it is increasing by orders of magnitude. If I had no filters turned on, I would get more spam than legit email. If the rate of increase in spam keeps up, I will see 10 spams for every legit email in about a year. This is just not right.
I don't have high-speed internet access available where I am. Spam costs me real dollars. Why should I pay for someone elses advertising?
Spam needs to be stopped at the source.
There is another argument that spam is a world-wide problem and that US laws wouldn't have much impact. While it's true that it wouldn't stop all spam, it would cut the volume. I also have no problem with a "usenet death penalty" for countries that don't take effective steps to curb spam. I already do this with a whitelist allowing the few international users that I do communicate with to communicate with me. Bottom line is that international spam stopped for me. I still incure a small bandwidth cost for the connection attempt, but not nearly as much as if I recieved the entire spam.
Am I just totally off base here? If so, why?
Isn't it better to worry about the 'evil' html up on web pages rather than in emails?
For most people, HTML e-mail is not "opt in". Just by browsing their inbox, they could be sending out requests to spammers' websites, thanks to e-mail clients that have preview features and HTML-rendering features.
Actively browsing the WWW, such as going to a warez site, is "opt in" just like going to the shopping mall. Browsers, such as Mozilla, which allow user control over JavaScript and cookies can help mitigate the risk of browsing the WWW (just like hiding things under the seat can help prevent your car from being stolen).
Healthcare article at Kuro5hin
Unfortunately, I don't grok LISP. Could someone please translate the code snippets into Perl or C so I can figure out what he's saying there?
Thanks,
Brant
xIf xYou xCan xRead xThis xYou xHave xWon xA xFabulous xVacation! xClick xHere xTo xRecieve xYour xPrize!
Spammers will start mispelling "hype words" to get them past. (They already do this in titles.)
I can envision having a spelling check to find such, but then you could be filtering out legitamate bad spellers, such as me.
Table-ized A.I.
LISP is prefix so instead of a+b you'd have +(a b)
/* if word is in good hash, return weight,
/* if word is in bad hash, return weight,
IIRC in c this would be similar (LISP guru's please correct me):
int g(char* word) {
else return 0 */
return 2*good_word_weight;
}
int b(char* word) {
else return 0 */
return bad_word_weight;
}
int main() {
if (g(word) + b(word)
To read makes our speaking English good. - X. Harris
I question his testing methods. If I read the article right (oops, slashdot faux pas, I admitted to reading the article) he built the Bayesian map from about 4000 messages, then tested the efficacy of his algorithm against those same 4000 messages! He waves his hands about why that's OK, but wouldn't it make more sense to take 10 minutes to build his map against the first 2000 messages and test it against the remaining 2000? I really don't trust algorithms that use the input data combined with the desired results derive those same results against the same input data.
;)
Secondly, over time, assuming that spammers put forth any effort into bypassing his filters, the filters will become much less useful. Spammers will intentionally misspell key words to lower their total spam rating. The easy solution to this is to make the map using a running total of only the messages from the last 3 months, or 6 months, or whatever period works best, but he should have at least mentioned that. Otherwise, over time the massive weight from the old emails will drown out any new spam identifying words.
All in all, it sounds like a great system, though, pending the results of a real test against emails other than the one you built the map from
Make it Distributed and make it work with eudora, and i'll gladly use it.
spamnet (see link above) promises to make it so that, if you add a filter to your email, and it (or you) shows promise as a good spam filterer, that filter gets added to those that all subscribers get. unfortunately, it's currently only for outlook, but i expect it will either add support for other clients, or someone will come up with an open source alternative...
- Entertaining Bits from the Ancient Kernel Tree
Not to filter posts for spam, but for, you know, quality!
That'd be easy to filter out.
My primary concern comes from the fact that most of the spams I recieve are either Korean or English, while most of the legitimate mails are in Norwegian. Sending me Korean mail is pointless anyhow, but I fear that simply the _use_ of English will make his scheme produce lots of false positives.
I don't get it. Does Korean spam use something other than WORDS to communicate? Or do their mail headers look any different than Norwegian ones? What makes you think your deleting Korean spam, and thus marking those Korean words (heck, all of them) as spam will be a problem? The filter gets built up for the user, based on the user's email. How could this not work for you? Why would marking of a few English words as not being spam be a bad thing?
If you have a driveway that connects to a public road, then people can park there. Your house is connected to a public road, I can walk in and watch TV. Your car is on a public road, I can use it without your permission.
A spammer that I tracked down was very unhappy that I knocked on his door. He claimed I was tresspassing. How could I, he opted in by having his house accessible by a public road.
If spamming is legal and honorable, why don't you post your real name, address, and phone number with each spam and on each website that you spam about?
Fight Spammers!
OK, I'll try. He's trying to score the word on a scale from .01 to .99. The value is the probability that the word is a spam word.
// word hasn't occured enough in previous messages // to have a valid score
// nbad is number of bad messages in database // ngood in number of good messages in database
.01, maximum is .99
g = 2 * (count of how many previous "good messages" the word has appeared in)
b = (count of how many previous "bad messages" the word has appeared in)
if( g+b 5 )
return 0;
fb = b / nbad
fg = g / ngood
score is fb / (fb + fg)
minimum valid score is
"Almost every wise saying has an opposite one, no less wise, to balance it." - George Santayana
How terrifically annoying it must be to correspond to you. Yeah, humans can work it out. Humans also like to use address books. Humans also don't like to have to check the calendar each time they email someone.
My deviantArt site
Nice system for list matching:
a-s-k.sf.net
It's the content I want to block. I don't want the spam to be sent to me in the first place. I don't want it to use up my bandwidth, which is half the reason for refusing spam in the first place. Plus, when handling other people's mail, it's one thing to block suspected spam sources for them; it's another thing entirely to examine the content, even if it's just computer logic doing it. If I am able to deploy the ability to examine mail for unacceptable content, then what else will I have to test for later? What will the government expect me to be able to do?
I'll stick with blocking dedicated spam houses, ISPs that harbor spammers, open relays, open proxies, dialup pools, and certain countries, by IP address and/or domain name. And I'll continue to block anything that can't get their reverse DNS right (this feature alone took out half the spam with very little collateral damage).
now we need to go OSS in diesel cars
Example: 10 spams get the word 'blunderbuss' but he has no regular email with that word. Therefore, any future email may be rejected because of the word 'blunderbuss', even though there is no basis to know whether the word CAN be used legitamately.
.99 probability of being a spam word, the message is not rejected based simply by matching this word. This one word will influence the final probability calculation (eg, moving towards a "hit" for spam), but the other 199 words could (and will) push the email towards the "valid mail" threshold. Thus, the .99 is negated by the near-zero probabilities of other words in the email (sorry, I'm not a mathematician -- my apoligies if that's confusing). There is an example in the article which explains this.
;)).
I don't claim to understand the article fully, but I'll take a stab at responding to your example...
Let's say you have an email with 200 words in it (including the header, etc). Let's assume that your friend the history buff is sending you some pictures of a blunderbuss. Of course he's kind enough to provide a description of the pictures and a short history of the blunderbuss.
Now, this goes through the filter which splits up the words into 200 tokens. Even if blunderbuss has a
If the email contained only the word blunderbuss (disregarding the words in the header), then it's probably spam. However, if the email contained only the word blunderbuss it's probably not very useful in the first place (unless you're an international spy
Well in the UK (not sure about elsewhere) dialling 141 before the number witholds caller id. Worth remembering :-)
no sig.
Even if I did want the Raleigh 3-speed, and I received an email (unsolicited, automated, spam, whatever) from anyone who I haven't spooken to before about my desire for the bike, I would want it deleted.
Even if I was looking for and really, really, really wanted that bike, if the opportunity to purchase it came to me through a spam email, I would still delete it. I don't want to give the spammer any revenue or to encourage spam in any way.
Even if some spam is actually offering you something you want, you probably don't want to receive it in your email. You may not want to send your spam to /dev/null, though, if you might get an offer for something you actually want. In this case, you'd want to set up a file that it gets sent to, and then search it for interesting stuff on occasion.
Of course, anyone offering something interesting to a list of email addresses probably also has a web site, and you could find the information with Google when you want it.
He sent his resume to a bulk-mailing list, that's spam for sure. People will be able to send resumes, just not to everyone in shotgun fashion.
Senator Mary Landrieu
724 Hart Senate Office Building
Washington, DC 20510-0001
Dear Senator Landrieu:
Earlier this month the Federal Communications Commission (FCC) issued a record fine of nearly $5.4 million to Fax.com for transmitting unsolicited advertisements via fax machine (ie. "junk faxing"). Coincidentally, the Associated Press published a series of three articles covering the state of unsolicited e-mail advertising ("spam"). I'm left wondering how the FCC can come down hard on junk faxers but how spammers (arguably of a lower moral class) are allowed to continue to operate nearly unmolested.
The law Fax.com was found to be guilty of breaking is Section 227 of Title 47 of the United States Code. The relevant text follows:
Restrictions on the use of automated telephone equipment:
It shall be unlawful for any person in the United States (...) to use any to use any telephone facsimile machine, computer, or other device to send an unsolicited advertisement to a telephone facsimile machine(.)
It is my understanding that the reasoning behind this law is based on the ownership of resources. Fax machines are purchased and maintained at the owner's expense and only the owner's expense. An unsolicited advertisement sent to this fax machine amounts to nothing less the use of these expensive resources without prior consent. In effect "junk faxing" is considered theft and as such the offenders are held accountable by law.
What does this have to do with spam? In my opinion, everything.
Receiving an e-mail is by all accounts more expensive than receiving a fax. While several companies are now offering stand-alone e-mail clients, I have yet to see one of those with a lower price tag than a fax machine. But even if their price tags were the same, an e-mail station requires that the owner not only pay a monthly fee for a telephone line but also a second monthly fee for the e-mail account itself.
Of course not even an end client is enough to receive an e-mail. The e-mail account you would be paying for is maintained on a very large (and very expensive) e-mail server, complete with its dedicated (and pricey) connection to the internet. There is simply nothing comparable to an e-mail server in the faxing domain. While a bank of fax machines doesn't cost more than the price of the machines and their associated telephone lines, the price a dedicated e-mail server and the associated connections can easily resemble that of a small car.
So why is it that the FCC is given free reign to crack down on junk faxers but spammers are free to consume valuable equipment they do not own?
If you are familiar with the AP articles I mentioned earlier you will know that spam is steadily eliminating the usefulness of e-mail itself. It has been estimated that spam accounts for up to 80% of the e-mail traffic to major e-mail domains such as Hotmail and Yahoo, a problem that their respective owners are all but powerless to fix. As more and more internet resources are tied up by these advertisements, the owners of these resources have had to resort to cutting off offending service providers from the rest of the internet entirely. Customers are finding themselves unable to use the internet access they have paid for simply because another customer of that same provider is abusing theirs.
But even then the providers are powerless to drop spammers. Spammers in the recent AP articles have proudly boasted of the way they outright defraud unsuspecting internet service providers when signing up for an account. And when the provider threatens action, the spammer threatens the provider with legal action. In recent months a spammer was even successful in receiving a legal injunction against their service provider, preventing the provider from stopping the spammer from abusing their resources.
I have little problem with receiving advertisements through the U. S. Postal Service. I know that the delivery cost for every article in my mailbox has been entirely paid by the sender. And while I am not happy with the current situation with telemarketers (I must pay for local telephone service before I have the "privilege"of being contacted by telemarketers), I must grudgingly admit that the state and federal laws designed to restrict telemarketing have been mostly successful. But I am not happy about paying several thousand dollars for a computer and $20.00 a month simply to have my e-mail account flooded to capacity with advertisements for products and services I have no interest in (and preventing legitimate e-mail from reaching me in the process). I am sure that you yourself have been bombarded with advertisements for websites featuring "nasty teens" or "penis enhancement." I have noticed that your office no longer maintains an e-mail address accessible to the public.
The examples of spam I mentioned in the last paragraph bring me to another point: I have noticed on your website your stated commitment to enforcing decency laws on the internet, to protecting children from access to objectionable material on the internet. It should be obvious by now to even the most casual of internet users that the biggest offender in this area is the spammer. While a user must actively attempt to locate a website in order to find such material on the world wide web, the mere existence of an e-mail account all but guarantees that the owner will have such material delivered to them on a daily (if not hourly) basis.
In my opinion the solution to this problem is very simple: expand 227 U. S. C. 47 to prohibit unsolicited e-mail advertisements in exactly the same way it prohibits unsolicited fax advertisements. Nothing more, and certainly nothing less.
I have seen some ineffective bills drift through both houses of Congress that are written to allow unsolicited messages so long as they have an "opt-out" mechanism. Ignoring the fact that such legal loopholes would essentially negate the law entirely (can you prove that you tried to opt out?), it quite literally sickens me the way some of your fellow members of Congress feel that spam is somehow an issue dealing with the freedom of speech. The mere existence of the internet and the supposed changes it has on how business and the legal system work (even though such "changes" have been shown to be a lie) have helped to convince these poor fools that people should somehow have a right to use and abuse the property of others. Does my neighbor have the constitutional right to break my kneecap so long as they provide me with the ability to "opt out" of future kneecappings?
The United States Constitution guarantees that all citizens are free to say what they want. It does not guarantee a soapbox upon which they can say it. Just as I am not guaranteed the right to have a billboard on Interstate 10, spammers should not have the "right" to use the resources of others simply because they're there.
Expanding 227 U. S. C. 47 to include e-mail is an extremely important issue to me and I hope with your stated interests on your website that it is also an important issue to you as well. I know that you are up for re-election this November and I intend to find out how your competitors feel on the issue as well.
Paul Graham on Fighting Spam:
Wham wham wham wham wham wham wham.
(its deeper than you think)
slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
Then again, it doesn't matter how great of a system could be built if the ISPs don't use it - just look at the open relays that still exist!
I really hate signatures, but go to my website.
Javascripts
I generally have feelings about JavaScript that mirror my feelings about HTML e-mail, and the underlying problem of bad default software configurations is the same. But I'll defer ranting about JavaScript, for now.
I agree that browsing doesn't always seem "opt in", but I was trying to point out that browsing is actively going out and about, which is different than reading through e-mail.
HTML and JavaScript in e-mail is more like a disease-loaded letter in a person's own mailbox (no one asked for it, no one needs it, it's just there, and it's dangerous). WWW browsing, on the other hand, really is more like shopping or traveling. There are real risks in going out into the world into unknown places, but we accept them as small and not worth sheltering ourselves because of them.
Healthcare article at Kuro5hin
Yup. Use the intelligence of hundreds of thousands of fellow spam haters across the internet.
Vipul's Razor: http://razor.sourceforge.net/
Pyzor: http://pyzor.sourceforge.net/
DCC: http://www.rhyolite.com/anti-spam/dcc/
Yes, they do work. no spammers can't get round them just by changing formatting or including random characters.
Government of the people, by corporate executives, for corporate profits.
Bernie is a moron spammer.
Fight Spammers!
It'd be great now if he offered an implementation which we coudl all use.
I think a progressive, ever-going implementation is best. I also think its best to filter based on headers first, and not download any spam (to save bandwidth) and then filter based on message content (for the messages downloaded) and move any spam to a spam folder.
Then the user simply looks at the spam folder and looks for false-positives, and marks them as "legit". Then the Bayesian filter recalculates.
Same thing for false negatives, and for the messages not downloaded. The user can look at the headers of the messages not downloaded and say if they're spam. Then the Bayesian filter recalculates.
Another good thing to do is to give a "password" to your friends for them to put in headers sent to you. I.e., 13y4890dshfpljk2134y9073254y32p9ur. Any message with that in the header would be given a 0% probability of being spam, as only those you gave that to would know to put it in the header. Should it become compromised, you can change it (or just don't give it to people who might compromise it).
Back to the Bayesian filter, another good thing might be to have varying levels of "spam". I.e., if something is almost certailny spam (i.e., 99.99999999% likely to be spam, as would a message with the header "Get fucked for free and make lots of $$$$$"), it would be placed in a DEFINATELY SPAM FOLDER. Other things would be placed in a "PROBABLY SPAM FOLDER". Etc.
Anyways, Bayesian Analysis is a really great method.
If your interested in Bayesian Analysis, there's a great phylogeny program which gives you (basically) a bootstrapped maximum likelihood tree (calculated from millions of trees) via Bayesian Analysis: MrBayes.
social sciences can never use experience to verify their statemen
Is there a shortcut that I'm missing?
-jon
Remember Amalek.
I just invented this great spam filter! It counts the number of people in the cc: field! Then I multiply it by 10, and that's the percentage chance it gets chucked! Only 14% as much spam gets through, with NO false positives!
Note to M1-ers: a curt but otherwise insightful message is not "Flamebait" or "Troll".
Well, I suppose if you wrote something like:
Hi, my sexy naked Russian teen lolita! I've increased my penis size to 45 inches by phoning for sex along with other like-minded people who I click with!!!!!!!!!!
I'll be around later, unless you want to opt-out, but it's not an idea I'd subscribe to!!!!
Then you might just generate a false-positive...
Code, Hardware, stuff like that.
There's a big difference with spam versus the illegal drugs trade and child porn: SPAMMERS WANT YOU TO SEE IT! This is their anchovies heel, there's clear evidence of the origin of the spam (or at least the incompetently administered mail server) - so that's where the fault lies.
I agree that there should be more effort on the part of the US government to remove these bags of sh*t from the 'net. Where available, the ISPs should be forced to diveulge the customer account used to post the spam. Any companies advertising via spam should be fined per item of spam. This last would remove a lot, since the spammers wouldn't have much to do without companies paying them. Lastly, if a mail relay is used repeatedly, either force it to close, sue the company responsible for it, or blacklist it all mail from it for at least a year (kind of like a prison sentence for the server host).
I waste far too much of my time scraping up the crap these parasites spew into my mail server - they deserve the harshest of penalties - at least equal to child porn swappers.
Code, Hardware, stuff like that.
I'm really trying to see a difference, but I just don't.
Another way of putting it that I just thought of:
We have a right to read our mail privately in our homes with no one peering in. We generally dispense with a notion of privacy when going away from home.
The gray area between e-mail and web browsing is due to them frequently occurring on the same computer over the same network connection; however, I just tend to view them in the same way as traditional activities. My habits tend to reflect this, as I use separate tools for e-mail and web browsing, and I ensure my web browser isn't configured to do e-mail-like activities. If I wanted to take things further, I would run my web browser in a separate and limited user account (which I already do for my Windows VM, but that's a different matter) or even configure a special browsing-only workstation that has specific firewall privileges. This isolation helps protect my computer from the fact that the browser automatically processes whatever data is sent to it.
As long as my e-mail is never automatically rendered and is always displayed as text, it is generally not as risky as web browsing and doesn't require as much isolation to be safe.
Healthcare article at Kuro5hin
Paul's
Why Arc is not especially object oriented
I would personally like to see Paul Graham spend even more time fighting OOP than spam. The second one is a lost cause arms-race IMO.
Here is something that rang true with me on his OOP musings:
Object-oriented programming is like crack for these people: it lets you incorporate all this scaffolding right into your source code. Something that a Lisp hacker might handle by pushing a symbol onto a list becomes a whole file of classes and methods.
I think using databases (properly) are the same way: a single relational formula does most of the work of a bunch of classes and "hand-indexing" these classes and methods together. (AKA GOF-math)
OOP hard-wires the "noun structure model" into the code (what Paul calls "scaffolding"). LISP and relational techniques tend to use *formulas* to manage these instead of physical code structure. IOW, we don't build structures, we order the information to build *itself* into the needed structures. (OO has the concept of "self-handling nouns", but it lacks the concept of self-handling structures, or interlinks, between those nouns.)
It less disruptive to change a formula than change the physical structure of the code.
OOP fans spend too much time looking for "the proper pattern or model". If you do it right, there is no one proper model or structure: it is virtual views that you create on an as-needed basis and can change on an as-needed basis without a bunch of code rework. You can also have multiple different views without them stepping on each other.
OOP creates code and work that is unnecessary and fragile.
(oop.ismad.com)
Table-ized A.I.
(* I'm not sure what the benefit would be to having a few words from the text. For me (and most likely other people as well), that is enough of an inconvencience that I may as well just scan through the entire email. *)
The point is to make it easier to eye-scan if you are worried about false positives. It helps by: 1. Making it easier to review many messages, and 2. Ranking so as to not check the flagrant ones if desired.
Table-ized A.I.
His algorithm works because spam uses the same repetive syntax. Because so many spam/emails are sent out - it can be flagged by pattern recognition... based on the assumption that it is written in English!
Huh? Where do you get that? The algorithm has NO KNOWLEDGE of syntax or structure. It knows only the presence (or absense) of words in the message, nothing of how they are grouped, positioned, ordered, related, structured, etc. There is zero grammar / pattern recognition as far as I can tell. As long as your corpus or database of reference mail is in the same language as the emails you wish to test, then the algorithm would work just fine. Perhaps you were thinking it used Markov chains?
Most of the spam I seem to get is in non-alphbetic character sets (Korean/Japaneese/Chineese, I'm not sure, I can't read it). I guess I hit the VIA support site in Taiwan one too many times or something.
I don't know much about that character set, but I suspect they don't use the same separator characters that his filter is looking for to separate its tokens.
Sorry but you can't legislate common sense. Believe me, it's been tried.
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
They also have a paper from 1998 describing it here
This wouldn't work for me anyway since my personal correspondance frequently contains the words "sex" and "sexy" not to mention "stud muffin".
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
I'd say my experience learning LISP was obscene
It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
Maybe the concept of a P2P network could be harnessed in order to fight spam. For each spam tagged as actual spam by a real human, by a ridiculously large CRC (1024 bit or something--to rule out possibly tagging innocent mail), the CRC could be traded via the P2P network. Automatic updating, almost instantly. A client could be written in about 2k of code.
Interacting with the email client would be another story, but just an idea.
The only problem I can think of would be sabotage. Anyone could tag legitimate mass mailings as spam (such as a mailing list).
Any comments on this idea?
This would make spam detection EASIER! Sure, the first spam written like that would probably pass through undetected. But when the user hits the (mythical) delete-as-spam button, then the filter would recognize that of the emails containing phrases like "MAKE", 100% of them were spam and 0% were real emails. When the second "MAKE" email arrives, it will be immediately detected as "fishy".
cpeterso
RTFA: Graham clearly believes that highly efficient filters at the inbox level will have the long-term effect of making spam unprofitable for most spammers.
Sure, in the short term you don't reduce your bandwidth costs, but imagine if a significant percentage of the population were using trained Bayesian probability filters! So little spam would get through that nobody would bother sending it anymore.
Any sufficiently well-organized community is indistinguishable from Government.
Is the spam for Taiwanese products, or just routed through open mail relays in Taiwan? If it's the latter, we could certainly outlaw using spam as a marketing tool for US entrepeneurs. If your company or home business sends out spam from Taiwan to US computers, you would still be breaking the law.
Don't forget that Friday is Hawaiian shirt day.
2) If you don't read the spam, they have no revenue.
3) You're gaining the valuable benefits of spam without paying for them.
4) Therefore, not reading spam is STEALING!!!
Oh, and
5) ???
6) Profit!!!
We can all talk all day about Spam-busting techniques, but honestly, can we all get together and make sure that our nine year old doesn't get porn mail all the time? Stopping porn spam would really knock the wind out of the sails of all spammers everywhere. I mean, this thing seems like slam dunk legislation. I know that many of you will say that this is a slippery slope of legislation and scream "THINK ABOUT OUR FREEDOMS," but no one wants their children to see pornography.
/. crowd can really sell that tagline to our local legislator and put a real strike back in the spam wars.
Really, all we need is some new-era Tipper Gore to scream the phrase we all hate at a Senate hearing... and no more porn spam:
"Won't somebody please think about the children?!?"
The chilling effects of this will be monumental. Why the current Right-Wing U.S. administration hasn't gone after this is totally beyond me. Its a cheap and easy target. Shows that they reinforce family values. I hardly agree in anything super-right wing, but this whole children-looking-at-steaming-hot-teens thing is ridiculous.
Whether enforced or not, in the United States soliciting pornography to a minor is still very much illegal. I think that the
I think the point of his filter is that your filter is unique to you. The filter is designed to pick up that you like foo, but don't like cc or something similar and it asigns a probability of the message being something that you would delete based on you current kept and deleted files. So unless there would have to be a word that everyone kept on their good list, and the spammer would have to keep up with the fact that your good list changes.
If foo was one of the words that had a low probablity of being a spam, but spammers started using foo, and it still saw enough other bads to delete it, it would probably lower the likelihood that foo indicated a good message. It looked like a really solid system, hope someone finds a way to get this added to more inboxes quickly.
Degaussing scares the bad magnetism out of the monitor and fills it with good karma.
There are several classification techniques in the field of machine learning that are all more powerful then simple native bayes. In fact in graduate school I built one that outperformed N.B. by a significant margin.
If people want to claim a "great new idea" they should research what has been done in the field first.
(* Gee, a bit OFF TOPIC wouldn't you say? *)
The topic is partially about Paul Graham, and he holds the view that OOP is oversold.
It is a "grey area" WRT topic relavancy, but not black.
Usually a particular moderator that beleives the superficial OOP cliches will knock it down a point or two. He/she/it must be on vacation today.
Table-ized A.I.
True, but if you implemented it at the institutional level (corp mailservers, ISPs, &c.), then t3H stUpiDz won't even know what they're missing!
Any sufficiently well-organized community is indistinguishable from Government.
Yeah they could, but it would take too much effort. They first have to make the probabilities factor change by sending you a whole lot of legitimate email (as if) and then later send you a spam message that can finally contain the words they made less likely to be spam.. wait a minute!
hmm sounds like a great idea, how about this a elisa style bot starts a mail conversation with you after sending 10 mails back and forward the bot sends you a spam message, the bot has beaten your spamfilters because the filters don't think someone on your contact list would send you a spam message right and you will read the spam message quite focussed, the spam message will be actually read, because you don't understand it and you think the bot is a person!
well don't be surprised if you experience it one day, remember this message, it started it all!
Consider:
If they are REPEATED innocuous messages that match against PAST "innocuous" messages that I decided were spam, that is going to pick this up.
Then your message goes into the corpus as "spam."
And messages that are written as multipart/alternative with statistically similar "innocuous" messages will be matched as spam.
You don't know the parameters. The parameters essentially involve the subjects I discuss with my family, or with friends, or with business associates, or with technical associates.
How can you possibly construct, as a "spam-meister," messages that resemble those without being someone that I regularly communicate with?
No, this "defeat" represents nothing of the sort.
If you're not part of the solution, you're part of the precipitate.
ANytime someone asks for my e-mail addres, it's their_business_name@conesus.com or their_personal_name@conesus.com.
If I ever get spam from a certain address, I can block the address, and goto the site in question and change my address to something else.
But the coolest part is if anybody sends a mass-email to me and my buds, they usually include a personal_message_to_me@conesus.com.
Don't eat your soul to fill your belly.
conesus.com
But this isn't enough, by itself, to classify a message. Messages do not solely consist of one or two words; they consist of many. And collecting the statistics together requires calculating a "relevance factor," based on all the words.
The one used for Naive Bayesian Inference is as follows: Rf calculation , and you'll notice it involves doing a logarithm-based weighting.
The formula doesn't care what words are used, or that you think one folder contains "spam" and that another contains "gold."
In my corpus, the word sex is used in 65 different mail folders, mostly probably pretty "innocently."
Drawing conclusions based on one or two words is, unfortunately, pretty incomplete. It might well be that the one use of "sexy" in a particular message doesn't force it into the Spam/Phonesex folder because it makes even more extensive mention of Enlightenment and WindowMaker and GTK Themes and winds up being very strongly tied to the X/WindowManager folder because there are several other words not related to sexual activity that make it (correctly) appear relevant to a discussion of window managers.
Graham is drawing an analogy based on two words (words likely to grip adolescent attention!); reality involves adding everything up, and those two words certainly don't tell the whole story of the whole corpus.
If you're not part of the solution, you're part of the precipitate.
The "foreign language" Spam that I get gets nicely refiled by Ifile into my Spam/Foreign folder.
That folder has a corpus of messages assortedly written in Han, French, Kanji, Korean, Finnish, French, Spanish, and Russian, and Ifile nicely recognizes that words in those languages provide evidence that messages seem most relevant to go into that folder.
Ultimately, it all involves human classification:
I go through them, and read them, perhaps just browsing titles when I see that spam seems appropriately filed.
By leaving the messages in the folder, indicate that they were correctly filed, and should become part of the corpus.
That then involves human intervention as I move the messages to where they should have been.
Note that IFile is useful for filing good messages, not merely at throwing away spam.
Indeed, the more that you use Bayesian filtering for, the more folders with distinctive kinds of message that you have, the better it gets at discriminating where messages should go. I don't have one "Spam" folder; I've got about 8 for different sorts of spam. I don't have one 'inbox' for all my "good" mail; the mail gets thrown into a veritable huge chasm of mail folders. The more there are, the better.
If you're not part of the solution, you're part of the precipitate.
The typical formula is
Relevance - Rf
There may be a bit of a "fight" between the words, but if all the messages containing the string my_wife@frobozz.org go in the Honey folder, and occasionally contain phrases like That dress was so sexy or the likes, that will change the Ff(w) value for f = Honey , and the message will be appropriately routed, perhaps into the subfolder Honey/Rendezvous where you put the weekly messages of that sort from your wife.
Of course, there's then the non-technical problem, namely locating a wife that would actually send that message.
If you're not part of the solution, you're part of the precipitate.
As Ifile source code is available that dates back as far as about 1996.
If you're not part of the solution, you're part of the precipitate.
I've had this theory for a long way on a technique that could be used to defeat spam once and for all. Despite what the author of this article states, trying to fight spam by analyzing the content is not going to defeat it, and as has been pointed out, there are many ways to work around that solution.
Targetting the sending addresses, and most other techniques like that simply lead to wars of one-up-manship as the spammer and spam fighter struggle to find better techniques to hide and detect spam, respectively.
So what's the theory? Fairly simple, really, and the technology is already available, but not widely implemented. Spam largely suffers from an identity problem. Consider that junk mail that arrives in the post box can easily be identified and/or blocked through legal means if necessary, largely because we know where it comes from. The reason spam has proliferated is because SMTP traffic is largely anonymous - mail servers basically trust the mail they receive and have no real way to verify the information being presented to them. Yes, they can check From: and To: headers to verify that the email is local / remote / relay attempt, whatever. But with the number of open relays on the net, it's easy to forge and bypass these checks.
By using SSMTP (SMTP over SSL), all email can be logged with identifying information from the original sender. If enough servers on the net start to support SSMTP, and increasingly mandated its use, eventually I'd be able to block all regular SMTP traffic. This has the added advantage of making email more secure.
But how does this stop spam? Well, it doesn't directly stop spam, but it means that we would legitimately be able to identify who originally sent the email. Once that happens, the spammer can no longer hide behind anonymous gateways. It probably wouldn't even matter too much if open relays were accidently left open - so long as the open relay didn't support SMTP but only supported SSMTP.
Ideally, every user would require their own secure certs to properly identify the sender, but this would probably add too much cost for the average user, and may be rejected for privacy reasons. But so long as the mail servers themselves were configured this way, we would always be able to identify very quickly where the email was originally sourced, thus giving a recipient an easy place to target (and hence sue if it comes to that).
As this takes off, it may actually be a way to make spam legitimate. The secure cert attached to the email could have an incentive allowing users to opt-in or opt-out automatically. A user could set their mail to say "yes, I'm willing to put up with ads if you're willing to pay me for it" putting the cost back on the person responsible for the spam in the first place - the advertiser.
Anyway, it seems to me like a fairly simple way to solve this - but it does take a lot of co-operation to get there. Something that hasn't happened yet for IPv6, another new protocol that doesn't really seem to be getting off the ground. So what am I missing?
We're talking about Naive Bayesian Filtering, where the assumption is made that we can assume the use of Bayes' formula even though we know it's not quite independent.
What you're missing is that the real formula doesn't just involve two words; it involves all of the words in the message.
The usual formula is Rf, and you'll notice that it involves multiplying the occurrances of words in the message with the logarithm of their frequencies in each folder.
The word "sexy" may usually be enough to consign messages to the Spam/Websex folder, but if there are some occurances of the term "sexy window manager" in a discussion of some window managers, the fact that the names Enlightenment, WindowMaker, stupid , memory-hungry and themes occur rather a lot in X/WM and never in the Spam folders means the relevance total will most likely favor the right folder.
If you're not part of the solution, you're part of the precipitate.
If you get a new email message that has a bunch of "non spam" words in it, it seems likely that it will not be marked as spam. As the article said, spammer's vocabulary is really limited.
> If my machine crashes, I have to start all over?
No, you just restore from your backups. You do DO backups, don't you?
Not everything that can be measured matters; Not everything that matters can be measured.
I'd like to see mail browsers add a nice big "SPAM" button that will can do a number of configurable actions, and has a useful default. I suggest as the default that it forge and send back a "no such user" message, save the message in a "past spam" folder, and occasionally invokes a naive Bayesian statistical analysis program (as Graham describes) to create a filter for the future (then filter out email with a high probability of being spam). Perhaps it could optionally do other things, such as forward a copy to a list of email addresses (e.g., your local "abuse" account, the newsgroup news.admin.net-abuse.sightings, and email addresses of well-known spam killers), or calling on other spam killers to check it like SpamAssassin.
Perhaps there could be checkbox beside each action like "don't do it when you press SPAM", "do it when you press SPAM", or "confirm before doing it when you press SPAM" - that way, you could get rid of chain letters without sending them to net-abuse.
By building easily-invoked SPAM-handling capabilities right into the mail browsers, people will be able to fight back more easily.
I know the Mozilla folks are considering anti-SPAM measures; I hope they're willing to build in this kind of functionality, so that it's enabled by default.
- David A. Wheeler (see my Secure Programming HOWTO)
I did this sometime ago, Unique Spam Invoicing System, USIS aka "Spammer Nailer". And am really planning to bill the spammers. The idea: spammers collects email by harvesters: this page contains an unique address and a service agreement, which says that by sending an e-mail to the address, you agree to the terms of service, which you can read at the url. And as the address is unique and I got the weblogs, there is atleast even some chance of nailing the spammer.
Another idea is to start putting lists of legislators' email addresses (as well as email addresses of their major supporters) on web pages so that spammers start spamming them, too. Legislators hire others to read their emails, and they surely have filters (false positives aren't a problem here!), but it could eventually become obvious even to legislators. Especially if you get the personal email addresses (according to many legislatures, it's legal to share the email address with spammers - if they don't want it to be, they'll need to pass a law to make it illegal!).
Another idea: a non-profit organization creates and maintains a database of HASHES of email addresses that do NOT want spam (say MD5 and SHA-1 of canonicalized email addresses, e.g., all lower case; an entire site could be represented by "@mycompany.com"). Anyone can download the database, for a small fee. Anyone can add or remove their email address from the list for FREE (and it must always be free); they just need to subscribe/unsubscribe, with a separate email to confirm (to show that they really did add their email address to the list; entire sites could require "root" or "postmaster" to represent them). Then legislation can be enacted that gives serious $$ penalties to any spam to the "no-spam" list. Capturing the database wouldn't do any good; it would only provide hashes and date/time stamps.
Anyway, just an idea.
- David A. Wheeler (see my Secure Programming HOWTO)
No, it's like saying that if you not only don't have a car alarm, but leave it unlocked and the keys inside and put a sign on it that says, "Drive me to your heart's content", you don't get to complain when people do so.
Dyolf Knip
No, it is if you say, move the car, if you are blocked. Then you decide to take the car for a long drive.
Spamming is stealing.
Fight Spammers!
(* Yes, but how many spammers are going to reply to your challenge? Zero! And that alone will make the challenge an effective tool. *)
If confirmation requests become a wide-spread practice, they *will* take advantage of that.
Too many techniques here assume that what works in obscurity works en-mass also.
Not the case. Spammers tarket the widest-used techniques. When something becomes wide use, kazaam!
Table-ized A.I.
And I understand your second point about the utility of having an agent learning your prefrerences. But that is of maximum value in the situation you describe: reading news. That's a situation where you don't care if you miss 1% of the valuable stories. That's not true of email; or even if it is for some specific people it probably isn't acceptable performance for someone trying to distribute something called a "spam filter".
I also understand your point about including headers vs. just the content, but I think it was a good choice on his part. I assume you've done some work on spam-fighting software. I have, and I can assure you that (at least a few years ago) some headers are truly spam-only-markers. He'd be passing up a great filtering chance if he didn't look at them.
Perhaps the software needs to go that extra step: rather than have a "this is spam button", it probably already has a "move to this folder" option. Why not create a probability array rather than a single spam probability assigned to each dictionary word? Tie the folder names to probabilities in the array.
It could then transparently learn to move all my email, sending my Pilotgear mailings to my Pilotgear folder, etc. It would also reduce the "value" of automated senders as being only spam-related.
I still wonder how an agent will be able to discern the difference between a chain letter from Aunt Bertha and a "Hi, this is Aunt Bertha, meet my plane tomorrow please?" Every message from her gets a "Love, Aunt Bertha/get your free Juno account" tagline at the bottom. So, "love" "aunt" "bertha" all become words that are very strongly associated with ChainLetter-taint, when the real fingerprint probably should be the "Fw:" at the head of the subject line (as well as every line beginning with '>') Without at least one good letter from Bertha, her email will always end up in the ChainLetters bin. But with an array of sorting options, at least they wouldn't end up heading straight to /dev/null.
Perhaps other message-cumulative characteristics should be used in conjunction with word counts, such as message length, total count of exclamation points or dollar signs (or even of all individual ASCII characters,) grammar checker score, spell check score, etc. I think the overall concept of using a probability based mechanism rather than a score/threshhold mechanism is sound. I think we both agree that his approach needs more refinement.
John
Would this also work with email virus? I think it would since the virus would also have a defined patern to it and the program would pick it up after the first one.
I actually proposed this on Advogato many moons ago, in February of 2001.
-Waldo Jaquith