Slashdot Mirror
MIT Spam Conference Conclusions
RT Alec writes "The 2003 Spam Conference has concluded, reports InfoWorld. (related read: abstracts of the conference discussions). I was unable to attend the conference, but it appears all that was discussed was filters (client and server). I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25 (you do have permission to use that SMTP server, don't you?). It is not too tough to set up an SMTP server to require authentication, or at a minimum to run off a different port. I am suprised that this is never mentioned as a cure for spam. If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive). I was pleased to see that Barry Shein, president of The World (a Boston based ISP) was included in the talks. I am not sure by the abstract (see link above) posted if he mentioned blocking port 25. In a recent interview he did not mention it."
"We conclude that spam sucks."
;-D
Tax money well-spent
Reply or e-mail; don't vaguely moderate. Ex-O'Reilly/MIT employee, now a full-time Google employee.
Great! Well great if people didn't make their living and devote serious time and effort to spamming. Anything you do, will just cause a shift to other methods. Make it not profitable(or illegal). That's the simple solution.
Seriously, who doesn't block AOL?
Could somebody just modify a virus scanner to detect spam? I think when a virus scaner looks for virus behavior, the problem is the same.
How does a spam filtering language, such as CRM114, determine between one person's junk and another person's treasure?
How can any statistical method or programming language be '95%' accurate in such a complex and dynamic system as email and spam?
My solution was to simply abandon email in favour of GAIM (The excellent IM client for linux!) and IRC. Those are the only CIVIL ways to communicate now anyway.
I am a filthy pirate.
but what if people want to run their own mail servers? For their own domains?
Are you saying that if I want to run my own mail server, I should get in touch with the mail admins of every single mail server of everyone I might ever want to send an email too so that I can send it on another port?
That's ridiculous. I shouldn't need to subsidize MX providers.
Otoh, a good solution might be traffic shaping, or even a sort of intelligent traffic shaper that limits the number of actual emails per day.
Personally, I think SMTP is just obsolete. Schlepping anti-spam mesures onto it is like trying to put copy protection on CDs. It's just not going to work. What we need to do move to new protocols. Ideally two separate ones. one for personal mail, and one for commercial/bulk mail. The personal system would make it difficult to send out tons of mail, but easy to get into people's boxes, while the commercial system would make it hard to get into the box (i.e. you need to be pre-authorized) but, by definition, you could send out as much as you want.
Digital certificates and encryption would be helpfull, for one thing
autopr0n is like, down and stuff.
Please don't promote blocking port 25, whatever happens. That would be very annoying.
I'm already annoyed at being collatoral damage in the war against SPAM. I use mutt as my e-mail MUA, which is not an MTA and doesn't support use of an SMTP server. No problem; use sendmail or exim on my macine to actually *send* the mail. Except that I find out that some of my mail is bouncing, because my cable modem is in a blacklisted range (the range that includes "all cable modems"), and therefore being rejected by some SPAM filters. I don't run an open relay, I'm just using a program to send mail from my computer in the way that it is designed.
Very annoying.
So I have to configure my MTA to forward to a gateway SMTP server which won't be on the various RBL lists. A pain, but fine, I can do that. I've managed to get that set up... but I'm not using Comcast's SMTP server. Maybe I should, but after briefly using @Home's mail services, I've leanred simply not to trust the cable modem ISP services for anything. I've got web hosting outfits I pay for, so I can use those SMTP servers, configuring my exim to forward to them and use SMTP AUTH. But if Comcast starts blocking port 25, then *that* won't work, and I'll be stuck again. (And, of course, "getting another ISP" isn't an option, because where I live, the cable company's got a monopoly as far as broadband access goes. I *do* have another ISP I pay for for things like news and mail, on top of the cable modem. But, unlike where I used to live, I don't have the option of going with DSL and choosing the ISP to use with it.)
Let's please not put forward this idea. There's enough collatoral damage as it is. And it won't really cut back on the spam, either. It's very very fuzzy logic to assert that since 50% of the spam now comes from AOL customers, that shutting that down would cut spam by 50%. The spammers out there will just find other places to spam. Going after the spammers themselves, and not just some of the tools they use, is the only way to stop spamming. Anything else only temporarily inconveniences them, and meanwhile greatly inconveniences innocents.
-Rob
So you block port 25. So what? So they start polling all your other ports looking for an SMTP server. Oh. Right.
Does anyone have an idea how much spam comes through open relays vs. spam friendly ISPs?
It's now common knowledge in most academic circles that one can customize their email client to block spam via the utilization of a standard Bayesian filtering mechanism that keeps a document corpus of messages that have been marked as spam by the recipient of the emails. Any further emails received are then fed through the Bayesian filtering subroutine and marked as spam if they're tested as such.
.96. If you based the probabilities on word pairs, you'd end up with "special offers" and "valuable offers" having probabilities of .99 and, say, "approach offers" (as in "this approach offers") having a probability of .1 or less."
As Paul Graham writes, "A few simple rules will take a big bite out of your incoming spam. Merely looking for the word "click" will catch 79.7% of the emails in my spam corpus, with only 1.2% false positives.
One idea that I haven't tried yet is to filter based on word pairs, or even triples, rather than individual words. This should yield a much sharper estimate of the probability. For example, in my current database, the word "offers" has a probability of
Reply or e-mail; don't vaguely moderate. Ex-O'Reilly/MIT employee, now a full-time Google employee.
How many spammers use real addresses?
The problem is that they use an AOL connection to get online, then spoof through a korean SMTP sever.
I like the idea. But, also do it for most of the dial-up services. Cable and DSL does provide a way back to the spammer's home.
Fight Spammers!
Blocking port 25 is not the answer. It creates more problems than it solves. I am a senior sysadmin at a mid size hosting center, and we run mail services for a lot of our customers. The single biggest problem with mail is dealing with ISP's that block port 25.
Saying "oh, just run it on a different port" is not as simple as it sounds to us geeks. Sure, we offer SMTP on another port to get around those ISP's, but your typical nontechnical user doesn't even understand the problem, much less know how to apply the workaround. And during the time they can't send mail, they're blaming you. They're blaming your "broken" mail service, because the mailbox their ISP provided them with is working just fine.
So you set up the nonstandard port and tell them "point it here." Now you're wasting untold amounts of tech support time on the phone with the nontechnical users -- you have to figure out what operating system and e-mail client they're using, and hopefully it's a setup that someone in your tech support organization is familiar with. Then you have to walk them through the process of setting up SMTP on a nonstandard port, and setting up authentication if necessary. During that time, you've spent enough tech support time to make that account unprofitable this month, and the spammers have found some other way to deliver their mail anyway.
Blocking egress on port 25 is not a good solution.
Tired of FB/Google censorship? Visit UNCENSORED!
No, the key problem is ISPs that don't disconnect spammers and charge them for violating the AUP, as well as ISPs that don't even have anti-spam AUP's. Open relays are next on the list. True, blocking outgoing port 25 traffic on the routers might eliminate a lot of spam (not a significant amount: in my experience the majority of spams I get are from various Asian countries, though configuring Postfix to reject connection attempts from a dozen or so subnets in China has cut down drastically), but then again, dropping every packet would solve the problem even more effectively, because:
As soon as an ISP blocks port 25, any spammers using that ISP will run their spammachines off of different ports. If an ISP requires SMTP AUTH connections to their mailservers, how long before spammers start relaying through their own ISP servers? Ultimately, blocking port 25 will have no measurable effect on spamming, because if the ISP provides a means around it for sending legitimate mail, it will be abused to send spam. All your proposed remedy will do is make life difficult for those who run legitimate mailservers.
This is by far the fastest way to destroy an open and accepted standard that I have ever heard.
While they started out with the bayesian algorithm described by Paul Graham they quickly discovered that the effectiveness of his algorithm tends to depend on the values of some quite sensitive tuning parameters and that diffrent people can get wildly differing degrees of success depending on their configuration and the types of spam/ham that they receive. Gary Robinson wrote an interesting critique of Paul's algorithm and helped the spambayes team incorporate his so-called chi-squared combining scheme (which apparently isn't bayesian at all) which doesn't seem to depend so much on 'magic' numbers and their testing framework showed that it works surprisingly well for both small and large sets of messages.
It's still under active development although most of the ongoing work is centered around the user interface components (POP proxies, Outlook plugins, etc...) whereas the actual spam classifier hasn't changed much in a while.
Well worth looking into if you're getting too much spam. Who isn't?
Okay, maybe I don't get what the poster is saying, maybe I'm just a fool, but blocking port 25 will not just kill all spam relaying, it will kill all incoming mail. See, SMTP isn't just a protocol used by your email client to send mail, it is also used by mail servers to talk to eachother.
Perhaps the suggestion is to only allow relays on an authenticated connection? Okay, that solves the problem, but there is a world of difference between doing that and simply blocking port 25 altogether.
Theo deRaadt of OpenBSD fame has put together a nasty little spamd, a daemon that attempts to tie up a spammer's resources. Basically, it slows down connection attempts and then sends a temporary error code back, sticking the spam in the mailqueue and letting the spammer try again, and again, and again. Designed to use up as few of your resources and as many of the spammer's as possible.
Excellent description of how to use it with your own self generated blacklist at http://www.benzedrine.cx/relaydb.html.
Unfortunately, it's only on OpenBSD so far. Can some one please port this to Linux by tomorrow?
This conclusion is simply and fundamentally WRONG.
It is critical for the future of the Internet that ISPs provide unmolested IP service. When ISPs are permitted to filter anything, for any reason, you start down a slippery slope. As soon as ISPs start trying to prophylactically control what goes on through filtering, they will find new things they need to control, for "security" or "liability" reasons. This will screw the end users by changing the 'net from its current state to a choice of which ISP's walled garden you want to be trapped in -- which ISP's filtering and censoring you want to pay for the privilege of being subjected to. It also screws the ISPs -- technologically it's expensive, it creates new problems for their customer service to deal with, draws the ire of some of their customers and civil liberties types, and the more they try to filter/control/censor, the more ISPs will be legally required to (the principle behind common carrier -- if I provide a neutral and blind service, I can be exempted from being required to control many things, but if I provide a controlled service where I can know what's going on, then I'm required to use my control and knowledge to prevent certain things or I can be held as aiding those things being done)
And it won't stop the bad guys. The worst thing about the spammers is that they're just smart enough that whenever any effective anti-spam measure comes around, they just find a way around it. Yes, AOL filtering outbound port 25 today will stop a lot of spam TODAY. And guess what? The spammers will just do something else. Open -- or cracked -- proxies are the up and coming new spammer tools. Please explain to me how cutting off outbound port 25 solves that problem. Please explain to me why spammers will just go away and stop spamming because you're blocking port 25 as opposed to finding some other way to spam.
This is a solution where the users lose because they lose functionality and are likely to lose more with it as precedent. It's a solution where the ISPs lose because they incur new costs and liabilities while only temporarily slowing down spam. It's a solution where the spammers lose least of all, they've been shut out of ISPs before and they've been blocked in various ways before and they already know how to do their deeds differently if they need to.
This is a really bad idea.
I am disturbed that a bunch of supposedly clueful folks came up with this.
http://www.gatewaydefender.com
I don't use other SMTP server then my own computer. But this means that my computer have to resolve the recipients address, find the mail gateway and send the mail to the recipient. If the ISP starts blocking port 25 I'll have to relay everything thru them. That's absurd. The spam have simple solution and it is using mail address aliases that are unique for the person/organization that will use this address. Wells Fargo (my bank)nows me as wellsfargo@mydomain.com. They use this address when they send messages to me and If I start receiving spam with a recipient wellsfargo@mydomain.com who do you think will be responsible for this? And the solution is quite simple - just delete this alias and go on.
A better idea might be a DNS hack. if the reverse lookup of the sending server's address doesnt include mx.domain.tld, require AUTH. It is less of a problem than blacklisting all cable modems or blocking outgoung traffic to SMTP and will do at least as much to kill off spam.
Pointing a domain to your Broadband or dialup address is easy, but adding a PTR record to your ISP's server is hard. Hopping from colo to colo is a lot harder than getting a new dialup every other day.
I am writing this document with the hopes of proving the viability of a design. Slashdot is a wonderful community to see if something can fly. I also recognize that a successful solution cannot come from a single vender and the more people that see this and implement it the better the possibility of it becoming reality.
As a lead software designer, I am paid to analyze problems in my company and to provide solutions. I attempt to understand relationships, contemplate the results of our actions and generalize our behavior. My personal problem for a while now is that I am sick and tired of spam. It is bad enough that my public email address has been eaten alive by porn ads and pyramid schemes, but even my work address is starting to slowly accumulate items. Getting un-work-related email is bad enough, but I just love getting emails for teen women who willing perform degrading acts with various barnyard animals in my inbox at work. Now of course, I know it is obvious that many companies are untrustworthy. Any free service is just begging to place you on an email list. However, many claim that they will keep you strictly in confidence, or at least allude to that effect. They take advantage of the fact that once you get on a lot of email-lists it becomes impossible to figure out who gave out your email address. An astute friend of mine made an observation about job-seeking sites. Shortly after an email address was in the site, spam started to trickle and then poor in. It is obvious that once your email address goes to just one unscrupulous company, the game is over and you are now in the war for ferreting out anything useful in your in-box. My friend had then made a brilliant suggestion, "Why not create an email account just for the job-seeking site and see what shows up in the box?" I was going to do it to see just how private my email address really was and then it got me thinking.
As an analyst, I have learned that the most difficult thing to differentiate sometimes is what is a problem and what is a symptom. In order to start a fire, you have to have three things: oxygen, fuel and heat. If you prevent just of those things from working, then the fire goes out. All too often people attempt to put fires out by dousing the flames. A firefighter knows instead that the goal is to use water to cool the fuel down so that the fire goes out on its own. At one point I came to my first realization, "Spam is not a problem, it is a symptom." The simple truth is that it is very cost effective to spam. Even if you are running some questionable diet pill, if you can email one-tenth of the populous of the U.S. then you can get thirty-five million hits. If only one in a hundred thousand buys the product, then you have 350 people buy the product. Since the cost of the Internet is shared amongst all of its inhabitants and most places just eat the cost of having the Internet available to their location with an always-on connection (email, web-browsing, etc.), the cost of spam is next to nothing in comparison to more traditional mailed advertising. The question then becomes, "how does one make the act of spam no longer cost effective?" It would no longer be cost effective if the vast major of the email sent was rejected automatically because it was unwanted. Therefore, the problem with spam really is, "How do I determine if the email that I am receiving is wanted or not?" If I can make it so that the mail server automatically rejects unwanted mail, then it no longer becomes cost effective to spam.
A lot of effort has been placed into attempting to write software that determines if something is spam. The bulk mail folder in many systems attempts to prevent these types of emails from taking up your work time. However, it is actually the human component that decides if something is actually wanted or not. As we can easily see, any automated system can only be an approximation of your requirements because it can never know your needs perfectly. We will always, after the fact, have to add to the rules some new source to omit some new type of unwanted mail. This is known as a negative system. The system assumes everything is ok and one must provide a rule to prevent (a negative act) an undesired behavior. This means identifying spam perfectly will always be a mathematically impossible endeavor. This lead to my second realization, "The problem can not lie in identifying spam, but must lie in identifying the offenders."
It is interesting to note that we have already solved this problem with practically every service that is out there--authentication. In order to know if you want to let someone into your FTP server, we authenticate. In we want to know if you want to let someone into your private web server, we authenticate. If want to let someone retrieve their email, we authenticate. But, to send email to someone, we don't require authentication. I propose a simple concept to authenticate email-- a really, really big, unique number to which I propose calling an email certificate number (ECN). In order to not have your email rejected by the server, your ECN must be on the acceptance list of the email server. The scenario would play out like this:
I go to site XYZ.com and in order to let me download their free software they want me to give them my email address. They have on their site a button that requests authorization from me to send email to me. The web click automatically pops up a dialog that states that I am giving them my email address and my browser is assigning them a unique ECN. Perhaps it automatically fills out a description for the ECN stating that this ECN is for XYZ.com. After clicking "OK," the web site has my information. Unbeknownest to me, but beknownest to their fine print, is that they are going to trade my email address with "select partners." Translation--the entire friggin' Internet. Spam begins to file on in to my inbox. This time, however, I can do something about it. Each email has my ECN number that is unique to this group of people. This allows me various choices. First, maybe all the mail is useful to me and I can just accept it. Second, I like getting mail from XYZ.com. After all, I choose to sign up to their email and wanted the newsletters. It would be an easy to create an email rule to reject all email from that ECN except for mail from XYZ.com. Finally, I could feel that they have abused my email address and therefore I do not wish to deal with them anymore. Therein, I revoke the ECN entirely and my email address is now useless to them. In short, if a company gave out their email addresses and those affiliates pissed off the customer base with sexual aid products, then it would no longer be cost effective to mass mail. The incentive would be come to treasure good email addresses and to not abuse them.
This of course also allows you to control your personal email. For personal email, you have to take a slightly different approach. You have to preauthorize the email server for the address. As an example, I run into an old friend and we decide that we are going to keep in touch. We exchange email addresses. When I go home, I authorize an ECN to the address. When the first email from the friend comes in, my server automatically responds with an email establishing the ECN. When I email him the first time, I get the same treatment. If whatever reason I wish to break contact, I can just revoke the ECN and that email address can no longer send to me.
The real trick to this system, assuming someone doesn't come up with a serious hole to this design, is usability. A fair percentage of systems will have to implement and require this behavior in order to drive the entire market to behaving this way. It would also have to be very user friendly by being very transparent to end-users so it is simple to implement and control. I think this is one place where the Open Source community could really shine. This would be a real innovation and I figure if people started on it now, enough systems could push the rest of the world to adopt it if they want to continue to be able to send email. I'd say it could probably be in force in two years, which would a wonderful amount of time to see the end of spam.
Bel, the mostly sane.. "Of course I can't see anything! I'm standing on the shoulders of idiots." -- Me
Barry's proposal for that last point was a fundamental change in the economics of spam, as follows:
Basically, it boiled down to "Spam is currently in a gray area legally, so let's legitimize spam in order to divide the spammers into legal spammers (who pay handsomely for the privilege) and illegal spammers (who do hard time, just like people who cheat a utility company).
Challenging proposal, and great fun to hear him speak.
Send spam using AOL's e-mail client and your account is nearly-instant toast, thanks to automated rate-limiting software.
AOL set up rate limiting sometime around 07/98. Yes, it was THAT long ago. Note, as another poster has said, this wouldn't stop someone from using AOL as their ISP and connecting to another SMTP server for spamming purposes, but considering how slow (not to mention expensive) AOL-provided net access is, I doubt any real spammer would use it for even that.
Since most of the
---
DRM is like antifreeze, to the MPAA/RIAA it's sweet, to the consumers it's poison.
There are potential customers using AOL. A significant percentage of my existing client base either is using or have used AOL since before they became a client.
I really don't like the idea of ISPs blocking ports. That should be the responsibility of the end user.
Instead of blocking ports why don't they force users to sign an agreement that they won't send spam and if they do they'll pay each recipient $50/incident.
Then if a bonehead sends spam they can go after them and enforce their TOS. I believe AOL requires a valid credit card number to even do the free trials, but I'm just guessing.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
It's interesting to see that the talks focused on heuristics exclusively. The main problem with all of these techniques is that they may classify legitimate email as spam as well.
...
Since two months, I've been using the Active Spam Killer (ASK) now, and this has been mostly successful. In short: If a person writes me an email, they will have to confirm the mail, unless they are on my whitelist or the email contains a magic key (which is included in my sig and will thus be included in a reply). Confirmation also places a person on the whitelist, automatically. Since most spammers forge the From: address, they are not able to confirm their mail, even if they wanted... -> Pretty much no spam (dropped from approx. 20-30 spam-messages per day to 1-3 per week). Sure, if you order a book at amazon, their computer might not confirm. Thus I look into the confirmation queue from time to time whether anything in there is legitimate. Thus far it has not yet occurred that a person would not confirm his/her email, by the way. ASK is well documented, written in python and easy to setup.
There is another similar system (which I haven't checked out): TMDA.
I am wondering why big corporations, universities, ISPs are not providing such a (preconfigured) system as an option in their email packages
I didn't realize this luncheon meat was so popular.
Did anyone record the presentations given at the conference? If so, can you put them online?
Prevent email address forgery. Publish SPF records for y
As usual, nobody is reading the article, and hence everyone misses the real meat. Ignore the silly web-zine hack writers and just go here:
http://spamconference.org/
The talks are online.
I don't use my ISPs SMTP server, I grew tired of it always being down and or getting tons of spam. So I set up my own SMTP server here at home, I filter spam by blocking whole sub nets and entire domains (like aol.com). Relaying is NOT allowed except from my machines and I don't get spam addresed to non-existant users on my e-mail server.
:)
Yeah it can be a pain to admin but I enjoy working on it and just haven't gone the DNS-SBL from SPEWS route yet..
Besides, I get a real kick from reading the mail logs
"I bow to no man" - Riddick
I'm utterly confused as to why the other excellent response to this post has been marked "troll" twice.
First of all, CRM114 is just a language. Bayesian filters could just as easily be written in Perl or C. The language makes no discrimination whatsoever.
Secondly, the very point of Bayesian filtering is that it learns what you consider trash and what you consider treasure. You start with a training set of several hundred "legit" messages and several hundred spams, and it goes from there.
The reason it works so well on a person-by-person configuration is that certain phrases (eg, email addresses of people you know in the "From" header) correlate very strongly to good mail, while phrases like "click here" and "enlarge your" are almost certainly spam indicators. Everything between is personal; if you're on a BDSM list, your filter will learn that you like that stuff. Given a training set with your personal tastes, rates well in excess of 95% are possible.
Incidentally, this is why Bayesian methods aren't that great for site-wide filtering (that, and they would be tremendously slow); it's much harder to establish what a *group* of people considers to be "not spam."
Actualy, I think something more like this, at least for 'personal' mail protocol.
You have a Certificate Authority, say your ISP, VeriSign (gag), Me, whoever and when you send an email you digitally sign it, and send a copy of your public key (to verify), which in turn has been signed by the CA. If I trust the CA, then my mail server will accept your mail. Otherwise, bouncy bouncy...
If a CA gives out a lot of certs to spammers, they'll get taken off the list of valid CAs.
autopr0n is like, down and stuff.
"I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25 (you do have permission to use that SMTP server, don't you?). It is not too tough to set up an SMTP server to require authentication, or at a minimum to run off a different port. I am suprised that this is never mentioned as a cure for spam. If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive)."
In my opinion, this is a terrible idea, for a number of reasons.
The first reason is the First Amendment of the U.S. Constitution. This would inhibit free speech by anyone who wants to send mail to anyone else. You know how you love to have port 80 blocked to your computer, don't you? This would continue the terrible trend of allowing read-only Internet access. You can read all you want, but if you want to upload anything or enjoy the pleasure of having unfettered bidirectional Internet access, you are going to have to pay $10 a month for an IP address, plus a BS charge of $300.
In it's most expensive form, an IPv4 address from ARIN costs about 7 cents per month. Granted you have to buy in bulk, but all ISPs do. So why can't you have a routed allocation if you meet the requirements for BCP12/RFC2050? Network operators are lazy and arrogant -- I know, I used to be one. I used to be an engineer at Global Center and GlobalCrossing.
It is absolutely not an ISP's responsibility to filter packets or frames based upon any protocol or service -- that is your job. Furthermore, most Internet routers simply could not perform with such requirements. If you want to pay your ISP to waste clock cycles and memory to block ports for you, you may ask. Or maybe just you could just get a firewall instead.
The reason that your suggestions are never mentioned as a cure for spam is because they would not work.
If you want to isolate yourself from the Internet and prevent yourself from ever being able to run your own DNS, STMP, HTTP, IMAP, and other servers off of your Internet connection (like I do), you may do so upon your own discretion. But please don't give the (dis)service providers any new ideas. Things are bad enough as it is.
I use e-mail autoforwarding to track spam. Every time I give my email address, I specify who I'm giving it to, ex. blah.com goes to blahcom@mydomain (anything@mydomain goes to the same hotmail box), so when I receive a spam, I can see which site sent it or sold the information, and block any e-mail coming from that site and everyone they sold it with To: line filters. Since most of the sites I wish to receive e-mail from are sites that don't spam me, this method has been successful in eliminating the vast majority of spam that I receive, down to only about 1 piece per day.
Your email provider delivers an email to you only if
it has a "Reply-To" field in the header AND
the Reply-To value has been accepted as a valid email address by another customer.
So in order for a person that just created an email address to email you, they would have to get their new address validated first and would receive a message to that effect the first time they tried to email you. They would have to get in touch with you or someone else under your email provider to get validated.
If you get some spam, you report it to your email provider and the ISP deals with the customer who validated the "Reply-To" address.
Email providers would set up peering relationships wherein they can share validated email addresses.
If the Reply-To value is faked, it would have to point to a validated email address and would probably bring severe damage to that email account. This method would push spammers into using this strategy, but it would certainly get them into more trouble that they currently get into.
I'm sure there are holes in my idea, so shoot away and educate me.
Rank comments and posts against each other at We-Rank.com
I disagree. I filter spam at the server level. My record for one week is now up to a little over 230k pieces of spam. Bigger than many but a drop in the bucket compared to some. I can't recall the last time I actually received a piece of spam from an AOL IP. I can't remember the last time I saw an AOL address used as a spammer's dropbox either. It's been a long, long time. AOL isn't a source of spam anymore. Tier-1 providers are the current problem. Those places are so big the people in the know aren't involved in the signup process for new customers. They continually sign up spammers and don't realize it. And when it is brought to their attention, they are practically impotent when it comes to dealing with the problem at hand. They do nothing until a widely used DNSBL like SPEWS or the SBL (SpamHaus) list them. Then the provider gets a clue, but not until DNSBLs get a little more bad press. What we need to do is clue in the larger carriers. We are after all ultimately their customers.
is for users to charge $.01 per spam they receive, to be collected by (and split with) their ISP. Users can have 'white lists' which, if the sender is on, means they can send without charge. Users will get docked $.01 for each email they sent to a non-whitelist destination, and creditted $.01 for each email they received from a non-whitelisted customer. ISPs automatically filter out any email that is not on the user's whitelist and comes from an ISP that is not part of the system. ISPs monthly will 'settle up' with each other each month by transferring the balance in cash. ISPs will have an incentive to join this system, both because they'll make money, and because users will patronize ISPs who join.
- I think the key problem is ISPs that do not block egress traffic on port 25
And think a big part of the problem are the nuts who think filtering port 25 network wide is a viable option. Here are some real world numbers...Router #1:
30 second input rate 21782000 bits/sec, 6210 packets/sec
30 second output rate 12294000 bits/sec, 4651 packets/sec
Router #2:
30 second input rate 7543000 bits/sec, 2133 packets/sec
30 second output rate 12182000 bits/sec, 3183 packets/sec
(and that's business traffic at 0030ET Sunday -- it goes a lot higher during business hours.)
Routers have a lot of work to do already without having to look for spam. Devices along the lines of a Packeteer could be used to perform in-line packet inspection, but that'll get old real fast.
Yes, it's perfectly doable to filter dialup users either at the ppp line or the next hop router by either explicit blocks or redirection. Many ISPs already do this. (UUNet requires it, oddly enough.) But an equal many don't. Plus, there's a growing amount of broadband in the world.
Most companies buying network connectivity and hosting their own email systems expect them to have direct control over those systems and the routing of their email in both directions. It's a simple task to set a mail server to use a "smart host", but then one is at the mercy of those controlling that server(s).
Oh, and just how exactly will this stop them from sending spam? Exactly. Simply put, it won't. It just changes the origin of the spam and maybe speed up the response time for blocking it and dealing with the user. HOWEVER, it introduces a much larger annoyance: blacklisting of the ISP server(s) and thus hundreds or thousands of companies and/or users.
Next I suppose the ISP should be looking at the email to judge it's spamliness? Well, I'm gonna have to play my lawyer card on that bit of stupidity. The instant an ISP begins any type of content filtering, most of the protective provision of various laws cease to apply. In the eyes of the law, this would be exactly the same as the post office opening all of your mail to determine and discard what they feel is "junk mail".
In the end, spam is what it is because of the [censored] creatans who think they can make money by participating in any of a growing number of scams. Basically, technology cannot protect the internet from stupid people. (esp. when the standard was constructed in a "stupid people" void. I guess we've bred better idiots.)
And I've tried sending spammers a bill for $50/email.
I use Charter for cable internet access. The day they start blocking ports is the day I leave them.
You can't judge a book by the way it wears its hair.
Compuserve bought The Source and then AOL bought Compuserve. :^P
There isn't enough spam.
Eventually, if spam is allowed to proliferate, we will all live in a world with lower APR on our credit cards, countless anonymous women in love with our cocks are that have grown 4" bigger guaranteed.
Enough of this conservative conspiracy.
On a serious note, I hate arbitrarily blocking ports. It won't do shit to stop spam, it's more about the ISPs wanting to block all the ports possible, to reduce the amount of traffic an end user can have.
I don't need no instructions to know how to rock!!!!
here's a good reference to the conference webcasts so you can skip to the section you want to listen to...
:)
Session 1
0:00:30, Teodor Zlatanov, spam.el Maintainer, "Gnus vs. Spam"
0:10:00, Bill Yerazunis, MERL, "Sparse Binary Polynomial Hash Message Filtering and The CRM114 Discriminator"
0:32:30, Jason Rennie, MIT AI Lab, "Adaptive Spam Filtering"
0:52:00, John Graham-Cumming, POPFile, "The Spammers' Compendium"
Session 2
0:00:00, John Draper, ShopIP, "Following Their Patterns"
0:14:00, Paul Judge, CipherTrust, "The Case for Spam Research Infrastructures"
0:37:00, Paul Graham, Arc Project, "Better Bayesian Spam Filtering"
0:56:00, Robert Rothe, eleven GmbH, "eXpurgate: a different approach in filtering E-Mail and detecting SPAM"
Session 3
0:01:30, Matt Sergeant, MessageLabs, "Spam Filtering at the Network Level"
0:21:30, Barry Warsaw, Pythonlabs at Zope Corporation, "Anti-Spam Techniques at Python.org"
1:05:00, Jean-David Ruvini, e-lab Bouygues SA, "Smartlook: An E-Mail Classifier Assistant for Outlook"
0:41:00, Barry Shein, The World, "Spam: Threat or Menace? An ISP's View"
1:23:00, Eric Raymond, Open Source Initiative, "Lessons from Bogofilter"
1:44:30, Joshua Goodman, Microsoft Research, "Spam Filtering: From the Lab to the Real World"
Session 4
0:00:00, Michael Salib, MIT, "Integrating Heuristics with n-grams using Bayes and LMMSE"
0:22:00, David Lewis, Independent Consultant, "Forty Years of Machine Learning for Text Classification"
0:34:00, Jon Praed, Internet Law Group, "How Lawsuits Against Spammers Can Aid Spam-Filtering Technology: A Spam Litigator's View From the Front Lines"
1:01:30, David Berlind, CNET, "Desperately Seeking: An Anti-Spam Consortium"
1:26:30, Ken Schneider, Brightmail, "Fighting Spam in Real Time"
1:47:00, Panel Discussion
thanks to schmelzle.net for the table.
Large print giveth, and the small print taketh away
... is that you can get a message from anywhere without any real challenges or permissions involved. I honestly think that work needs to be done to replace email on both the client side and the delivery/protocol side. I'd go into detail about how that'd work, but I really wouldn't be suggesting anything new. I just want email to be more like instant messaging. "You want to message me? Well, first I have to authorize you..."
Fortunately, it's not a burning issue with me. The people I really want to hear from are all on IM. Anybody outside of that has filters that expressly let them through.
i am glad this conference occured, but i am afraid their efforts is being blocked on the political side of things by the PEL.
The PEL you say? why of course the Penis Enlargement Lobby! Read this: Anti-Spam Legislation Opposed By Powerful Penis-Enlagement Lobby
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Bayesian filters could just as easily be written in Perl
:)
enter my fave project, popfile.
Large print giveth, and the small print taketh away
Will we get Shakespeare? Will it take a million years? Stay tuned, same slash-time, same slash-channel!
Port 25 egress blocking is a good start to the spam problem for two reasons: First, it prevents a spammer from signing up and just doing direct-to-MX spam from that throwaway account. Not many spammers do this anymore, because its easily tracead and bigger ISPs kick those accounts fastest. Second, it limits a spammer's ability to abuse open proxies and relays on a network. Say clueless users are running a WinGate open proxy or an open sendmail relay on an older default Linux/BSD install on their cable or DSL line. A spammer could try to relay spam through it, but the egress block would stop it.
I see alot of complaints here about how such a block prevents you from running a mail server on your broadband line. People, this is residential service you are getting here. If you need to run your own mail server you need to find out about that when you sign up for service. A typical residential user never needs to connect to any SMTP relay except the ones the ISP provides. These users are also more likely to cluelessly leave their computers open to abuse. If you're responsible enough to run a mail server, and you really NEED one, get a real account.
Another option is to relay your mail over a non-standard port through a third-party email provider, if you really loathe your ISPs relays. This is my situation, and I use Lux Scientiae. They run a SMTP AUTH relay on a secondary non-standard port. It's locked down to prevent abuse, and SMTP AUTH lets them track down any of their users that abuse it. They don't accept incoming mail on that non-standard port, only relay for users, so it's not like they're re-defining SMTP to use a different port.
Of course, there will always be those ISPs that really don't care about preventing abuse. This is why blocklists even exist, to allow users to shut out the bad neighborhoods on the net. It would be nice if all those residential broadband users' computers couldn't be hijacked by spammers. As it stands, they are, so one way or another port 25 traffic is blocked.
Beer wants to be free
a) short messages don't get caught- no words that are going to be blocked, just a URL. The URL doesn't match because it's several words stuck together without spaces.
b) misspelt words don't get caught. If the spammer deliberately misspells the key words, then it goes through.
c) common words- if the spammer only uses common words, it is unlikely that the spam can get caught; the spammer can check all the words he uses for being common before he sends it.
d) pictures- if the spammer sends his advert in a GIF, the Naive Bayesian can do nothing.
Overall, I am pessimistic about whether filtering will work in the long run, but in the short run it works pretty good.
-WolfWithoutAClause
"Gravity is only a theory, not a fact!"I used to run a tiny ISP. What I did was *redirect* traffic outbound to port 25 to a local mail server. The mail would still be delivered, and that server was (obviously) set up to allow 3rd party relay from the correct set of addresses. I had a small customer base, but I never once had any complaints about this policy. The users could forge the From: header all they wanted, but the outgoing mail would always have a proper Received: header, at least.
As long as the mail server doesn't do anything more agregious to the mail than add a Received: header, I find it unlikely that any legitimate complaints could be made about this practice. It's certainly a much more gentle answer than simply blocking port 25 egress completely. At least this way it's more or less invisible to the end-user.
The problem with changing SMTP is that it's well-established and generally a good protocol. The problem with changing the default configuration for installation is it only affects new installations. Basically anything you propose which requires changes on the server, requires operators to agree. No strategy as such will work, unless operators are not given a choice, because their customers demand the upgrade.
I'd propose a slight change to SMTP servers so that they automatically block incoming mail from other servers that act as an open relay. It would not discriminate against open relays when sending mail, however.
What this does is effectively drops all users of open relays off the map. Once enough servers out there start doing this, all the open relays start getting fixed, because their users demand mail to stop bouncing. Open relay spam ceases to annoy everybody behind a protected server immediately, however, and you don't really care when or if those servers get fixed.
This isn't going to fix the general spam problem, where valid addresses are used for spam, but at least you can block domains that annoy you.
But the truth is, spam will never calm down until every unsolicited/untrusted message costs a nominal sum, which curteous people return in the form of a reply from valid messages.
Any connection between your reality and mine is purely coincidental.
If a spammers is just taking advantage of an open relay, having 25 cut off from them will stop them, but "Big guys" like Ralski won't be harmed because they'll be using their own 'legitimate' machines overseas.
autopr0n is like, down and stuff.
Here's what you do instead: you configure all the email servers to take the FROM address specified in the SMTP exchange itself, then look up all the MXes for the domain the FROM address claims to belong to, then compare the actual address the connection is coming from to the list of addresses you just got back. If you don't get a match, you drop the connection right then and there.
End result: anyone who is running their own domain or who is using a legitimate mail server is able to get through, and nobody else is. Suddenly most open relays become totally ineffective. Spammers now have to go to the trouble of acquiring a domain and setting up MX records, and if they don't have a static IP then they'll have to use a dynamic DNS service. End result: killing a spammer is as simple as telling their dynamic DNS service to shut them down.
If there needs to be a way to differentiate between email receivers and email senders, then define a different type of MX record for email senders and do a lookup on them as well.
Thoughts?
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
After having some problems with my mail server, I re-enabled my hotmail account and I noticed a lot of spam these days consist entirely of graphical images, with some random, non-sensical words thrown in for good mesure. I don't think it will be long before you Bayesian filter will be obsolite.
autopr0n is like, down and stuff.
Legislation is not the answer. We know how tech-savvy politicians are. Do laws stop corrupt CEOs from plundering corporate pensions or cooking the books? Do laws solve problems?
Terrorizing spammers is not the answer. Again, this is not solving the problem. Pestering less than intelligent people who exploit less than intelligent methods of mass communication does not solve the problem. It might be a thrill short term, but there are too many people who will spam if the current mail protocols persist.
So what is the problem? Strangers send me e-mail I don't want. What is the solution?
I won't pretend to be an expert. I'm not. However, I'm surprised better men and women have not come up with something, ANYTHING, to solve the spam problem. I am NOT suprised to see 90-100 unsolicited e-mails (from strangers) in my inbox every day. Somebody needs to come up with something. So here goes...
First, classify e-mail accounts. Home/personal accounts should be bulletproof. You only receive messages from people you have on your list of acceptable senders, your "inner circle." Shopping/e-commerce accounts: you can receive messages from merchants who register with some central agency/server. Business/work accounts: I dunno. Ideas? How should we handle mailing list type accounts? Second, every e-mail sent has something solid identifying it with a sender included. The identification is sent to the recipient. If the recipient has this identification in his list and it matches 100%, then the recipient fetches the message from the sender. So instead of the sender wielding the power, the potential recipient makes the call. Why allow just anybody to send an entire friggin' message to scores of people? Messages go no where until the recipient says so.
Finally, and this is where the law comes into play, if someone manages to fake out your list by saying he is someone he is not, sic the prosecutors on him. That's identity theft, pal. As it is now, e-mail headers are raw schitzophrenia.
So step one, classify e-mail accounts. Different classifications have different list of people you are willing to accept mail from. Step two, the sender sends his identification and maybe a subject header to the recipient. Step three, the recipient accepts the senders request and fetches the message himself, rejects it outright, or adds the sender to his list and fetches the message.
I don't know 90 people whose mugs I'd piss on if they set themselves on fire. Why should any of these rat bastards be able to dump a second or third bit in my inbox?
Quoting from Tim Peters (the real TimBot, but I digress <wink>) spambayes has a more effective classifier:
While spambayes isn't really Bayesian (anymore; it started out roughly that way, as I understand it), the name stuck
It is a statistical filter, and you do need to train it with your personal collection of spam and ham. However, most of the work in the last month or two on the spambayes list has revolved around building user interfaces, and finding appropriate places to inject the filter into your mail processing, not on improving the classifier.
"Spambayes. Try it. You'll like it!" (But be warned, it's still pre-alpha...)
It IS a terrible idea; if you want to offer a public data service, then that's what you offer. You don't get to make exceptions just because you feel like it, unless you are declaring, in essence, that you are providing the service of selectively restricting traffic. And in that case, you become liable for every judgement you make about who to service and who not.
A bar/pub/saloon can restrict you all sorts of ways just because they feel like it. But this doesn't give anyone the right to stop you from getting drunk, trying to pick up strangers, or making a fool of yourself in public. A public communications service is different, and for a very good reason. bars and saloons are primarily there to provide a space for private associations; a communications infrastructure is there to provide a public infrastructure. and the internet points this out very well; it's public, accept the fact or build your own fucking internet.
It comes down to this; you are advancing the idea that the primary argument is "it's mine, i can do whatever i want with it". but in the interest of creating a just society (one where few people have an interest in destroying it), we recognize many "level playing field" exceptions to this. separate water fountains, "whites only" policies, etc. tell me who and what you are and i'll tell you how you depend on this fundamental fairness. i'll also point out that the internet isn't yours and if you can't play by its fundamental rules of openness then you have no business connecting to it.
Expanding a vast wasteland since 1996.
My guess is one day we'll see a web of trust used by our e-mail client to determine whether our e-mail gets delivered to our inbox or junk-mail folder.
Someone using a signature for spam would see himself removed from the web of trust, and those that verified the person as a non-spammer.
Just don't ask me how somebody that doesn't know anybody else with an e-mail account gets somebody else to vouch for him. (Maybe your ISP will vouch for you if you verify yourself with a CC or something?). Any thoughts?
Speaking of which. Spammers are getting more sophisticated.
----Stories you'll not see on "/."
Spain uncovers hi-tech cashpoint fraud
First-ever dividend for Microsoft shares
Microsoft's privacy officer resigns
GameSpy could let crackers mount network DDoS attacks
I chose my ISP because they let me run my own domain for web or e-mail or whatever else. If you think for a second that they should force me to use their SMTP servers, you are missing something important. My mail server delivers directly to the recipients of mail. It doesn't relay. It only serves me. In order for the ISP to provide equivalent service, they would need to host e-mail for my domain as a virtual domain on their server. This seems like a service that would cost me more. I'm already paying a premium price for an AUP that I find acceptable, I shouldn't have to pay more for service that I could provide myself.
By the way, I love my ISP - their customer service is top notch and they are kind enough to provide me with reverse dns service. I don't think you can beat that.
I have had other ISPs decide that they should block incoming mail from all subnets of my ISP. This made it difficult to send e-mail to my mother, and they were very difficult when trying to resolve the issue. It was finally resolved, but they never got back to my request for information as to why they took their course of action or what I could do in the future to expedite the correction of that.
Not intended as a post but the article links "The World" to http://www.std.com rather than http://www.TheWorld.com not a big deal but it'd probably be more resonant if fixed.
-Barry Shein, World Leader
It is mentioned just about every time the subject comes up, but then it gets drowned out by the Crips and Bloods... err Black Hole advocates yelling at the legal advocates.
The sooner people stop arguing about social vs legal solutions the sooner a technical solution will arive.
i finally managed to set up my Mail::Audit filters
to include spam-filtering by the so-called Bayesian filter
program bmf. It trains on spam/ham corpora
and uses the word frequencies to compute a
'spamicity index'. Seems to work for me
with pretty large numerical separations between spam
and non-spam. I trained on about 1200 spam and
several hundred 'ham'.
At QuikNet.com, I can go to a webpage and enable spam blocking. When it detects spam, it modify's the subject line and added "Possible Spam [Accuracy: #]". This makes it easy to filter into mailboxes.
I work at belgians largest Cable ISP (Telenet. We block port 25 for our customers. This not only reduces spam but stops virusses to spread trough built in smtp engines.
This whole discussion is getting really old, really fast. Let me try this on y'all:
SPAM IS NOT A PROBLEM.
See? Problem solved. New protocols, micropayments and blacklists: now THOSE are problems. But SPAM? It is EASY to filter spam. I really don't see the point and I hope all of you who make a big problem out of it just drop off the face of the earth or something.
If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive).
If it did reduce spam by 50%, it would only be for a very short time. The vast majority of spammers that would be foiled by this would simply find another way to send their spam. So in the end you'll have pretty much the same amount of spam, but you'll inconvenience all of the people who had legitimate reasons for using port 25. Doesn't sound like a great idea.
These will *never* work. The reason spam works is because it is very cheap and easy to send, and because all their responses are from people who want to buy.
The answer is incredibly simple. Every time I get a spam, I use my fake free e-mail account to send them a message that simply says, 'boy am I interested!' and do my best to get a real person to spend as much time with me as possible before blowing them off.
I always make a point of keeping telemarketers on the line as long as possible. I have come to enjoy it. Make them repeat their spiel several times, ask pointless questions, and talk them into letting you put them on hold. After wasting 20 minutes of their time, I tell them that I had no intention of buying, I just wanted to waste their time. The resulting colorful language is music to my ears.
Just think, if for every 100,000 spams you sent out you got 50,000 fake replies that were indistinguishable from genuine buyers. You would spend so much time weeding out the fake replies that your business would become instantly unprofitable.
Best of all, this requires no additional legislation, technology, etc. It is 100% legal and ethical to request more information and swamp them with individual inquiries. Further, there is absolutely no countermeasure to this that spammers can use. If you want to stop spam, you must make it unprofitable. This is the only way to do it.
Somewhat related is this approach I've been trialing quite successfully for the last month. I haven't been able to find any reference to anyone else doing this, and would welcome any comments. If it's a 'new site' (not dealt with regualrly and not seen recently) and it shows up clean on the variosu DNSBL's I use, then I send a temporary error code back. If they retur (after a suitable time delay - I use 15 minutes) and still come up clean, then I let it through. Advantages: * many spammers don't retry - ever (perhaps they get shut down, or someone closes their open relay, or they concentrate on more receptive targets) * those that do retry (often many hours later - average is 7.6 hours for spammers) are usually listed on the DNSBL's by then * I get to collect the list of mail addresses they are trying to send, and if they hit one of my spam traps (and there are many obvious dictionary attacks) then they immediately get marked bad even if they are not DNSBL'd * Doesn't waste bandwidth (or the hijacked resources of a open relay 'victim') which continually using a tar pit does Disadvantage * Genuine email from a new/infrequent source gets delayed 15 + (until their servers retry) minutes. Most geuine ISPs try at reasonable intervals - though some wait an hour. I'm willing to wait an hour for mail from someone new, who's not on my whitelist, given the amount of spam this simple technique filters. Obviously if everyone adopts this approach then spammers would deliberately work around it - but it would complicate matters for them - the time delay and reptetive nature of their attempts would make them even more obvious as spammers, and more easy to shut down. And they can't avoid the spam traps. Forgive me if this is obvious and well known - I'd appreciate any pointers to where this has been applied and any comments.
Recycle PCs and build a wireless community network www.hillsborough.org.nz
Short answer: Because you teach your filter what is spam. And everybody else teachers their personal filter.
So if your personal mail often has "penis" in it, the filter learns that it is not a good indicator for spam.
I use POPFile http://popfile.sourceforge.net/, and I have noticed that one of the best indicators seems to be certain server names.
Everyone has ideas, but who is doing anything about it?
There are plenty of people reading this who are dedicated to the Open Source movement. Can we not make a start? I'm sure there are plenty of worthy projects on FreshMeat etc. What we need to do is ENDORSE one of them and start using it. If all /.ers were to start using a secondary method of email communication, perhaps others might start to take notice.
There's no-one who's going to one day say, OK, we're now going to use some other protocol. This is going to be driven by the grass roots.
Anyone got any sensible ideas on how we make a start?
Spam is like the "war on drugs". It will continue as long as the spammers can make money doing it! It doesn't matter what we do, they will continue until they can no longer make money. period, end of story.
Turning my notes for the day into something vaguely coherent, here are some hightlights from the proceedings. There are a couple of speakers that I didn't write anything down for, but from mid-morning on this should be pretty comprehensive. Apologies in advance if my notes lead me to attribute certain comments to the wrong speaker -- if anyone notices any mistakes please feel free to add corrections:
Bill Yerazunis - CRM114 & MailFilter
Because Perl "freaks him out", Yerazunis came up with the CRM114 minilanguage (points for anyone that gets the joke in the name without googling for it :), then wrote MailFilter in CRM114 as an implementation of a filter that can be used with Procmail or SpamAssassin or what have you. The basic idea is to decompose a message into a set of "features" composed of various permutations of single words, consecutive words, words appearing within a certain distance of one another, etc, such that the set of features N is very much bigger than the set of words X. You then analyze the features in various ways and if you get above a certain arbitrary threshold, you flag the message as spam & handle it accordingly.
He claimed that with this software he could get better than 99.9% accuracy in nailing spam, and a similar percentage in avoiding "ham" (the term everyone was using for false positives -- legit mail that was falsely identified as spam). One of Yerazunis' observations is that the best way to defeat the spam problem is to disrupt the economics: if a 99.9% or better filter rate were to become the norm, then the cost of delivering spam can be pushed higher than the cost of traditional mail and the problem will naturally go away without requiring legislation (which would be nice anyway, but we can't count on it).
The drawback of CRM114/MailFilter is that it can only handle about 20k of text per second, so it's not appropriate for large scale use yet. Still an interesting project to watch though: crm114.sourceforge.net
John Graham-Cumming - POPfile
Most of his very entertaining talk was about the ingenious tricks that spammers resort to to obfuscate spam against filters, including most diabolically one example that placed each column of monospace text in the message into an HTML column, so that the average HTML-capable mail client would render the message properly, but it would be absolute gibberish to most mail filters. The ultimate lesson was that any good filter has to focus not on "ascii-space" (the literal bytes as transmitted) but the "eye space" (the rendered text as seen by the user), which by extension may mean that any full scale spam parser/filter could also have to include a full-scale HTML & Javascript engine. Yikes!
As for Graham-Cumming's software, it's a Perl application, available for all platforms (Windows, Mac, & of course Linux) that allows users to filter POP3 mail. Interesting stuff if you're a POP user: popfile.sourceforge.net
John Draper - ShopIP
Most of Draper's work seemed to be focused on profiling spammers, as opposed to profiling spam itself, by throwing out a series of honeypot addresses & using data collected to hunt down spammers. spambayes.sourceforge.net
Paul Judge, CipherTrust
Judge's big argument, which no one really disagrees with, is that spam has become not just a nuisance, but an actual information security issue. To that end, he is advocating much more collaborative effort to address the problem than we have seen to date: conferences like this, mailing list discussions, better tools, and public data repositories of known spam [and ham]. To that last point, one of his observations (which others made as well) was that there are no universally agreed on standards for what qualifies as spam, so repositories for spam will not be accurate for all users (spam for your programmers will be the bread & butter of your marketing department, etc). Plus, there are obvious privacy issues in publishing your spam & ham for public scrutiny. And to add another wrinkle, one danger of public spam/ham databases is that spammers can poison them with false data, screwing things up for everyone. That said, he encouraged users to help out with building spamarchive.org.
Paul Graham
The man who organized the conference and kicked everything this week off with his landmark paper from last fall, A Plan for Spam. Graham's spam filtering technique famously makes use of Bayesian statistics, a technique popular with nearly all of the speakers. The nice thing about a statistical approach, as opposed to heuristics, simple phrase matching, RBLs, etc, is that they can be very robust & accurate; the down sides are that they have to be trained against a sufficiently large "corpus" of spam (most techniques have this property though) and they have to be continually retrained over time (again, this is common). Graham was too modest to produce numbers, but subjectively his results seemed to be even better than what Yerazunis gets with MailFilter, by an order of magnitude or more.
Like other speakers, he predicted that spammers are going to make their messages appear more & more like "normal" mail, so we're always going to have to be persistent about this -- as one example, he showed us an email he received IN ALL CAPS from a non-English speaker asking for programming help, and although it was legit, the filters insisted otherwise. "That message is the one that keeps me up at night."
Everyone interested in the spam issue should go read Graham's paper immediately.
Robert Rothe, eXpurgate
Rothe works for Eleven, an ASP company from Berlin selling a spam management service/application called eXpurgate. His talk was short on details about how the tool worked (mainly that it searches for bulk mail), focusing instead on the high level functionality it provides to users -- basically, they classify mail as safe, questionable, or dangerous, and let the users handle them accordingly. Another speaker that sees spam as a network security issue, so they built their system accordingly, with privacy of the client's mail content in mind etc.
Like many speakers, he warned about the dangers of an anti-spam "monoculture": that Bayesian techniques might be great, but if that's all anyone uses then spammers will catch on and adjust their messages to look more like normal mail, to the point that Bayesian filters won't work anymore. As a result, we're going to need to attack the problem from several angles, using different techniques, to keep the spammers off balance as much as possible.
Matt Sergeant, SpamAssassin
SA is a well known Perl application for heuristically profiling messages as spam, adding headers to the message saying for example "I am 72% sure this is spam because it has X Y Z", and passing off the message to procmail or whatever to be handled accordingly. SpamAssassin can handle a message throughput great enough that it can be deployed at the network level (whereas some of the others, which might have somewhat better hit rates, are still too inefficient at this point). Deployed this way, the differences in effectiveness for single vs. multiple users becomes very apparent, as 99% effective rates fall down into the 95-80% range. This happens because, again, different users define different things as spam, so mapping one fingerprint to all users can never work quite right. For an example of a tool that your company can deploy right now & get fast, decent results, SA looks like a good choice; but for the long run it looks like a Bayesian technique is going to get better performance, and SA is adding a statistical component to its toolkit. Good talk.
Barry Warsaw, Python Labs
This was another example of the "monocultures are dangerous" philosophy, as Warsaw explained how he is helping to use a variety of anti-spam techniques -- from clever Exim MTA configuration to good use of Spam Assassin & Procmail to fine tuning of the MailMan mailing list engine -- to work together to manage the spam problem for all things Python (Python.org, Zope, many mailing lists, a few employees, etc).
He pointed out that some very simple filters can be surprisingly effective: run a sanity check on the message's date; look for obviously forged headers; make sure the recipients are legit; scan for missing Message-Id headers; etc. In response to the person that originally posted the article, yes, he did mention blocking outgoing SMTP as an effective element of a many tiered spam management approach.
Among other tricks for getting the different filtering tiers to play nice together, they make heavy use of the X-Warning header so that if an alarm goes off in one tier of their mail architecture, other components can respond appropriately. Cited projects included ElSpy and SpamBayes.
Barry Shein, founder & CEO of The World -- or as he laughingly put it, "President of the World". Har har har
This talk was mostly a let down for me -- Shein has made his views very well known, and his ranting, rambling talk didn't really introduce any new ideas for anyone that had read that interview (some good jokes & quotes though).
His core argument is that spam is "the rise of organized crime on the internet", that filters are nice but that the mail architecture itself is fundamentally flawed, and that ISPs like his -- in 1989, The World was the world's first dialup ISP -- are being killed by the problem. Shein was very annoyed that all these talented people are having to clean up a mess like this when we should be out working on more interesting stuff, and not having to worry about this issue. His big hope seemed to be that legislation will someday come to the rescue, but he sounded very pessimisstic. (Others in the room seemed to feel that this was a very interesting machine learning problem, and weren't really fazed by his pessimism -- but then most of the people in the room don't run ISPs.)
He also suggested that we need to find a way to make spammers pay for the bandwidth they are consuming (rather than having users & ISPs shoulder the burden) but didn't seem to know how we might go about implementing this. At all.
Fun rant to cheer along to, but for me it wasn't very constructive in the end.
Jean-David Ruvini, eLabs SmartLook
This was an interesting product. Ruvini's company is developing an extension to Outlook 2000 & XP that will watch the way users categorize messages into folders, come up with a profile for what kinds of messages end up in which folders, and then try to offer similar categorization on an automatic basis. Think of it as Procmail for Outlook, without having to mess with (or even be aware of!) all the nasty recipies.
Obviously if you have a spam folder, then spam will be one of the categories it looks for, but more broadly it will try to categorize all your mail as you would ordinarily categorize it. This makes SmartLook a broader tool than "just" a spam manager.
SmartLook is another statistical filter, though it uses non-Bayesian algorithms to get results. eLabs' tests suggest that the product is able to properly categorize messages about 96% of the time, with no false positives, and (for their tests, mind you) that it performed better than Bayes filters over three months of usage.
One nice property of this tool was that it works well with different [human] languages -- some strategies fall apart &/or need retraining when you switch from English to some other language. For certain markets (eLabs seems to be a European company, perhaps French?) this is a crucial feature, and having a tool that works with one of the biggest mail clients out there (most people don't use Mutt or Pine, sadly enough) can be very valuable. Very clever -- watch for the inevitable embrace & extend three years from now.
Eric Raymond
He didn't say anything about guns, but he did try to correct one of the other speakers for misusing the term "hacker."
Like Graham, ESR is a Lisp fan, but he knows that the vast majority of people aren't, and he also knows that the vast majority of people need to be using something like Graham's spam software. So on a lark, he came up with a clean version in C, named it BogoFilter, and put it on Sourceforge, where a community sprung up to, well, embrace & extend it.
As good as Graham's Bayesian algorithm is, ESR felt -- as did many of the other speakers -- that the nature of your spam/ham corpus is much more significant than the relative difference among any handful of reasonably good algorithms. (Back to the often repeated point about how corpus effectiveness falls apart when used for a group of users, as opposed to individuals.) To that end, he strongly feels that the best way to deal with the spam problem is to get good tools into the hands of as many people as possible, and to make them as easy to use as possible (ahh, the old "open source UIs always suck" argument :). As an example, one of the first things he did was to patch the Mutt mail agent so that it had two delete keys: one for general deletion, one for "get rid of this because it's spam." That second key, and interface touches like it, seem like the way to get average people to start using filters on a regular basis.
Joshua Goodman, Microsoft Research
Unlike ESR, Goodman felt that algorithm selection does make a big difference, but this being Microsoft he refused to disclose what algorithms his team is working with -- except to say that, when delivered, they will be more accessible for average users than SpamAssassin, Procmail recipies, or Mutt :)
Microsoft has been working on the spam problem since 1997, but because of how big they are they've had unique problems in bringing solutions to market. As a case in point, they tried to introduce spam filters to a 1999 Outlook Express release, but were immediately sued by email greeting card company Blue Mountain because their messages were being inaccurately categorized as spam. With that in mind, they have been very reluctant to bring new anti-spam software out since then because they would like to see legislation protecting "good faith spam prevention efforts."
As a very large player, Microsoft faced certain difficulties in developing useful filters -- it may make sense for you as an individual to filter all mail from Korea, but this doesn't work so well if you are trying to attract customers *from* Korea :). This has forced them to put a lot of work into thoroughly testing different strategies before offering them to the public.
In spite of what millions of webmail users may have expected, Hotmail & MSN are currently being filtered by Brightmail's service, and plans are underway to reintroduce spam management features to client side software again. (Just imagine how bad it would be if they weren't paying someone to filter for them! Unfortunately, no hecklers piped up to ask if they are really selling Hotmail's user database to spammers, and if that is a source of annoyance for his team.)
An interesting barrier his group has had to grapple with was what he called the "Chinese menu" or "madlibs" spam generation strategy: that it's easy to come up with a template for spam -- "[a very special offer] [to make your penis bigger] [and please your special lady friend all night!" vs. "[an exclusive deal] [for genital enlargement] [that will boost your sex life!]" etc -- and have a small handful of options for each 'bucket' multiplying into a huge variety of individual messages that are easy for a human to group together but almost impossible for software to identify.
Michael Salib, extremely funny MIT student
Unlike nearly all other filter writers of the day, Salib's approach was heuristic: find a handful of reasonable spam discriminators, throw them all against his mail, and see how much he can identify that way. "It's sketchy, but this is a class project. I don't have to be realistic. [...] These results may be completely wrong."
Much to his surprise, he's trapping a lot of spam. He pulls in a little bit of RBL data ("the first two or three links from Google, whatever"), looks for some patterns and so on, and then churns it through LMMSE, an electrical engineering technique that as far as he can tell doesn't seem to be known in other fields. Basically this involves running the messages through a series of scary-but-fast-to-calculate linear equations). It turns out that he can process this much faster than a Bayes filter, to the point that customizing his approach for each user in a network would actually be feasible.
For a small spam corpus, he got results better than SpamAssassin did, though for a large corpus his results were worse; he couldn't really account for why this would be the case, or predict how things would scale as the corpus continued to grow.
When questioned about the RBL tactic by a member of the audience [who was apparently familiar to Salib -- I don't know who it was] about whether authenticating remote users might be the answer, Salib's response was "yes, I agree, but then you *do* work for Verisign, who is in the verification business, so you would say that."
Right on, Salib -- his talk was easily the funniest & breezy of the day :)
David Lewis, general researcher
The core of Lewis' argument, as ESR said earlier in the day, is that for any machine learning technique the quality of the learning corpus is much more important than the algorithm used. Bayes is one such algorithm, but there are many other good ones in the literature. In a dig at Goodman's refusal to disclose algorithms, Lewis pointed out that all of this has been publicly discussed since the first machine learning paper was published in 1961.
Observations: "lots of task inspecific stuff works badly, but task specific stuff helps a lot." It is important to use different corpuses [corpi?] for training and for general use, so that you don't train your machine to focus too much on certain types of input (this is a point that Microsoft's Goodman made as well).
As Graham did, Davis emphasized that spam is going to slowly start looking more like natural text, and we're going to have to deal with this as time goes on. www.daviddlewis.com/events/
Jon Praed, Internet Law Group
To a burst of tremendous applause, this talk began with the sentence "my name is Jon Praed, and I sue spammers."
He brought a legal take on the "not everything is spam to everybody" angle, emphasizing that we need a precise definition of what qualifies as Unsolicited Commercial Email (UCE). In particular, it has been difficult trying to pin down if the mail was really unsolicited, as this is where the spammers have the most wiggle room. However, if you can track down the spammer, they have to date rarely been able to verify that the user asked for mail, and so Praed has been able to successfully prosecute several spammers on this angle. He doesn't expect this to work forever though.
According to Praed, "laws against spam exist in every state, and more are pending", but he doubts that a legal solution will ever be completely effective as long as spam is lucrative. By analogy, he pointed out that people still rob banks and that has never been legal.
Praed informed the audience that there are several ways to get back at spammers, including injunctions, bankruptcy, and contempt, and all of these can be very effective. He pointed out that, to be blunt, a lot of these people are desperate low-lifes, and spam has been their biggest success in life. After these legal responses, their lives all get much worse. It hadn't occured to me to see spammers as pitiful before, but I can now. Most importantly, Praed stressed that these legal remedies can be very effective, and he strongly warned against taking vigilante action. This is almost always worse than the spam itself, and it only serves to get you in even deeper trouble than the spammer.
Identifying the sources of spam, most comes from offshore spam houses, abuse of free mail accounts (Hotmail & Yahoo, free signups at ISPs, etc) and bulk software (which may apparently soon become illegal in certain areas, provided that a law can be found to ban spam software while allowing things like MailMan or MajorDomo). Interestingly, he questioned the idea that header spoofing is a big problem, and claimed that in every case he has dealt with he has been able to track down the messages to a legit source sooner or later.
Suggestion: if you get a spam citing a trademarked product [e.g. Viagra], forward it to the trademark holder and they will almost always follow up on it. Suggestion: be fast in trying to track down spammers, as some of them have gotten in the habit of leaving sites up long enough for mail recipients to visit, but taking them down before investigators get a chance to take a look. Legal observation: spam is almost always fraud, and can be prosecuted accordingly.
Praed wrapped up his talk by citing the encouraging precedent that the famous Verizon Online vs. Ralsky case set: [a] that the court is interested in where the harm occurs, not where the person doing harm was when causing it (so if you send spam to someone in Alaska and spam is a capital offence in Alaska, you can be tried as a citizen of that state even if you caused the harm from somewhere else), and [b] it is assumed that you have to be familiar with a remote ISPs acceptable usage policies, and ignorance is no defence (just as you can't say "I didn't know it was illegal to shoot someone", Ralsky couldn't say that he didn't know Verizon prohibits spam -- (he had to have known that the AUP wouldn't allow what he was doing, so he deliberately didn't read it)). That precedent makes future prosecution of spammers much more encouraging. While, again, legal solutions may never eliminate the spam problem, a precendent like this can be an important supplement to filtering efforts (the stick to the filter's carrot, or something -- my lousy analogy, not Praed's).
David Berlind, ZDNet executive editor
His talk was primarily about how he receives a huge quantity of email from ZDNet readers, and he can't afford to use any spam filtering solution strategy that would allow *any* false positives. As one of the speakers said -- sorry, I forget who (Microsoft's Goodman?) -- getting a 0% false positive rate is easy: just classify nothing as spam. Getting a 100% hit rate is also easy: just classify everything as spam. Any solution besides those two is always going to have some degree of error either way, and determing how much of what kind of error you want to accept is up to you. Most users will tolerate a moderate false negative rate (some spam gets through) if it means that the false positive rate (legit mail is deleted) is very low. In Berlind's case, the false positive rate has to be vanishingly small, because reading all customer mail is a critical sign of respect for him.
Further, his business is also a legitimate mass emailer, sending out millions of free newsletters to users every day, and if Shein's proposal to bill bulk mailers were to catch on then even a very low rate would quickly put his company in the red. One obvious solution, which wasn't mentioned: start charging a subscription for these mailings, and make them profitable. I don't want to see this happen but if it did then the economics would tilt back toward making things feasible again.
Berlind is appreciative of the anti-spam work that is being done, but at the same time is skeptical of how pragmatic most of what is being proposed can really be. He feels we need a massive effort to rework the way mail is handled [Y2K anyone? It could get IT people back to work...], and to that end hopes ZDNet can help promote such a cooperative effort between the parties working on this. They don't want to be involved -- they are journalists & publishers, not standards developers -- but they are eager to get things going & want to cover the story as it progresses.
Like Shein said, he feels it's a waste for all these talented people to be working on combating penis enlargement offers, and hopes that we can find a way to get past this and work on real problems, "like world peace." This comment got a chuckle from the audience, but he seemed like the kind of guy that really meant that, and more importantly, he was right. A smart guy like Paul Graham or Bill Yerazunis shouldn't have to waste time tinkering with how many Viagra offers he can automagically delete when there are more fun things to be doing.
Ken Schneider, Brightmail
As mentioned earlier, Brightmail provides an ASP service for real time filtering of both incoming & outgoing mail. As would perhaps be expected, bigger ISPs and networks attract larger amounts of spam: 50% of mail coming into big ISPs and 40% coming into big companies is now spam. Brightmail offers the Probe Network, a <slashdot-killfile-term>patented</slashdot-killfil e-term> system of decoy honeypot addresses that gather data for analysis at their logistics center, which in turn distributes spam filtering rules to their clients where a plugin for $MTA (using the open source or proprietary MTA of the client's choice) can act on the database.
An interesting property of their system is that they have a mechanism for both aging out dormant rules as well as for reactivating retired ones, so that the currently active ruleset can be kept as lean & effient as possible. A big source of difficulty for them is legitimate commercial opt-in lists, because things have gotten more shady & blurry over time and it's now hard to tell this mail from much of the spam out there. Whitelists help here, but the problem is still difficult.
After each speaker had his turn, there was a panel discussion, but not much really happened there, and the moderator cut things short after only a couple of minutes. The original plan was for everyone to go out for Chinese food afterwards and continue the discussions over dinner, but when 580 people signed up that plan obviously fell apart. :) And so, here ends the notes...
DO NOT LEAVE IT IS NOT REAL
I have my own domain and run my own sendmail box. First my "real" e-mail address is very well gaurded. second, when I deal with any commercial activty I have a "BS" address that is simply redirected to my real address, when the spam gets really bad I simply change the alias. Personally I don't get spam, but thats just me....
stupid is as stupid deserves.......................
Blocking outgoing port 25 traffic is ridiculous. I should not become the victim because others abuse the system to send spam. I want full access to all ports, and I will never pay a provider that blocks access to any hosts or ports.
Not only do I wish to use my own mailservers (I don't right now, but I'm planning to because of overly restrictive rules on my providers outgoing smtp server), but I also would like to check access to remote mailservers I admin and do relay checks and scans on them from my home connection. Not to mention I'd be screwed royally next time my ISP's relay would crash again.
I understand this would stop a lot of spammers, but I think it's too high a price to pay. First came dynamic ips, then NAT's and subnets, firewalls, speed restrictions, download restrictions, blocking incoming ports (my cable ISP blocks everything 1024 incoming, even though it's a dynamic ip), content filters for PtP... some isps even try to enforce a policy that lets people only use the emailaddress provided by that isp! What's next, paying extra to access any ports other than 80 (http)?
No way. I think it's time ISPs finally spend some more time and money on the problem by checking for abuse of their own servers, instead of restricting the average user even more and taking away more freedom and transparency.
It's a useful approach that I don't think has been widely deployed.
since you are proposing enabling authentication,
why bother with moving the server to a new
port? just implement authentication (required
to relay email, any email that results in
local delivery needs no auth). use the starttls
protocol to encrypt just the authentication
portions of the transaction for security. then
you save yourself the hassel of setting up
a special smtp listener on another port.
I ws working on my own review, and if there is still interest in hearing what I have to say about the conferance, I'd be willing to submit it (and hope that it is posted).
In short:
Just about everyone was talking about filtering, which is at the wong end. Very few people where talking about the problems that spamming cause or solutions to end spamming. I say a few. There was talks by John Draper on taking down spammers and by Paul Judge of Cipher Trust on Spam Research, and one other person (Bill Y?) said that spam will continue until the cost of sending that email costs much more than it does now.
Here's where my opinion varies from the filting crowd, and I'll use myself as an example. I just returned from a 3 week holiday in Europe (visiting family and skiing - yeah, I know, I suck ;-) ), and in those 3 weeks I recieved 8,000 emails. No lie. Of the 8,000, 20 where worth looking at (7 where from airlines with deals of the week- ham in the spam language, 3 dealt with people asking where they could buy liquid nitrogen (see url if you have to know), which left me with 10 messages that where of any real interest to me.
The 7,980 emails which were spam should have been marked and labled as such before it ever reaches me.
That's the problem: the email system is going to die under the weight of junk mail being sent, and people are going to start to not deal with email at all becuase the usefulness of email has been ruined by the flood of junk mail over real mail. Filtering of email is after the fact and does not address the issue of the infrastructure failing under the weight of the crap.
So, what is the solution? Is there a solution? I hope so.
III.IIVIVIXIIVIVIIIVVIIIIXVIIIXIIIIIIIIVIIIIVVIII
How effective has it been to block port 80 in preventing web virii from spreading? I am getting scanned by my subnet neighbors all the time. Does the ISP do anything about it? No! There's no money in it for them. The only time they took any action is when it inconvenienced them
As for running SMTP servers, it's less load on their boxes so they probably wouldn't care.
Blocking port 25 would bust a lot of stuff on a typical Linux box and make the hobbyist hacker have to pay extra just to set up his own home domain email server.
And would it stop spam? No! Why? Because if I can find a way around it by sending email from another port you can bet your Salary that the spammers will to! And since they don't have to receive any email, they will set up systems with sweeping port scanners to avoid getting blocked again and again.
Blocking port 25 sounds like a solution proposed by a Politician and not a Technician.
Every where I read about spam mail the basic recomendation is that I never must reply to a spam mail -always ignore it (replying confirms theemail adress).
Because of that, only people who want to buy something from the spammer replies!! This makes it easy for the person or organisation who have something to sell, to use email as a direct marketing solution.
But what happens if 20 % procent or more of the targetted email adresses replies to the spam mail.
By clicking on the provided link, find a working email adress on the purcase page. -Send an email to this working adress, stating that sending spam is wery wrong.
This actuallly confirms your email adress, but what the heck! they have it allready!
Okay... I'd like to start out by saying that I loath spam as much as the next guy. Now I run a SMTP server off my DSL connection, and I like to use it. Now Earthlink, my previous ISP does block port 25. But their own server can only be used on their connection. So am I supposed to switch SMTP servers each time I travel? What if I want to use a SMTP server that does authenticate? Those are the problems with blocking port 25, and why I switched to Speakeasy, and am now happy. BTW, my SMTP server does require authentication, which forces it to run as root, making it more prone to security holes, but oh well.
:
(Machines Mentioned: firewall/server)
Pentium 100 with 48 MB Ram, 20 GB hard drive
Linux Kernel 2.4.18 and Debian 3.0
Apache/SSL and Exim using pam for SMTP Auth
Sshd and oftpd. Iptables firewalling/nat.
It's annoying enough that my ISP blocked INCOMING port 25 (so I can't receive mail locally) but now they want to block outgoing port 25 so I can't send it directly either? Sure, spam is a problem, but the only good way to deal with it is to replace the email system completely. I believe someone proposed an email-like extention for the Jabber protocol at one point with some nice NNTP-like characteristics...
Luke-Jr
So long as the ISP's mail server actually does properly relay the mail for you and doesn't do anything else out of spec, there's nothing (much) wrong with them forcing you to use it.
Big if, and that is why I run my own outgoing mailserver too.
Their mailservers suck, they require authentication for outgoing email, and they require I use their email address.
Much easier to setup an out only mailserver, and ignore their crap.
Then the ISP will knowingly be providing a service for the purpose of sending spam.
Which would make them a party to sending the spam for commercial gain (premium service) and some jurisidiction somewhere would hit them under anti spam laws.
Laws are social/economic solutions. When we take your money away and put you in jail, that would be attacking you on the social & economic level
I think Fahlman's suggestion http://www.research.ibm.com/journal/sj/414/forum.p df is more promising than any filtering/blocking scheme. It also doesn't suffer from "false positivies".
Now who's going to be the first one to set up an interrupt-token agency?
Personally, I think we could do a great deal to slow down spam if we stuck SMTP honeypots on port 25 and checked out mail from a different port.
Spammers would sit there an send all their mail to the honeypots with no idea that it's really going into oblivion and in the meantime, we'd be able to send out our mail (with some authentication, yes, please!).
I've actually been writing my own personal SMTP honeypot for this very purpose. I don't imagine I'll have much trouble getting friends to run it.
"Their aim is to find a spam filter so effective, that spammers would receive few, if any, responses, making sending unsolicited bulk e-mail a financially prohibitive task"
Ya know, I know how this quote was intended to be interpreted. But it presents an interesting line of thought if you read it differently: What if ISPs who implemented spam filtering (with something like SpamAssassin, not just a keyword filter) bounced back the replies that users sent to the spammer? And for spam that pushes you to a temporary site, what if the ISP put up some generic instructional page on what spam is instead of just pulling it down.
I think that a tiny bit of end-user education, put in place by just a few large ISPs, could really go a long way.
-Fatty
No, it is not. As a matter of fact, any spammer could do it. That is why you have to use filters... The idea that law will protect against this is as stupid as Micro$oft copy protection schemes. Everybody has to realize that the internet includes all of the world. There are countries were laws ar meaningless...
It's good to see you thinking about port 25 and SMTP. There's value in the sorts of thing you advocate and many ISPs have done them. Spam remains - it isn't enough. Same for filters and blocklists. Something more needs to be added - spam is growing (the growth of spam is in large part the result of the success of the other methods: the spammers must send more spam to get a return.)
m sn.co m[rab]: 7bit
8 04 80571000500500460990971090451090970460991111100991 01110116114105099046110101116058
One can complain on and on about systems with unsecured port 25. That isn't solving the problem - time to stop and take a look. Spammers send relay spam, to send relay spam they must find open relays, to find open relays they send test messages. Most email system managers can tell you that they see failed relay attempts in their logs.
How much more of a clue is needed? You see that the "secure the relays" campaign isn't working, you see that the spammers test and test and test, looking for open relays. Taking them away hasn't worked - try GIVING THEM OPEN RELAYS. Partly open - ones that only deliver their test messages. Now what do the spammers do? If deceived (right now they mostly are) they send relay spam to the fake relays, where it is simply stored (one can look through the trapped spam and find ways to cuase further hurt to the spammers.) It isn't delivered, the spammer doesn't make a dime from it. No matter how clever the spammer is in disguising what he's sending it doesn't matter - the trapping is based on the spam delivery path, not on its content.
Don't want to deliver tests? OK, don't even do that. Just set up a system that captures them. I have one, I have a test message that appears to have originated within a few miles of the MIT conference the same week as the conference:
Received: from ts009d22.cam-ma.concentric.net by X.X.X;
Wed, 15 Jan 03 17:56 CST
Message-Id:[lab]049049049049049049049049049@
Date: Wed, 15 Jan 2003 18:57:16 -1700
From: candy@webname.com
Subject: pick up the phone
To: porcha@SoftHome.net
MIME-Version: 1.0
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.00.3018.1300
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300
05405204605004604905204804604905505605811611504
I munged the system name and I changed the encoded IP of the system in the message-ID to all 1's. The numeric strings are decimal ascii: "048" is "0," etc. [lab] is a left angle bracket, [rab] is a right angle bracket.
Get enough people trapping spam and even more people trapping and reporting relay tests and you put pressure on the spammers that, up to now, they haven't felt. It is a wide-open area for using technical means to bring them down. Wide open - note that well.
In the SE part of the US is a system operator who has stopped spam to over one millon recipients so far this year.
When this idea is presented there's frequently someone who objects "but the spammers could simply put one of their own addresses in the list of spammed addresses and see if the mail got through. That way they could detect the honeypots." That's true. So far they do not do it. Even if they do there's no way they can eliminate the danger to them from trapped and reported relay tests. If they get clever and test only though open proxies then the center of activity shifts to fake open proxies. That's also easy. A couple of people (including Michael Tokarev, in Moscow) have had great success with open proxy honeypots. There's another objection that can be made. It can be overcome but I won't state it here and give spammers ideas.
Spam is not defeated. It would seem that every reasonable defense should be employed while that is true. Running fake open relays is fantastically easy and is very close to 100% perfect. Do it on an IP that has no legitimate port 25 traffic. Then everything that comes to that port 25 is spam or a relay test. Do it on a Windows system with no MTA - port 25 isn't even in use.
Run Jackpot:
http://jackpot.uk.net/
There are, uh, a few Windows systems out there?
An earlier expression of this same idea is indicated by sendmail -bd. At one time that would mean sendmail accepting but not delivering messages - now you have to od more to stop delivery attempts. In general, any MTA configured to accept anything but deliver nothing will trap relay tests. Force delivery of the relay tests before they're too old and you deceive the psammer who sent it.
END spam in 1Q2003. It has gone on far too long.
If you block port 25, then i can't send mail via my (paid for) Yahoo! email account from my local machine.
Yahoo! is making money from this service and i doubt they would like it if ISPs suddenly made half of that service impossible to access! This is NOT a good thing for people who travel and have to borrow many different internet accounts along the way.
the whole matter whould be solved if we all just started signing all our mail. that way, we can have mail filters weed out everything but signed mail from peopel we want mail from.
Sometimes I wish I was a plumber, then I'd know how to deal with other people's shit.
I think the key problem is ISPs that do not block egress traffic on port 25.
If that's what you think, look deeper. Major ISPs work on Cisco routers, and Cisco access lists aren't efficient at blocking by TCP port. They work, but most (if not all) bump the traffic up to the main CPU to do the filtering. That doesn't cut it at high speeds.
Besides: barring a heavily custom mail system, the spammer could as easily send via the ISPs mail server and some do. Why burn money on the first phase of the problem without a ready solution to the second?
It is not too tough to set up an SMTP server to require authentication
Doesn't work out-of-the-box on most mail servers, and links to arbitrary external authentication mechanism on very few of them. If the sysadmin has to write code then you havn't found the solution yet.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Spam filters really don't change economy of spam.
They only block spam from receivers who take
the necessary steps to combat it. Spam filters
are not mandatory for end users and some of them
do use spammers' offerings. Advanced users who
install and configure their spam filters are not
going to do business with spammers anyway.
Voluntary spam filters won't stop spam going around.
Only active measures and possibly some social engineering
would make difference. False replies sent to spammers
would change economics of spam. Other possibility
would be re-engineering email system to some form
of web of trust scheme.
No, because surely the spammers that used AOL will merely take five to find another open relay?
The problem with blocking like this is that it'd be useless until all the major open-relays are locked up tight. Only then will we see a reduction in spam, and even then it I doubt that it would be as much as 50% (but it'd be nice if it was).
Suppose I have my own SMTP server because I own my own domain. It is inside, behind the firewall, not accepting inbound mail from the internet, because I have a hosting provider which accepts mail on my behalf (from which I grab mail to my home systems by using fetchmail, but that's irrelevant).
So. If I have to use my ISP's SMTP server, which requires use of a username and password, how do I tell my local sendmail to authenticate itself to my ISP's system when sending outbound email? And ideally, how do I do it through the M4 configuration method, since I'm not a .cf wizard?
you should read everything on the internet as if it had "but I'm probably talking out of my ass" appended to it.
They use free web based mail addresses as collection points.
If hotmail/netscape/yahoo just charged a one time fee of five bucks for signing up, that would be the end of those emails...
Screw that, has Barry Shein ever met KIBO!
> If just AOL blocked port 25, this could reduce
> spam by 50% (I base this figure on close
> examination of the headers of the spam I
> receive).
Most of my spam comes from spamhouses with their own domains. Most of the header lines mentioning ISPs that I do see are forged.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Some of the best comments on this story point out that it is unlikely that spam can be combated technologically: Spammers have a financial incentive to get their messages through and will combat increasingly sophisticated technologies with their own increasingly sophisticated technological countermeasures. The issue, then, is to remove the financial incentive for spammers. I think the best method in this arena is to educate those who actually fall for spam products and made spamming profitable. What if e-mails providers illustrate the reality of spam with prose on their homepages and on customer's inboxes? I'm not talking about educational e-mail... just some sidebar on the interface that informs the user that if they even open spam e-mail they are contributing to the problem. Inform them that if they even click on a link in the e-mail they are sending a clear message to spammers that their methods work and there's no reason to stop the deluge. Or the sidebar could just tell customers that it doesn't matter to women what size they are.
There's already a Canadian ISP that has been blocking port 25 except for it's own mail servers for over a year now. Now if only every other ISP would do this, it would drastically reduce the amount of spam (at least in the short run. I'm sure spammers would find ways around this).
It's better to burn out than to fade away
What is it with these story submitters and the inane comments they attach to the story? I seriously doubt "RT Alec" would have been a VIP guest at the conference if he feels port 25 blocking is the solution to spam.
I think the key problem is ISPs that do not block egress traffic on port 25.
No.. the ISPs that block port 25 already care about spam, they just block it to reduce their administrative load. It reduces the spam cases they have to deal with - but they still cut off spammers. If they didn't block 25, they'd still cut off the spammers. The actual problem is ISPs that don't care about spam. These ISPs don't deal with their spammers so how can you expect them to block port 25?
If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive)
Funny, I base this statistic on the fact that you pulled it out of your ass. AOL has had spam problems, but they do deal with their spammers. It's ludicrous to suggest that they are responsible for half of all the spam on the Internet.
Tell me "RT Alec," how is port 25 blocking going to deal with rogue ISPs, who have a bulletproof connection through Verio? How about the clueless open relays that dot the maps of China, Brazil, and Argentina? What about for users of business DSL? Do we say, "you can't use your own corporate SMTP server, because you could be a spammer and we don't want to bother to deal with it?"
Argh. Breaking transparent end-to-end connectivity in "interesting" ways, randomly implemented by every ISP in existence, will not kill spam. And it will annoy the hell out of everyone in the long run. People behind stupid/nat-ing firewalls are already second-class citizens. People behind transparent proxies / unexpected port filters they can do nothing about will be third-class. Is this something we want?
</microrant>
...and convince the Bush administration to blow up Shenjun China. That would eliminate about half the spam that I get.
Spam Conference Reports As long as people are willing to push spam and people are willing to pull out there VISAs for products contained in spam, the problem will never go away. We need to start educating the newcomers to the internet that don't know better or help the people who can't contain themselves from impulse buys. ISPs should have a newcomers guide. What is SPAM and why you should avoid it. The work from home spammers are the equivalent of street pushers. The war on drugs hasn't been successful in stopping these criminals, what makes us think we can. They are inticed by the quick cash. And the addicts are inticed by the crap they buy. The only thing left is to fix SMTP and create end-to-end accountablility. Then sit back and wait for the next version of Spam to be developed. And start the cycle all over again.
> AOL set up rate limiting sometime around 07/98 [google.com]. Yes, it was THAT long ago.
And it made a big difference to the level of AOL origin spam.
> Note, as another poster has said, this wouldn't stop someone from using AOL as their ISP and connecting to another SMTP server for spamming purposes, but considering how slow (not to mention expensive) AOL-provided net access is, I doubt any real spammer would use it for even that.
AOL implemented transparent SMTP proxying during 1999-2000. They don't block outbound smtp entirely, but all outbound SMTP traffic is forced through their servers, is rate limited and is inspected for basic spamminess.
The admins can and would like to do more heavy duty filtering, but AOL legal won't let them.
AOL also rolled out their own DNSBL - ORBS style- but this was killed by AOL legal after open Earthlink customer relays smarthosting via Earthlink's main servers caused that ISP to be blocked.
Instead of fixing the fucking problem, Earthlink started screaming to the media about anticompetitive practices and threatening to sue.
Never min that AOL already won that battle - against Sanford Wallace in 1995 (Cyberpromo vs AOL - AOL was the defendant) - AOL legal forced the immediate shutdown of AOL's testing and blocking systems.
AOL admins would _like_ to do more about outbound spam. Their lawyers are a bunch of pussies and won't let them.
It may be a minor inconvenience for legitimate users, but at least *I* would prefer that I not have any ports blocked at all, and am willing to pay more for an ISP that doesn't block 25/129, etc, inbound or outbound.
Trying to stop spam by preventing spammers from accessing the Internet is pretty much a braindead solution. It's not feasible. There are too many access points around the entire world.
This is the same thing people tried to do with firewalls. "Block everything except 80". Then all the people actually trying to get work done simply tunnel everything through 80, or use Web Services, and the problem is right back in your face, except now the whole damn network is less efficient.
The *real* solution is simply to use whitelists -- eventually, it *is* going to have to be done.
Anti-spammers have for years (ever since the fucking DUL started blocking the mail server that *I* ran, not because it was an open relay, but simply because I like to use a non-gatewayed mailserver on my machine and happen to be on a dialup connection) pissed me off far more than spammers. I can block spam to the point where I only get one every few months, but I can't do anything about the amount of idiots endorsing the more intrusive anti-spam measures.
May we never see th
It's amusing that the posting purporting to the conference carried as a payload (in fact mostly consisted of) a mini-editorial about port-25 blocking. Seems like Slashdot needs to do some filtering too.
No one who read the conference announcement should be surprised that a lot of the talks were about filtering. We said four times in the first three paragraphs that the conference was about spam filtering. There are of course other solutions to spam, and I'm all in favor of them, if only because they make filtering easier too. (For example, the fact that many spammers feel constrained by various laws to include even fake unsubscribe links is a great help to filters.)
let's encourage ISP's to destroy accessibility to an essential service on the internet, in a misbegotten attempt to lessen illegitimate access. I don't want my connection censored! I enjoy having home broadband and running my own little server on it. My sendmail is set up to disable relaying, it's not like it's hard, and that is the true solution to spam. Spammers will always find a service that allows them the access they need, but this idiotic talk of blocking/censoring vital services/protocals doesn't help the rest of us.
BTW: Cause I run my own port 25 and have a static IP and a domain name, I get hardly any spam, personally. Why? Because I give out a different novel seperate address to everyone, and keep them all aliased to forward to my main account. If one becomes contaminated by spam, I simply delete it. If it actually was an address I gave to a correspondant [and not to some website, which is almost universally is] I only have to inform one person of a new address... come to think of it, that's only happened once...
---
the pen is mightier than the sword, the sword is mightier than the court, the court is mightier than the pen.
I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25 (you do have permission to use that SMTP server, don't you?). It is not too tough to set up an SMTP server to require authentication, or at a minimum to run off a different port.
Yeah, great, that way I can only send out email to their SMTP server, which only lets me use their email addresses, which means I'm wasting my hosting money. Great idea.
My notes on the conference can be found at http://commons.somewhere.com/buzz/2003/Technology. Notes.from.th.html. The really quick summary--everyone's got content-filtering fever, and I think they are nuts. You're trying to filter something that is NP-complete (Javascript email) and then do natural language understanding on it? I don't think so. Just as an example, consider the following three spams I've received recently.
Content filtering is doomed.
Oh yes, about blocking port 25. This is always followed by "and then your sysadmin can run SMTP on a different port so that you can connect to it via that." And if this becomes common, how long do you think until the spammers start scanning for alternate SMTP ports and doing direct delivery? In any case, it's moot. 90% of your spam isn't being sent from this country anyway. You're not going to persuade those remote sysadmins to block outbound port 25 any more than we've managed to get them to close their open relays. This is big business and big bucks.
This is sort of like giving a useful purpose to crackers and leting them feel good about their life and how much they contribute to the community.
Spammers uselly leave web site address for you to go to, or even a "if you would like to not receive any more emails please contact us at some-email@spammeraddress.com" so that they can confirm that they have a good active email account to spam.
We could have a site where the the address spammers are advertising could be placed (much like one crackers use to show whose site has been hacked). Our freindly noble minded community spirted cracker could then, at the beginning of his day, peruse this site and select a spammers site to bring the good news to.
The important part, though, is that by design it will never identify a real piece of mail as spam.
--Mike
"Not an actor, but he plays one on TV."
Blocking outbound traffic on port 25 is a totally useless measure these days. It could have been effective 2 years ago, but no more. Major spam-operators like Ralsky use open proxies and jeem-infected boxes on dsl lines. Open relays are no longer the #1 method for sending spam. Their use continues to decline.
By the way, it's spam, not SPAM. SPAM is Hormel's trademark.
Now, if what you meant to say was "port 25 blocking should be instituted for people on dialup addresses", I might be slightly more inclined to agree with that. There's a lot less accountability with dialup (read: modem) addresses (due to free trial accounts) than there is with cable or DSL
That's hard to square with you previous statement about not judging based in IP address. Some of us don't feel like spending an extra $30/month for accelerated advert browsing. Give me a break will you?
The real solution is to make those who abuse email pay for that abuse by simply outlawing spam.
Friends don't help friends install M$ junk.
Blocking port 25 is not the answer. It creates more problems than it solves. I am a senior sysadmin at a mid size hosting center, and we run mail services for a lot of our customers. The single biggest problem with mail is dealing with ISP's that block port 25.
Very true. Blocking 25 outbound has little to no effect. You *cannot* secure all points of access to the Internet. *Every* ISP in the world would need to do this for this to work, which is *not* going to happen -- there are too many ISPs around the world.
On the other hand, this *does* have serious issues from a user support point of view, and impacts technically savvy users that *do* want to use 25 outbound.
I have serious issues with port blocking at all, but if you want it to have *any* effect, 25 inbound is the only thing with any point.
25 outbound implies that *all* points of access must be secure to have any effect.
25 inbound implies that each ISP firewalling prevents relays from operating on their network.
That being said, port blocking tends to cause support issues and doesn't really do all that much in terms of security.
Ports were designed as a convenience, not as a strong security system.
Also, keep in mind that the majority of spam (that *I* get, at least), originates from worms that send mail from legitimate users' computers. Port blocking would have no effect.
If you really, honestly want to avoid spam, you can set up a whitelist with GPG or S/MIME support.
The reason suggestions like "run a mail server on another port if you really need it" come off as completely stupid to me is that that simply means that there is now another port (say, 5305) to block. You want to use simply random ports for each ISP? Now your security is that of the port number, which is essentially nothing.
The people making idiot suggestions (like the guy that posted this story) are looking for quick fixes that will reduce spam for maybe six months, and in the process make everyone miserable. If you really want to fix spam, you need something like whitelists and authentication. You can't make a change like this and expect it to work.
The only reason heuristics like "block !!! in subject line" work at all is because not everyone uses them. If MS shipped Outlook with a default rule to do so, spammers would simply avoid it.
So "using a different port" or similar suggestions are short term fixes that end up causing a lot of pain, much like people that firewall SSH or block egress DNS access. Screws users, buys nothing.
May we never see th
I'm shocked and appauled that people at MIT would say such clueless stuff.
Friends don't help friends install M$ junk.
TRtech is AOL Thanks, but no thanks for the training wheels. After almost ten years of paying for AOL, I'm going to kill the account. Their email IS useless due to spam but I've thought of the $10/month fee as a Mozilla supporter. The money will now go to the Free Software Foundation.
Friends don't help friends install M$ junk.
Getting rid of the spam after it's arrived at your mailbox is not a solution. The problem of spam is that it costs the ISPs lots of money. And, of course, that cost gets passed along to you, their customer. So you're still paying for the spam, even if you don't actually see it. I'm sorry, but you can babble about your wonderful bayesian filters all you want, but it's not actually much of a solution for anything except a tiny bit of personal annoyance.
Furthermore, if the last decade has taught us anything, it's that spammers are smart! You think they don't have access to filtering software? If these filters become widespread, then they'll start using them to test their spam before they send it, and they'll make adjustments to get through, adding subtle misspellings or other weirdnesses to confuse and confound the filters. Treating censorship as damage and routing around it is something the black hats do just as readily as the white.
so what happens when ipv6 finally gets rolled out to everyone and we all have static addresses?
Oddly enough, they were discussing, "...a training set with your personal tastes" on the BSDM list just the other day...
In my opinion, the best solution to the spam problem is very simple: Clearly criminalize the exploitation of third-party mail relays and vigorously prosecute such cases. Forget civil issues. It must be criminal. No matter how much filtering you do, the spammers will always seek to circumvent the system. The only way to stop them is to criminalize the hijacking of mail relays. Ok, it seems this is already criminal but nobody pursues it. This needs to change.
How do you get around the exploitation of foreign networks who don't follow the rules? The backbone ISPs agree to not route traffic for any ISP or network which doesn't adhere to certain standards with respect to adopting a uniform policy of taking action against those who hijack mail relays (i.e. cyber criminal extradition policy or something like that).
These spamming scumbags want to set up shop in China using mail relays overseas? Fine. When they're caught, we extradite them to China and let the Chinese punish them.
Until mail relay hijacking is clearly criminalized, we will NEVER reduce the amount of spam, period. Clearly criminalizing mail relay hijacking will force spammers to set up their own networks and then adopt more benevolent solicitation policies in an effort to not be blacklisted by the Internet at large.
I'd like to suggest my ideas for why Spam continues to become more and more of a problem. Ironically, even though many entities claim that spam costs them money, even those that don't like it benefit from its existence, and this creates an inherent conflict of interest:
1. Backbone providers make money selling bandwidth. Conventional wisdom dictates that spam traffic consumes a substantive amount of bandwidth. Therefore, backbone providers have a financial incentive to not reduce spam. They don't care whether the traffic is legit or not because the more the merrier for them.
Case in point: Backbone providers such as Sprint will NOT intervene in DOS attacks against their customers UNLESS the pipes they feed are saturated. If you have some attacker using 80% of your T1, they won't stop him, not until it reaches 100%. Why? Because only then does it take money out of their pockets.
2. Almost every other "SPAM solution" proposed, such as filtering software, actually relies on the existence of spam as a means of supporting themselves. Ironically, the spam filter companies need spam to continue to increase to help boost their business. So they don't really want you to not ever have to deal with spam.. they want it to be an ever-increasing problem so you pay them more money for updates and newer program versions. Again, there is an inherent conflict of interest here. Filtering software is totally useless unless you like the idea of paying some company a fee to reduce 20% of your spam and potentially block legit mail in perpetuity.
Spammers love to use the "Freedom of Speech" argument to justify the protections they deserve. This is fine and dandy. I have no problem with their right to promote what they want. But almost all spammers do one unquestionably unethical thing, which is exploit third-party mail relays to distrubte their spam, and this creates huge problems for innocent parties.
My solution: You hijack a third-party mail relay, you repurpose some web site's formmail.pl script? YOU GO TO JAIL. Period. This is the ONLY way we'll be able to deal with spam. Everything else is a total waste of time.
The problem is, the FBI (Federal Bureau of Incompetence) seems to have no clue how to address this issue. I have yet to see one case of someone hijacking a mail relay or breaking into another computer system and getting nailed for it, even though there seems to be numerous laws that would be broken in such a case.
People need to rally for federal and international enforcement of computer break-in laws. Everything else has been tried: filtering, blacklists, civil penalties, etc., and none of it has worked. When are people going to realize there is only one way to stop this, and it doesn't infringe upon any freedom of speech issues? Spammers can't operate without hiding their identity and location... make them have to do so or else there are CRIMINAL penalties, and we'll see spam stop pretty damn quickly.
MX records tell you which machines receive mail for a domain. They tell you nothing about who may send mail from that domain. It's perfectly legitimate to have different machines for these functions, and many large senders of mail do so.
This is sort of a temp solution but this is what I do for my users [if they request it]. I just change their email address and cancel the forwarding from the old name to the new name. Problem gone (temporarily).
:)
Then I tell them to stop putting their real email address in at all those porn sites.
Its a bad idea for an ISP to block any port. The SMTP server should be smart enough to allow only authenticated users to access it.
Similarly, the SMTP ports should allow relaying provided the user is authenticated. I have reason to send mail from one domain (alumni.uvic.ca) from the ISPs SMTP server, and fortunately they allow it.
Yahoo is allowing spam to go relatively unchecked and unfiltered so that more and more will use their paid services. One that has been on the internet and using Yahoo for a few years can hardly survive with a free account anymore, even if you don't give out the email address. Of course Yahoo also allows fraudulent auction listings to make up for sales too. More than 70% of the current auctions in the Macintosh category are either fraudulent or illegal to sell or both.
Yell & scream & rant & rave... it's no use... you need a shaaaave ~ Bugs Bunny
In a nutshell,
- Lots of talk on Paul Graham's Bayesian approach and the derivative works that some people have been doing.
- Speakers were for the most part very inciteful, interesting, and funny (!)
- Some talk on the business side of things (Barry Shein)
- Some talk about exisiting "solutions". Our solution is the best (pretty boring. and nothing really interesting there).
Some stuff to remember and/or worth mentionning:
- When designing a spam blocker, use differrent corpuses of mail for developing, tweaking and testing. That will reflect better the real-world situation. (the only interesting thing the Microsoft guy said)
- The business of spam is more complex than it seems. It's about multi-layer marketing schemes and the spam itself is the product, not necessairly the Viagra or the penis enlarger
- Spammers are intelligent and getting clever to evade spam blocking software (one notable example of a mail written in monospace font, using HTML, and formattedd to write vertically, instead of left-to-right. The scanning software sees nothing recognisable!)
- The non-free e-mail subject did come up.
- You can always trace to the source. Maybe the sender is forged, but you can always go up the smtp relay chain. there will be a point where someone has an open relay (or it's the source itself)
- The MIT's infinite corridor is actually finite.
- Spam-control is really at its infancy, probably like anti-virus software was like in the mid 80s.
- Spam conference study have no need of penis enlargement, study says.
JP.
Please moderate this to that it can be seen.
--- Worst tagline ever.
after and account is canceled. Everyone I know that has had an account with them was charged for months after cancelling it.
You can't judge a book by the way it wears its hair.
gee. I send an e-mail to someone at aol.com. I SMTP my ISP (port 25 by standard) the message, their SMTP server relays it to AOL's SMTP server. Oh wait. that SMTP server isn't there on port 25. Congrats. you've just killed inter-domain e-mail.
You can debate the fine points of Port 25, SMTP and Bayesian filters all day, but unless you make the economics of spam less attractive, you will never get a handle on it through technical means. If you compare spam to physical junk mail, you'll find that it is a *lot* cheaper to contact (annoy) 30 Million folks by email, than it is by using printed letters or catalogs. Until that changes, the spam merchants will read the same technical posts that you do, and evolve their offense as readily as you evolve your defense. Until it costs actual money to send each email on the internet, there will be absolutely no incentive for spammers to ever stop what they are doing, and if there's even a small amount of money to be made, there is always someone deep enough in poverty or sleazy enough to do what it takes to make it. If you enjoy the technical challenge of fighting spam, then by all means have a really good time, but please don't delude yourself into believing that that there is anything going on here but traditional bottom line economics. Unfortunately, there is always someone low enough on the totem pole to be perfectly happy to step up and do those dirty jobs that only exist to annoy almost everyone else.
Art
Start off with a new e-mail address. Do this sooner or later because you already have a ton of contacts who know your current address. The longer you wait, the harder it is to switch.
To get a new address, I bought a domain name through DirectNIC (whose service is fantastic, btw) and set up a referrer myname@mydomain.org to point to my POP3 box. NOBODY ever gets the pop3 address. The contacts I trust get the @mydomain.org. For other online services, I create servicename@mydomain.org (or use sneakemail, which is also fabulous, btw) and can kill those if they get spammed.
My addresses NEVER appear on any web site, usenet, etc. without spam-guarding.
This method works, and at my real address I only get about four spams PER YEAR. (It's always the same spam too. Something about skin care.)
Now which would you prefer? Setting up tons of filters, spamcop, spamassassin, etc. or just acting with a little more caution from the start and avoidign the spam in the first place?
(Note to trolls: The e-mail address you see attached to this message is a spam-trap.)
how do you fight back against spam? send it back. get a mail server and send back the spam you get in massive quantities. be sure to change your name and email address from the spam that you received.
Here's how challenge-response works:
This has already been suggested as a Mozilla mail enchancement, as Mozilla bug 187044. If you like the idea, by all means vote for it at Mozilla and/or encourage other email programs to add it.
The danger with filters is that even if they're based on good statistics or heuristics, they're just that - statistics and hueristics - and they can sometimes mistakenly throw away valuable email. A password email system, however, is deterministic - in particular, it always lets in email from those you trust and those able to respond to your challenge. I think challenge-response email passwords, combined with filters (which wouldn't have to be as selective), could go a long way to controlling spam.
- David A. Wheeler (see my Secure Programming HOWTO)
> ISP B passes the cost on to their customer (if he's a legit
> spammer) or sics the law on him for theft of services (if he's not)
I think this is a great idea. I think a penny per email per recipient is about right - us$0.01 each. In fact, you could even charge this across-the-board, and not distinguish between spammers and nonspammers. If I send an email to a dozen friends, it costs me 12c, not a big deal. But a spammer's bill goes from $300 to $1 million.
The problem with spam is there is absolutely no incentive for spammers to narrow down their outgoing list. It costs more, actually, to send to less than the whole list, than to sell to the whole list. Therefore, women get penis enlargement ads, children get beach bimbo ads, etc. Spam costs the spammer like $0.000 001 per piece, whereas for instance, paper junk mail costs maybe $0.30 or $1.00. In the latter case, they have a motivation to target their audience.
Marketing-driven companies end up over-marketing their products. Engineering-driven companies end up over-engineering
I don't think a lot of readers are understanding the idea suggested in the original post. If you are an customer of a dialup ISP, use that ISP's SMTP server-- only. If you want to use someone else's SMTP server (including your own that you set up somewhere), then that SMTP server ought to be configured to accept initial mail submission on a port other than 25. Your "rank and file" customers will not have a problem with this-- they will continue to use the ISP's SMTP server and all is well. For those "power users" (define as you like) that have a need for external SMTP servers, well, have them do the work (sorry-- life sucks sometimes). The ISP posts a page explaining why port 25 is blocked, and suggests using alternate ports (e.g. 465, SMTPS).
If you are the admin of an SMTP server that external clients (i.e. unknown IP addresses) will connect to for intitial mail submission, you are doing the Internet (and your users) a disservice if such connections are allowed unauthenticated. Sendmail, QMail, Exchange (gasp!) all can be configured to require authentication for initial mail submission. Use SSL as well, and you will probably be using another port (465). Spammers are not going to port scan for a way to send mail! Admins-- get off your butts and secure your servers, or else you are part of the spam problem. Please don't gripe about how following industry standard practices for securing a publicly accessable server makes your job more difficult-- that is your job!
If I just get a reply with instructions to reply to the reply I don't see why this would be annoying.
ich bin der musikant
mit taschenrechner in der hand
kraftwerk
After all if port 80 is open everything is open as long as it's TCP based. HTTPtunnel is your (and every spammer's) friend. So - filtering all ports != 80 is a waste of time, and will only bother normal users without some basic knowledge. Duh.
I've been toying with the idea of forwarding all my Korean and Chinese spam (80% of the spam I receive is in those languages) to their embassies. Currently, .kr and .cn ISPs are being bribed into giving spammers free reign. The Chinese and Korean governments could put a stop to that (IIRC South Korea does have spamming legislation), they just need to be made aware of the seriousness of the problem.
Sending the government a few spams won't do that, but sending them all the spam anyone receives might.
I find it interesting that almost everyone prefers a communist solution to the email problem: no one pays anything so that the resources get allocated to those with the best connections and who are most able to exploit the system.
Of course, the idea that no one pays is completely false - everyone pays.
The reality of communism and market economies is that there are the privileged and the deprived. What is at issue is how well off the society as a whole is.
The current system is analogous to everyone contracting to have a mail box, or set of mail boxes. The people providing the mail boxes are required to give anyone who asks as much paper as they demand. Now the spammers drive around stuffing any and all mail boxes, along with individuals, and USPS, UPS, Fedex, etc. end up delivering it simply because they have no way to get rid of the spam.
Of course, this isn't the way the physical mail system works: if you don't pay the postage, the mail is either discarded or returned to you. And you don't pay to get mail, you pay to send it.
I think that it is possible to improve on the physical world mail system by making it clear to the recepient what class each piece of mail is.
Charge for 1st class email, with a network of authenticated mail relays (MTAs) moving this 1st class mail. The governance required to implement such a system of MTAs is exactly the same as is required to build the Internet. (You can't connect to the Internet without some peering relationship - if there is a way to do this, let me know so I can connect for free.)
I'd also suggest additional classes email, perhaps 2nd and 3rd class mail.
The current system of 3rd or 9th class mail can stay in place with mail service being "free" with delivery being subject to arbitrary and unspecified rules. (Eg., put 4 dollar signs in the message and it gets discarded, whether its spam or a personal message.)
Existing mail programs (MUA) generally support fetching mail from multiple mailboxes, so you can setup to fetch from your 1st class, 2nd class, and 9th class mailboxes and immediately distinguish between them.
There are a lot of problems to solve to implement such a system, but the biggest obstacle is to get over the "free email" illusion that many believe exists.
There are many who think that it was/is terrible that connecting to the Internet now costs money, as if it were free to connect to the Internet at some point in the past. Most people have gotten over charging for Internet access, now its time to get over charging for email.
You mean like the laws that alow me to charge $50 for each piece of spam I recieve? No? Only the cartel of ISPs with the "will to implement and enforce these changes" will profit. Sounds like the broadcast TV model where only three or four big corps get to abuse the public airwaves for fun and profit. No thanks, let's simply make a dreadful practice against the law and those who break the law accountable to all parties they inconveniance. There is no technical reason for the kind of restrictions Shien would pull over the rest of us who wish to run their own mail servers.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
Chrysflame posted detailed minutes for the proceedings, as pasted from Oliver Schmelzle's TechBlog.Readers may find it useful to cross-check my notes against his times when looking for talks they would like to listen to.
Matt Sergeant politely replied as well, noting that the impressive claims about CRM114's accuracy were yet to be thoroughly tested, that in other tests CRM114 had not been significantly more accurate than other Bayesian strategies, and that the current performance of CRM114 is so much slower than many of the alternatives that any gains it may have to offer are more than offset by the low volume it can currently handle. Grain of salt taken :)
No comments as of this writing.
An anonymous coward added a couple of corrections which are worth noting:
Jon Praed was questioning IP spoofing, not message header spoofing. It is relatively easy to fake at least some of the headers on an email, but when tracked down & brought before a judge, no spammer has ever been able to explain a credible technique for spoofing IP data in any trial Praed was aware of. When this comment was made to the audience, ESR spoke up saying that he could show Praed how to do it, but I don't know what if anything came of any conversation they had after the talk.
The AC also expanded on Michael Salib's talk & how much mileage Salib was seeing out of a comically non-buzzword compliant filtering strategy, but came back to the point that his results were "probably unrepeatable and it would probably be best if we all just treated them as outright lies." As the AC noted, Salib seems to have played a big role in organizing the conference -- I think I read somewhere that when the attendee list swelled to 500+ people, he helped to find a last minute venue big enough to accomodate everyone. So not only do we have to thank Salib for an entertaining spiel of quackery, but also for bringing everyone together in the first place. :)
I never said my notes were perfect :)
In his email, Spencer went on to expand on the value of honeypots, and how they seem like a very promising tactic for handling the spam problem. I agree, and maybe my writeup didn't give this enough attention, but I think many or all of the conference speakers would have agreed as well. Ken Schneider made it clear that Brightmail in particular seems to make heavy use of honeypot addresses: it sounded like when they set up service for an organization, they plant one or more dummy addresses at that organization as data points for spam collection efforts, and have mechanisms in place to gather & analyze this data in real time. Spencer suggests that honeypot addresses would be very hard for spammers to detect if they resemble legit MTAs as much as possible, and I have the impression that this is exactly what Brightmail is doing. I'm sure that others are using tactics like this as well, but Schneider was the most vocal user of the tactic that I noticed.
John Hanna wrote to me saying that he runs an anti-spam project at http://assp.sf.net, and noticed a surge in traffic after the conference. To answer John's question, I did not notice anyone mentioning ASSP [caps?] during any of the talks, but it could well be that people were discussing it amongst themselves off stage. *shrug*
I assume that Spam(r) is cool about the use of the term 'spam' to mean junk e-mail, but adding a converse makes it explicitly clear that 'spam=bad'.
And what do the pigs think about all this? Its their flesh we're talking about. The ultimate expression of love is to consume the flesh of another being; we are sending out a mixed message as to whether we love pigs or not, which will surely effect the quality of the eggs they lay.
By this token eating one's fingernails/bogies/earwax is a form of self-love, which is perfectly natural.
To which I have no comment :)
If I get any other material relevant to the conference, I may add it to the Slashdot or use.perl journals, but in any case I wanted to get this up while the pages are still getting traffic, so readers of one variation of the page are not missing out on what may be added to other variations. Thanks all for the feedback! :)
DO NOT LEAVE IT IS NOT REAL