My firm has recently consulted for an email service provider that handles mail for about ten million end user accounts. Until recently, they were running everything through a large and growing bank of content filtering servers. As traffic has increased, the load on their filtering machines has increased exponentially, as has the storage requirements for their anti spam quarantine system.
Whatever you do, please add some kind of throttling-style connection control in front of the content filtering systems to limit the rate at which spammers can connect to your content filters. With content filtering and blacklists alone, you will only get about 95% of the spam and your infrastructure costs will know no limit. Add connection control and you can get the last five percent under control while also significantly reducing the amount of mail that ends up wasting time and space in the quarantine.
My company sells a traffic shaping connection control system. Fancier appliance-based options such as the Symantec 8160 are also available if you have large amounts to spend and a propensity to go with the big name.
To my knowledge something like this is not yet available in open source -- probably because it has only until recently made sense for large mail receivers such as your client.
Sitekey is a pseudo-two-factor authentication system (pseudo because both factors of authentication are provided within the framework of the same bug-ridden PC). It absolutely does not resolve the phishing problem for Bank of America customers. It is also vulnerable to a trivial man in the middle attack.
Here's why it doesn't solve phishing: Phishers have and will continue to phish BoA customers for their personal information such as their Social Security Numbers, bank account numbers, mother's maiden name, etc.. by crafting email messages that appear to come from BoA.
The man in the middle attack works as follows:
1. Create a phishing web site.
2. Ask the user for their username in exactly the same way as the BoA site does with SiteKey.
3. When you have their username, contact the BoA site and download the list of authenticity questions the site wants to ask the end user.
4. Ask these questions of the phished user.
5. Pass the answers on to the real BoA site.
6. Voila. Not only do you now have access to the BoA site, you have successfully obtained further private information of the end user, such as the user's mother's maiden name.
I wrote about SiteKey on my blog, which for whatever reason is now viewed by Google as one of the leading authorities on SiteKey: http://mailchannels.blogspot.com./ Enjoy!
There's plenty of room for both Google and Yahoo to continue to make scads of money for years to come. Google may end up coming up with more innovative technologies, but this does not mean they'll conquer and kill Yahoo. History has taught many companies that it is often better to be second than first at introducing a new technology to the market.
Sxip is funded by the founder, Dick Hardt. If it was funded by VCs, there would have been more vetting of the soundness of the company's business strategy -- not to say it's necessarily bad.
What Sxip is, IMHO, is a startup promoting a very cool new technology for identity management. They just haven't found the right market yet. This is a fine way to run a startup. It just might take them a while before they find the sweet spot.
Plus, Sxip's not operating in stealth mode at all. Stealth mode by definition means not telling the world what you're doing. Sxip has published the details of their technology and products in intricate detail and provide open source implementations of the Sxip protocol. How is that in any way stealthy?
SNDS is a very good idea and I'm glad Microsoft has put it out there. It will help ISPs to identify spammers within their own networks (i.e. zombies) and should result in a drop in the spam entering Hotmail and other Microsoft properties.
To make this service even better, Microsoft should add a web service interface so that ISPs can automatically check their records. An alternative would be to email the ISP summary reports in a standard format -- in much the same way that AOL does.
Rather than blocking port 25, progressive and user-friendly ISPs (does such a thing exist?) would be well served to simply throttle port 25. By exponentially dropping the available bandwidth to that port as traffic on it increases from a particular host, the zombie problem can be for the most part eliminated while not unduly penalizing legitimate senders of email.
Blocking port 25 just shifts the problem around. With port 25 blocked, zombie owners are forced to use the ISP's outgoing mail servers. If throttling is intelligently applied to all port 25 traffic on a per-host basis, the feasibility of zombie spamming drops off.
Put it this way: Which would you prefer: having one of your customers blacklisted as a result of spamming, or having ALL of your customers blacklisted as a result of your own mail servers spamming...?
The OpenBSD team is working on a transparent traffic shaping proxy that will make magic like this trivial for the pf priesthood. IMHO this is yet another reason to support that excellent project by buying a CD or T-shirt.
I feel for you. My company has developed a system that uses a new kind of email address aliasing (we call it "domain-based aliasing") to allow messages from your important contacts to reach you. Unfortunately this won't help the reverse direction, but for many users we think it will solve a big problem (i.e. your American friends being able to get mail from you while blocking all the other crap from Russia).
I would be interested in feedback on our approach. Anyone interested in trying it out can sign up for the pilot program.
Re:Don't astroturf just say its yer site & com
on
Spammers' Upend DNS
·
· Score: 1
You're right -- this was astroturfing. Thanks for keeping me honest and I apologize to the greater Slashdot community for having done so.
What we have built is an automated system for creating and managing domain-based email aliases. What does "domain-based" mean? Basically this:
Regular Address: username@domain.com
Domain-based Alias: username@alias.domain.com... where the alias part is a pseudorandom sequence that acts as a shared secret between one or more parties who wish to communicate with each other.
Because the MX entry for the alias subdomain is consulted each time a message is delivered, it's possible to have a different MX entry for different aliases. In this way, mail traffic can be routed differently at the network level depending on who it originates from.
The use case that has received the most traction so far is to separate customer traffic so that it can be treated more kindly by the spam filter -- or bypass the filter altogether. In this way, you never lose a message from a customer (i.e. no false positives).
We are in the middle of a site update that will explain all. If you're interested, visit http://www.mailchannels.com in a few days' time or watch for a press release.
BTW -- the Apache::SMTP bits are a genuinely cool innovation courtesy of our CTO Will Whittaker. Look out for some articles showing up soon on this topic.
Regards, Ken
CEO, MailChannels (and convicted Astroturfer)
Correction: this is not a problem with the DNS
on
Spammers' Upend DNS
·
· Score: 1
Spammers are not "up-ending" the DNS, they're simply causing poorly designed anti spam systems to consume inordinate resources as a result of their naive assumption that DNS lookups don't need to be managed intelligently. I'm sure this is something that the anti spam vendors are looking at, but probably not something that will be fixed soon, since it's really quite a difficult problem to address.
Interestingly enough the same technique can be used against spammers. Take a look at what these guys are doing -- the site's content a bit slim but it looks like they're using a kind of DNS aliasing that could really hurt spammers in much the same way. I imagine techniques like these that operate at the DNS level are the next step in the evolution of anti spam.
With the dramatic improvements in spam filtering software, getting rid of spam is no longer the technical problem it once was. In my experience as a consultant to email administrators and as a market research in the messaging industry, other, derivative problems are now taking over. And these problems are the result of filtering.
There are several problems that now plague email administrators: 1) satisfying the vast resource requirements of a modern email filtering system, 2) handling an increased flow of end-user complaints (yes, increased), and 3) dealing with false positives.
Everyone knows that spam is an enormous problem. The 75% number quoted in this article is conservative. Many organizations I work with receive in excess of 90% spam. Dealing with a problem of this magnitude is of course absolutely necessary -- and most large companies have by now installed a spam solution.
Unfortunately, implementing a large scale spam filtering solution requires rolling out sophisticated enterprise software and managing expensive, complicated, and high maintenance storage devices. This storage is mostly eaten up by the spam quarantine (or "junk mail folder") -- something that is necessary to deal with the possibility of false positives.
Even assuming that the system is correctly installed, maintaining it is an ongoing nightmare. And with a spam filter in place, end-users tend to assume that any spam that does get through is the result of a system failure that should be reported immediately as a trouble ticket -- adding to the email administrator's burden.
Finally, even though the latest spam filters are pretty good at what they do, if you're looking for a 95% spam rejection ratio, getting a false positive rate of less than 0.5% in the real world is a challenge. And while most false positives are things like newsletters that you don't normally care about, occasionally something critical is eliminated. When that happens, the email administrator can lose his job.
So what does he do? He tunes down the capture efficiency of the filter to drop the false positive rate. In a recent survey, Sophos PureMessage (one of the big iron enterprise anti spam solutions) had a capture rate of 90% and a false positive rate of 0.04% (Network World Spam Survey from December 2004). IMHO, 90% is a terrible capture rate that would result in an unacceptable flow of end-user complaints. Why did Sophos tune their product this way? Because false positives are the number one concern of email administrators.
Bar none. Number one.
False positives get you fired. Spam gets you a few more trouble tickets. You decide.
Spam filtering will always be necessary, but a complete rethink is required to take the problem resolution to the next level without the attendant drawbacks of filtering. The rethink involves end-user authentication (read: this is not the same thing as SenderID's domain authentication), something that can be implemented today using an aliasing system.
Greylisting is fantastic for people who can accept waiting up to four hours to receive a response from a new contact -- for example, when you sign up for something on a web site and it needs to send you a confirmation email.
In most corporate environments, this delay is unacceptable. Another problem with graylisting is that it's difficult to implement with multiple MXs. If you have four mail servers that are all graylisting, they must share their graylist database or else incoming messages might be graylisted four times before finally being delivered through one of the MXs.
All things considered, however, if graylisting works for you -- go crazy! Because the corporate types aren't using it, it's likely spammers won't bother adapting to this technique for some time.
I empathize with technology workers in the US who are afraid of losing or have already lost their job to a third-world country. But it's important to realize that this process of outsourcing is a natural consequence of the enormous gap between the wealth of Americans versus those overseas.
Offshoring creates opportunities for millions of otherwise disadvantaged individuals in countries where incomes are lower than they are in the US. Offshoring boosts their incomes and helps these poor countries become rich countries by funding improvements in their infrastructure and allowing them to access external resources through increased foreign currency reserves.
Also, it is not the case that a job outsourced is a job lost to an American. Oursourcing creates capacity which allows more production of goods and services and therefore supports a larger workforce -- in the US and abroad. Certainly there will be a period of adjustment where some US jobs are lost and seemingly not recreated, but look at what happened throughout the 1990s when manufacturing jobs were lost to cheaper countries. Jobs were created in other areas of the economy and Americans ended up far wealthier than before.
There are six billion people on earth and the vast majority of them are clamoring in the depths of poverty for any kind of upgrade. It's only fair that we share the wealth with them -- or suffer the consequences of a downtrodden, well armed opponent...
In BC, the former leftist government introduced a zero over-time policy for technology workers. What this means is that if you work in BC in a technology capacity, you are not paid for overtime.
Companies like Electronic Arts, who have a massive game studio in Vancouver, lobbied hard for the legislation that is viewed as grossly unfair by workers, while providing a nice bonus for employers keen to provide cheap outsourced labour for American companies.
The anti spam research group (ASRG), which is a working group within the IETF, has specified a list of requirements that any successful "universal" anti spam solution must have (http://www.ietf.org/.../asrg-5.pdf)
Let's see how Yahoo's approach stacks up:
must minimize unwanted messages -- probably
must not affect delivery of wanted messages to the detriment of normal email -- probably
must be easy to use -- for the end user, yes, but for organizations no (cryptography is a hard problem to solve right)
must be easy to deploy, incrementally -- difficult to deploy because everyone has to upgrade their mta
must not depend on universal deployment to be effective -- rats! Yahoo's system doesn't work very well unless everyone buys in
must not reduce privacy -- cryptographically signing emails means less privacy
must have minimal administration overhead -- Yahoo's solution requires maintaining a cryptographic framework, which is difficult
must have minimal computation and bandwidth overhead -- how costly is it to sign each message? on busy servers it's very costly
must consider the threat and be robust in the face of such threats -- not sure about this one...
should consider how legal issues affect, support, or constrain the technical solution -- crypto is illegal in some countries
... both FatPort and Surf and Sip offer "free hot spot" in a box solutions for cafes that don't want to charge for service. For what you would pay for one of these boxes, it's difficult to justify trying to do it yourself.
And since FatPort and Surf and Sip have hardened these box down for use in their own networks, they'll provide adequate protection from Kazaa zealots and the new breed of wireless spammers.
Writing a patent isn't the black art that lawyers claim it to be. A friend of mine used a lawyer for a few hours to learn how to write a patent. Then he went off and wrote it himself. When he had finished most of the work, he returned to the lawyer, who in a few hours corrected all the mistakes. Then he spent $200 on the basic filing fee and voila! Patent on the cheap!
Up here in Vancouver, I get a 3Mbps DSL line with 640Kbps upstream for CDN$35/month -- that's about US$25. Or I could choose 4Mbps cable for CDN$40. Or get wireless service from a company like FatPort (www.fatport.com).
I recently saw a documentary about Project Plowshare at the Vancouver International Film Festival. The Plowshare demonstrated over its thirty-some-odd years of maniacal testing that nuclear weapons are not a good choice for moving dirt. The USSR's own project released enormous quantities of radioactive fallout into the atmosphere and never really achieved much more than creating a large crater one _might_ be lead into believing is a reservoir.
The primary problem with using nuclear detonations to dig holes is that they are immensely difficult to control. Digging rough craters may be an achievable goal, but cutting neat troughs through the mountain is basically impossible.
It was a flop, motivated by politics. So give it up, China. When your citizens, oppressed as they are, begin campaigning against the strontium-90 in their children's milk (by testing for it in their teeth), it will be only a matter of time before the whole thing craters out and adds another nail into the regime's coffin.
Slashdot Mirror
My firm has recently consulted for an email service provider that handles mail for about ten million end user accounts. Until recently, they were running everything through a large and growing bank of content filtering servers. As traffic has increased, the load on their filtering machines has increased exponentially, as has the storage requirements for their anti spam quarantine system.
Whatever you do, please add some kind of throttling-style connection control in front of the content filtering systems to limit the rate at which spammers can connect to your content filters. With content filtering and blacklists alone, you will only get about 95% of the spam and your infrastructure costs will know no limit. Add connection control and you can get the last five percent under control while also significantly reducing the amount of mail that ends up wasting time and space in the quarantine.
My company sells a traffic shaping connection control system. Fancier appliance-based options such as the Symantec 8160 are also available if you have large amounts to spend and a propensity to go with the big name.
To my knowledge something like this is not yet available in open source -- probably because it has only until recently made sense for large mail receivers such as your client.
Our home page: http://www.mailchannels.com/
Sitekey is a pseudo-two-factor authentication system (pseudo because both factors of authentication are provided within the framework of the same bug-ridden PC). It absolutely does not resolve the phishing problem for Bank of America customers. It is also vulnerable to a trivial man in the middle attack.
Here's why it doesn't solve phishing: Phishers have and will continue to phish BoA customers for their personal information such as their Social Security Numbers, bank account numbers, mother's maiden name, etc.. by crafting email messages that appear to come from BoA.
The man in the middle attack works as follows:
1. Create a phishing web site.
2. Ask the user for their username in exactly the same way as the BoA site does with SiteKey.
3. When you have their username, contact the BoA site and download the list of authenticity questions the site wants to ask the end user.
4. Ask these questions of the phished user.
5. Pass the answers on to the real BoA site.
6. Voila. Not only do you now have access to the BoA site, you have successfully obtained further private information of the end user, such as the user's mother's maiden name.
I wrote about SiteKey on my blog, which for whatever reason is now viewed by Google as one of the leading authorities on SiteKey: http://mailchannels.blogspot.com./ Enjoy!
There's plenty of room for both Google and Yahoo to continue to make scads of money for years to come. Google may end up coming up with more innovative technologies, but this does not mean they'll conquer and kill Yahoo. History has taught many companies that it is often better to be second than first at introducing a new technology to the market.
Sxip is funded by the founder, Dick Hardt. If it was funded by VCs, there would have been more vetting of the soundness of the company's business strategy -- not to say it's necessarily bad.
What Sxip is, IMHO, is a startup promoting a very cool new technology for identity management. They just haven't found the right market yet. This is a fine way to run a startup. It just might take them a while before they find the sweet spot.
Plus, Sxip's not operating in stealth mode at all. Stealth mode by definition means not telling the world what you're doing. Sxip has published the details of their technology and products in intricate detail and provide open source implementations of the Sxip protocol. How is that in any way stealthy?
SNDS is a very good idea and I'm glad Microsoft has put it out there. It will help ISPs to identify spammers within their own networks (i.e. zombies) and should result in a drop in the spam entering Hotmail and other Microsoft properties.
To make this service even better, Microsoft should add a web service interface so that ISPs can automatically check their records. An alternative would be to email the ISP summary reports in a standard format -- in much the same way that AOL does.
Rather than blocking port 25, progressive and user-friendly ISPs (does such a thing exist?) would be well served to simply throttle port 25. By exponentially dropping the available bandwidth to that port as traffic on it increases from a particular host, the zombie problem can be for the most part eliminated while not unduly penalizing legitimate senders of email.
Blocking port 25 just shifts the problem around. With port 25 blocked, zombie owners are forced to use the ISP's outgoing mail servers. If throttling is intelligently applied to all port 25 traffic on a per-host basis, the feasibility of zombie spamming drops off.
Put it this way: Which would you prefer: having one of your customers blacklisted as a result of spamming, or having ALL of your customers blacklisted as a result of your own mail servers spamming...?
The OpenBSD team is working on a transparent traffic shaping proxy that will make magic like this trivial for the pf priesthood. IMHO this is yet another reason to support that excellent project by buying a CD or T-shirt.
Forgot to add our website URL: http://www.mailchannels.com. Enjoy!
I feel for you. My company has developed a system that uses a new kind of email address aliasing (we call it "domain-based aliasing") to allow messages from your important contacts to reach you. Unfortunately this won't help the reverse direction, but for many users we think it will solve a big problem (i.e. your American friends being able to get mail from you while blocking all the other crap from Russia).
I would be interested in feedback on our approach. Anyone interested in trying it out can sign up for the pilot program.
You're right -- this was astroturfing. Thanks for keeping me honest and I apologize to the greater Slashdot community for having done so.
... where the alias part is a pseudorandom sequence that acts as a shared secret between one or more parties who wish to communicate with each other.
What we have built is an automated system for creating and managing domain-based email aliases. What does "domain-based" mean? Basically this:
Regular Address: username@domain.com
Domain-based Alias: username@alias.domain.com
Because the MX entry for the alias subdomain is consulted each time a message is delivered, it's possible to have a different MX entry for different aliases. In this way, mail traffic can be routed differently at the network level depending on who it originates from.
The use case that has received the most traction so far is to separate customer traffic so that it can be treated more kindly by the spam filter -- or bypass the filter altogether. In this way, you never lose a message from a customer (i.e. no false positives).
We are in the middle of a site update that will explain all. If you're interested, visit http://www.mailchannels.com in a few days' time or watch for a press release.
BTW -- the Apache::SMTP bits are a genuinely cool innovation courtesy of our CTO Will Whittaker. Look out for some articles showing up soon on this topic.
Regards,
Ken
CEO, MailChannels (and convicted Astroturfer)
Spammers are not "up-ending" the DNS, they're simply causing poorly designed anti spam systems to consume inordinate resources as a result of their naive assumption that DNS lookups don't need to be managed intelligently. I'm sure this is something that the anti spam vendors are looking at, but probably not something that will be fixed soon, since it's really quite a difficult problem to address.
Interestingly enough the same technique can be used against spammers. Take a look at what these guys are doing -- the site's content a bit slim but it looks like they're using a kind of DNS aliasing that could really hurt spammers in much the same way. I imagine techniques like these that operate at the DNS level are the next step in the evolution of anti spam.
With the dramatic improvements in spam filtering software, getting rid of spam is no longer the technical problem it once was. In my experience as a consultant to email administrators and as a market research in the messaging industry, other, derivative problems are now taking over. And these problems are the result of filtering.
There are several problems that now plague email administrators: 1) satisfying the vast resource requirements of a modern email filtering system, 2) handling an increased flow of end-user complaints (yes, increased), and 3) dealing with false positives.
Everyone knows that spam is an enormous problem. The 75% number quoted in this article is conservative. Many organizations I work with receive in excess of 90% spam. Dealing with a problem of this magnitude is of course absolutely necessary -- and most large companies have by now installed a spam solution.
Unfortunately, implementing a large scale spam filtering solution requires rolling out sophisticated enterprise software and managing expensive, complicated, and high maintenance storage devices. This storage is mostly eaten up by the spam quarantine (or "junk mail folder") -- something that is necessary to deal with the possibility of false positives.
Even assuming that the system is correctly installed, maintaining it is an ongoing nightmare. And with a spam filter in place, end-users tend to assume that any spam that does get through is the result of a system failure that should be reported immediately as a trouble ticket -- adding to the email administrator's burden.
Finally, even though the latest spam filters are pretty good at what they do, if you're looking for a 95% spam rejection ratio, getting a false positive rate of less than 0.5% in the real world is a challenge. And while most false positives are things like newsletters that you don't normally care about, occasionally something critical is eliminated. When that happens, the email administrator can lose his job.
So what does he do? He tunes down the capture efficiency of the filter to drop the false positive rate. In a recent survey, Sophos PureMessage (one of the big iron enterprise anti spam solutions) had a capture rate of 90% and a false positive rate of 0.04% (Network World Spam Survey from December 2004). IMHO, 90% is a terrible capture rate that would result in an unacceptable flow of end-user complaints. Why did Sophos tune their product this way? Because false positives are the number one concern of email administrators.
Bar none. Number one.
False positives get you fired. Spam gets you a few more trouble tickets. You decide.
Spam filtering will always be necessary, but a complete rethink is required to take the problem resolution to the next level without the attendant drawbacks of filtering. The rethink involves end-user authentication (read: this is not the same thing as SenderID's domain authentication), something that can be implemented today using an aliasing system.
Greylisting is fantastic for people who can accept waiting up to four hours to receive a response from a new contact -- for example, when you sign up for something on a web site and it needs to send you a confirmation email.
In most corporate environments, this delay is unacceptable. Another problem with graylisting is that it's difficult to implement with multiple MXs. If you have four mail servers that are all graylisting, they must share their graylist database or else incoming messages might be graylisted four times before finally being delivered through one of the MXs.
All things considered, however, if graylisting works for you -- go crazy! Because the corporate types aren't using it, it's likely spammers won't bother adapting to this technique for some time.
I empathize with technology workers in the US who are afraid of losing or have already lost their job to a third-world country. But it's important to realize that this process of outsourcing is a natural consequence of the enormous gap between the wealth of Americans versus those overseas.
Offshoring creates opportunities for millions of otherwise disadvantaged individuals in countries where incomes are lower than they are in the US. Offshoring boosts their incomes and helps these poor countries become rich countries by funding improvements in their infrastructure and allowing them to access external resources through increased foreign currency reserves.
Also, it is not the case that a job outsourced is a job lost to an American. Oursourcing creates capacity which allows more production of goods and services and therefore supports a larger workforce -- in the US and abroad. Certainly there will be a period of adjustment where some US jobs are lost and seemingly not recreated, but look at what happened throughout the 1990s when manufacturing jobs were lost to cheaper countries. Jobs were created in other areas of the economy and Americans ended up far wealthier than before.
There are six billion people on earth and the vast majority of them are clamoring in the depths of poverty for any kind of upgrade. It's only fair that we share the wealth with them -- or suffer the consequences of a downtrodden, well armed opponent...
In BC, the former leftist government introduced
a zero over-time policy for technology workers.
What this means is that if you work in BC in a
technology capacity, you are not paid for overtime.
Companies like Electronic Arts, who have a massive
game studio in Vancouver, lobbied hard for the
legislation that is viewed as grossly unfair by
workers, while providing a nice bonus for employers
keen to provide cheap outsourced labour for
American companies.
And since FatPort and Surf and Sip have hardened these box down for use in their own networks, they'll provide adequate protection from Kazaa zealots and the new breed of wireless spammers.
Writing a patent isn't the black art that lawyers claim it to be. A friend of mine used a lawyer for a few hours to learn how to write a patent. Then he went off and wrote it himself. When he had finished most of the work, he returned to the lawyer, who in a few hours corrected all the mistakes. Then he spent $200 on the basic filing fee and voila! Patent on the cheap!
Up here in Vancouver, I get a 3Mbps DSL line with 640Kbps upstream for CDN$35/month -- that's about US$25. Or I could choose 4Mbps cable for CDN$40. Or get wireless service from a company like FatPort (www.fatport.com).
I recently saw a documentary about Project Plowshare at the Vancouver International Film Festival. The Plowshare demonstrated over its thirty-some-odd years of maniacal testing that nuclear weapons are not a good choice for moving dirt. The USSR's own project released enormous quantities of radioactive fallout into the atmosphere and never really achieved much more than creating a large crater one _might_ be lead into believing is a reservoir. The primary problem with using nuclear detonations to dig holes is that they are immensely difficult to control. Digging rough craters may be an achievable goal, but cutting neat troughs through the mountain is basically impossible. It was a flop, motivated by politics. So give it up, China. When your citizens, oppressed as they are, begin campaigning against the strontium-90 in their children's milk (by testing for it in their teeth), it will be only a matter of time before the whole thing craters out and adds another nail into the regime's coffin.
Vovida has an excellent suite of GPL'ed IP-telephony stacks (H.323, SIP, Megaco, RTP, etc..). We are using them in our products.