> Now this internal blacklist is then shared to all the other customers who use Spamshark, so they are now protected too; resulting in a 5 nines hit rate on spam.
And more false posistives than you would actually like to have. I've been at the business end of one of Frontbridge's blacklists. One of the domains I admin got blacklisted a full three weeks after the hosting company screwed up and let phishers set up a paypal scam site as the "test1" user to live for all of 22 hours. Three weeks later, one of the company's main customers, who happens to be a frontbridge customer, is no longer able to receive mail from us. A an unfinished writeup is at bsdly.net - I just gave up in disgust after trying to write an article about the incident.
Subject says it all, really. The best approach is to set up an OpenBSD machine as your gateway, filter traffic using PF to any degree you desire, and please set up spamd in greylisting mode (the default).
That will take care of most of your spam right there, and you could usefully have something like a spamasassin and clamav combo running in the delivery phase on your real mail server.
We see a lot of junk hitting our greylist at the gateway, but 95% just ends there.
OpenBSD's spamd is a wonderful greylister, and it offers a few other options which will make a dent in the reminaing few if you can be bothered to set it up. See my blog at http://bsdly.blogspot.com/ and links from there for some examples.
There is a time tested solution: DocBook
on
Embedding XML In Docs?
·
· Score: 2, Informative
If you're already dealing with XML files, I would suggest that the main barrier to using a toolset such as DocBook (SGML or XML variants) should be gone already.
DocBook is excellent at enforcing proper structure and contains all the elements you need (really!) to write tech documentation.
Several high profile projects such as FreeBSD, KDE, GNOME and others use DocBook as their main doc format, as do I believe more tech companies than actually want to admit it. I maintain the PF tutorial at http://home.nuug.no/~peter/pf/ as DocBook SGML myself.
The tools most people use for DocBook are free (most likely just a few mouse clicks or commands away through your package system), but some proprietary/commercial tools are available too. The main reference is at docbook.org, it certainly would not hurt to check it out.
Re:They try to send, but don't really succeed
on
The New Yorker On Spam
·
· Score: 2, Interesting
It's not solved. As long as it's taking a measurable number of people working full-time to "solve" it, it's not solved. It'll be solved when we no longer have to spend huge chunks of bandwidth on it, no longer lose mail to it, no longer have mail delayed by it -- you do know that greylisting often delays legitimate mail, right?
Unfortunately, the age of innocence is past, and I have another shocking revelation or you:
There is no silver bullet.
Spam is a solved problem to a very large extent. We are successfully turning essentially all of it away with minimal annoyance to others.
Greylisting does delay delivery of the initial message from a new correspondent, but it is certainly no workaround - rather it's all about being a bit pedantic about adhering to standards. The workarounds are the odd things you need to do in order to compensate for poor configurations elsewhere. And of course you will need to scale your infrastructure according to expected loads.
We will never have a perfect world, and any method you can devise will have a non-zero error rate. These are the simple facts of life.
There are necessarily costs too, but by using available tools intelligently we minimize the costs.
Re:They try to send, but don't really succeed
on
The New Yorker On Spam
·
· Score: 2, Informative
You seem to be lumping several very different techniques together, thinking it's all about content filtering.
Content filtering costs a lot of cpu, greylisting and stuttering (replying 1 byte at the time) costs our end very little.
The cited techniqes are likely to save you significant costs by discarding the obvious cases at the gateway and letting your computation heavy content filtering deal with five percent or less of the load it is handling at the moment.
All I can say is read the articles. You really don't need that kind of extra gear you are imagining.
They try to send, but don't really succeed
on
The New Yorker On Spam
·
· Score: 3, Interesting
The amount of spam that "spam king" Robert Alan Soloway, indicted under the CAN-SPAM Act, is accused of sending over a period of four years is now pumped out about every 30 seconds, around the clock, around the world.
Well, they're trying to send a lot, but with a proper setup at and around your mail server, you will not be seeing much of it anyway.
Simple greylisting helps a lot, supplemented with greytrapping-generated blacklists (with 24 hour expiry) it's even fun to watch. The last 2-3 percent that actually makes it through to be seen by content filtering gets converted back to free electrons.
I've had a series of blog entries over at bsdly.blogspot.com about this and the conclusion is clear - with a competent system administrator, Spam is a solved problem (Links to other refs inside, follow links).
The essence of the article really boils down to "botnet herders may have the ability to update their DNS info quickly". Possibly makes it incrementally harder to track down every last one of the pwned machines, a tad more if your logs store only resolved names but no IP addresses.
Most/.ers likely knew this already, but I imagine this may be exciting and scary to some suits.
In other words, the world did not change much due to this.
There is not now and will never be such a thing as 'guaranteed email delivery'. SMTP is a collaborative, best effort thing. Read the fine RFCs.
In practice, with the myriad spam fighting methods out there, and the fact that some of the companies which pay up for the service will at some time or other have some of their systems take over by spam sending robots, there *will* be legitimate reasons to not accept (and optionally tarpit) attempts at mail delivery from hosts or networks whose owners have paid up for the 'guaranteed delivery' scheme.
Now, of course it is legitimate to dream about a mail delivery system without SMTP's warts and wrinkles, but this is not it, and it is not going to help solve any real-world problem.
"OpenBSD systems using default installations are vulnerable because the default pre-compiled kernel binary (GENERIC) has IPv6 enabled and OpenBSD's firewall does not filter inbound IPv6 packets in its default configuration."
- the default configuration on a fresh install actually has pf disabled (no two networks are exactly alike, and if you enable pf but do not supply your own rule set, a default rule set which passes dns, ssh and some icmp is enabled). Then again, any sane config with pf enabled will have "block all" as the default action and only pass inet6 if you are actually using inet6. This means that the vast majority of OpenBSD machines out there will have the equivalent of the advisory's block rule in their rule sets already.
"However, in order to exploit a vulnerable system an attacker needs to be able to inject fragmented IPv6 packets on the target system's local network. This requires direct physical/logical access to the target's local network -in which case the attacking system does not need to have a working IPv6 stack- or the ability to route or tunnel IPv6 packets to the target from a remote network."
This narrows the scope quite a bit.
Essentially the sky did not fall this time either, but I can see the OpenBSD developers taking another pass of "reading the code much like the devil reads the bible" over the ipv6 stack and the general neighborhood once again.
However, OpenBSD users have been instructed to update their systems already.
The last time the EU demanded that Microsoft produce usable documentation (as in, sufficient specs to program at least a working prototype implementation of the relevant network protocols), they kept insisting that the EU had demanded that they hand over all their source code. And of course large chunks of the press believed them.
I wonder what story they'll try to feed us this time around.
> If it were not for the money, language, and responsibility issues, I'd move to a Scandanavian country in a heartbeat.
huh? anything with alcohol in it is expensive here, but language is no issue. Most Norwegians at least (and AFAICT Danes and Swedes too) will be more than happy to 'practice their English' at you any time.
> We will not switch to IPv6 until the spam problem is neutralized to a great degree.
Totally irrelevant, but your choice.
> RBLs are the most effective method of stopping spam now.
*BZZT* wrong. RBLs would have been a good idea if there was a way to maintain them actively. Experience shows that none of them are maintaned in any useful way (leaving inactive addresses blacklisted for years in some cases), giving false positives at an alarming rate. Greylisting does work with only trivial to insignificant numbers of false positives (all of them RFC violations and stupid configuration errors), and if you're addicted to blacklists, there are greytrap-based lists available which are purged of anything older than 24 hours.
Moving to IPv6 will not change any of this. Getting rid of the unwashed masses of unmaintained, moron-operated machines with Microsoft products might help ease the spam load, and moving to IPv6 exclusively might actually help achieve that.
Your comments certainly do not match my experience here.
As far as I can tell from my logs, greylisting remains effective, getting rid of certainly no less than 85% per cent (likely more in the 95%+ range) of the spam attempts. Whatever spam does get past greylisting mostly gets dealt with by content filtering (spamassassin/clamav). Then again, there are several greylisting implementations, and the one you are using is perhaps not very good to start with or not too well tuned.
I did notice a few months back that spammers started trying to inject their payload via secondary or even teritary MXes for the domain they target. After a few weeks of that, I intorduced a setup rougly like the one described above on our secondaries, and it pretty much killed the remainder of our spam.
The manuscript at http://www.bgnett.no/~peter/pf/ is for a half day tutorial in setting up OpenBSD's PF firewall (also available on FreeBSD, NetBSD and DragonFlyBSD).
The response I get (yes, I'm the guy who wrote the tutorial) is that people find it quite useful.
The fact that it includes a few tips on how to give spammers a hard time helps too I guess.
It certainly isn't new, and IIRC the main reason why Linus ended up owning the Linux trademark was that somebody else had registered the Linux trademark sometime in 1996 and soon started sending letters to among others Linux Journal, Red Hat and basically anyone in the GNU/Linux busines demanding -- what else -- royalties. He didn't get anywhere among other things because Linus et al could prove that they had used the "Linux" name as early as 1991. Googling on "Della Croce Linux trademark" will turn up enough backround (but only 621 links nowadays) for anyone interested.
Once you own a trademark, the next phase (at least in the US) is to make sure it stays protected by policing the use.
There was a time when a virus could install itself just be latching onto a 3.5"
You had 3.5" floppies?
5 1/4"-floppies (1.2M) were the norm, and 8" ones weren't entirely dead yet either. Back then.
infect tons of machines without anyone having the slightest clue as to its existence.
Technically they possibly could pass unnoticed, but most of the viruses back then would do something to attract attention. Like displaying a low-res graphic, hiding the cursor, or trying to delete files or zap hard disks. Virus coders were generally attention-seekers too.
The piece is certainly one of the more well written of its kind, but it is very far from being an unbiased, fact-based article.
Quite a bit of the article seems to be based on the SCO style of reasoning, essentially that anyone who receives and uses code which was originally taken from somewhere without the owner's permission risks being sued for copyright infringement in the SCO case, patent infringement in the hypothetical case put forward by this article. The article then goes on to stress how incredibly costly patent searches are and how this makes open source software very expensive to use.
Their web site did not offer any obvious references to who funded the piece. It's a bit sad they feel the need to smear by proxy.
What about the contract dispute between IBM and SCO?
The probability that SCO is actually able to show that IBM copied SCO code into linux is, mildly put, fairly low.
We do not know if IBM may have done other things which were incompatible with their contract with SCO, and SCO has not been willing to specify any other alleged breaches as far as I know.
I'm not too familiar with the US legal system, but isn't SCO required to state rather accurately what acts by IBM they consider breach of contract, and which parts of the contract are breached by these actions?
Or do the procedures of US lawsuits allow introducing fresh allegations as you go?
My advice is, either wait until the disk 1 image appears on your local mirror, or, if you're really impatient, download the floppy images and go from there. (if you've got the bandwidth to download iso images, network install via ftp or http should be just as feasible).
Disk 2 is a live filesystem disk of a base system install, mainly good for rescues etc. This means no extra packages etc, making the image smaller than disk 1, which contains installer, various packages such as kde etc. If the transfers started at the same time, it makes sense that the smaller file would be completely transferred first, appearing to the public before the larger disk 1 image.
Slashdot Mirror
> Now this internal blacklist is then shared to all the other customers who use Spamshark, so they are now protected too; resulting in a 5 nines hit rate on spam.
And more false posistives than you would actually like to have. I've been at the business end of one of Frontbridge's blacklists. One of the domains I admin got blacklisted a full three weeks after the hosting company screwed up and let phishers set up a paypal scam site as the "test1" user to live for all of 22 hours. Three weeks later, one of the company's main customers, who happens to be a frontbridge customer, is no longer able to receive mail from us. A an unfinished writeup is at bsdly.net - I just gave up in disgust after trying to write an article about the incident.
Subject says it all, really. The best approach is to set up an OpenBSD machine as your gateway, filter traffic using PF to any degree you desire, and please set up spamd in greylisting mode (the default).
That will take care of most of your spam right there, and you could usefully have something like a spamasassin and clamav combo running in the delivery phase on your real mail server.
Useful references: Firewalling with OpenBSD's PF (tutorial)
The Book of PF
and Effective spam and malware countermeasures: Network noise reduction using free tools
And yes, I've blogged a bit about this too, over at my blog
It would have been very nice to see a slashdot review, but for obvious reasons I can not contribute one myself :)
We see a lot of junk hitting our greylist at the gateway, but 95% just ends there.
OpenBSD's spamd is a wonderful greylister, and it offers a few other options which will
make a dent in the reminaing few if you can be bothered to set it up. See my blog at
http://bsdly.blogspot.com/ and links from there for some examples.
If you're already dealing with XML files, I would suggest that the main barrier to using a toolset such as DocBook (SGML or XML variants) should be gone already.
DocBook is excellent at enforcing proper structure and contains all the elements you need (really!) to write tech documentation.
Several high profile projects such as FreeBSD, KDE, GNOME and others use DocBook as their main doc format, as do I believe more tech companies than actually want to admit it. I maintain the PF tutorial at http://home.nuug.no/~peter/pf/ as DocBook SGML myself.
The tools most people use for DocBook are free (most likely just a few mouse clicks or commands away through your package system), but some proprietary/commercial tools are available too. The main reference is at docbook.org, it certainly would not hurt to check it out.
It's not solved. As long as it's taking a measurable number of people working full-time to "solve" it, it's not solved. It'll be solved when we no longer have to spend huge chunks of bandwidth on it, no longer lose mail to it, no longer have mail delayed by it -- you do know that greylisting often delays legitimate mail, right?
Unfortunately, the age of innocence is past, and I have another shocking revelation or you:
There is no silver bullet.
Spam is a solved problem to a very large extent. We are successfully turning essentially all of it away with minimal annoyance to others.
Greylisting does delay delivery of the initial message from a new correspondent, but it is certainly no workaround - rather it's all about being a bit pedantic about adhering to standards. The workarounds are the odd things you need to do in order to compensate for poor configurations elsewhere. And of course you will need to scale your infrastructure according to expected loads.
We will never have a perfect world, and any method you can devise will have a non-zero error rate. These are the simple facts of life.
There are necessarily costs too, but by using available tools intelligently we minimize the costs.
You seem to be lumping several very different techniques together, thinking it's all about content filtering.
Content filtering costs a lot of cpu, greylisting and stuttering (replying 1 byte at the time) costs our end very little.
The cited techniqes are likely to save you significant costs by discarding the obvious cases at the gateway and letting your computation heavy content filtering deal with five percent or less of the load it is handling at the moment.
All I can say is read the articles. You really don't need that kind of extra gear you are imagining.
Well, they're trying to send a lot, but with a proper setup at and around your mail server, you will not be seeing much of it anyway.
Simple greylisting helps a lot, supplemented with greytrapping-generated blacklists (with 24 hour expiry) it's even fun to watch. The last 2-3 percent that actually makes it through to be seen by content filtering gets converted back to free electrons.
I've had a series of blog entries over at bsdly.blogspot.com about this and the conclusion is clear - with a competent system administrator, Spam is a solved problem (Links to other refs inside, follow links).
The essence of the article really boils down to "botnet herders may have the ability to update their DNS info quickly".
/.ers likely knew this already, but I imagine this may be exciting and scary to some suits.
Possibly makes it incrementally harder to track down every last one of the pwned machines, a tad more if your logs store only resolved names but no IP addresses.
Most
In other words, the world did not change much due to this.
There is not now and will never be such a thing as 'guaranteed email delivery'. SMTP is a collaborative, best effort thing. Read the fine RFCs.
t work.pdf).
In practice, with the myriad spam fighting methods out there, and the fact that some of the companies which pay up for the service will at some time or other have some of their systems take over by spam sending robots, there *will* be legitimate reasons to not accept (and optionally tarpit) attempts at mail delivery from hosts or networks whose owners have paid up for the 'guaranteed delivery' scheme.
This is some of the stuff I was on about in my BSDCan paper (now accessible at http://home.nuug.no/~peter/malware-talk/silent-ne
Now, of course it is legitimate to dream about a mail delivery system without SMTP's warts and wrinkles, but this is not it, and it is not going to help solve any real-world problem.
"OpenBSD systems using default installations are vulnerable because the default pre-compiled kernel binary (GENERIC) has IPv6 enabled and OpenBSD's firewall does not filter inbound IPv6 packets in its default configuration."
- the default configuration on a fresh install actually has pf disabled (no two networks are exactly alike, and if you enable pf but do not supply your own rule set, a default rule set which passes dns, ssh and some icmp is enabled). Then again, any sane config with pf enabled will have "block all" as the default action and only pass inet6 if you are actually using inet6. This means that the vast majority of OpenBSD machines out there will have the equivalent of the advisory's block rule in their rule sets already.
"However, in order to exploit a vulnerable system an attacker needs to be able to inject fragmented IPv6 packets on the target system's local network. This requires direct physical/logical access to the target's local network -in which case the attacking system does not need to have a working IPv6 stack- or the ability to route or tunnel IPv6 packets to the target from a remote network."
This narrows the scope quite a bit.
Essentially the sky did not fall this time either, but I can see the OpenBSD developers taking another pass of "reading the code much like the devil reads the bible" over the ipv6 stack and the general neighborhood once again.
However, OpenBSD users have been instructed to update their systems already.
The last time the EU demanded that Microsoft produce usable documentation (as in, sufficient specs to program at least a working prototype implementation of the relevant network protocols), they kept insisting that the EU had demanded that they hand over all their source code. And of course large chunks of the press believed them.
I wonder what story they'll try to feed us this time around.
> If it were not for the money, language, and responsibility issues, I'd move to a Scandanavian country in a heartbeat.
huh? anything with alcohol in it is expensive here, but language is no issue. Most Norwegians at least (and AFAICT Danes and Swedes too) will be more than happy to 'practice their English' at you any time.
> We will not switch to IPv6 until the spam problem is neutralized to a great degree.
Totally irrelevant, but your choice.
> RBLs are the most effective method of stopping spam now.
*BZZT* wrong. RBLs would have been a good idea if there was a way to maintain them actively. Experience shows that none of them are maintaned in any useful way (leaving inactive addresses blacklisted for years in some cases), giving false positives at an alarming rate. Greylisting does work with only trivial to insignificant numbers of false positives (all of them RFC violations and stupid configuration errors), and if you're addicted to blacklists, there are greytrap-based lists available which are purged of anything older than 24 hours.
Moving to IPv6 will not change any of this. Getting rid of the unwashed masses of unmaintained, moron-operated machines with Microsoft products might help ease the spam load, and moving to IPv6 exclusively might actually help achieve that.
As far as I can tell from my logs, greylisting remains effective, getting rid of certainly no less than 85% per cent (likely more in the 95%+ range) of the spam attempts. Whatever spam does get past greylisting mostly gets dealt with by content filtering (spamassassin/clamav). Then again, there are several greylisting implementations, and the one you are using is perhaps not very good to start with or not too well tuned.
http://www.bgnett.no/~peter/pf/en/spamd.html gives you a rough idea of how to do MTA independent greylisting with OpenBSD's PF (doable on other BSDs as well).
I did notice a few months back that spammers started trying to inject their payload via secondary or even teritary MXes for the domain they target. After a few weeks of that, I intorduced a setup rougly like the one described above on our secondaries, and it pretty much killed the remainder of our spam.
The response I get (yes, I'm the guy who wrote the tutorial) is that people find it quite useful.
The fact that it includes a few tips on how to give spammers a hard time helps too I guess.
It certainly isn't new, and IIRC the main reason why Linus ended up owning the Linux trademark was that somebody else had registered the Linux trademark sometime in 1996 and soon started sending letters to among others Linux Journal, Red Hat and basically anyone in the GNU/Linux busines demanding -- what else -- royalties. He didn't get anywhere among other things because Linus et al could prove that they had used the "Linux" name as early as 1991. Googling on "Della Croce Linux trademark" will turn up enough backround (but only 621 links nowadays) for anyone interested.
Once you own a trademark, the next phase (at least in the US) is to make sure it stays protected by policing the use.
You had 3.5" floppies?
5 1/4"-floppies (1.2M) were the norm, and 8" ones weren't entirely dead yet either. Back then.
infect tons of machines without anyone having the slightest clue as to its existence.
Technically they possibly could pass unnoticed, but most of the viruses back then would do something to attract attention. Like displaying a low-res graphic, hiding the cursor, or trying to delete files or zap hard disks. Virus coders were generally attention-seekers too.
This story was posted yesterday too, wasn't it?
I see this story mainly as a reminder that your default firewall policy should be to block. Then open up only what you need.
Seriously, 617 may be a very nice number, but the number of host with a real need to access that port on your machine is likely to be a short one.
Oh well. See http://undeadly.org/ for links to a vaguely relevant lecture / tutorial.
> Does that mean he from the marketing department?
& mode=classic ?
why wouldn't he be?
BTW, does anybody remember http://ars.userfriendly.org/cartoons/?id=20000228
The piece is certainly one of the more well written of its kind, but it is very far from being an unbiased, fact-based article.
Quite a bit of the article seems to be based on the SCO style of reasoning, essentially that anyone who receives and uses code which was originally taken from somewhere without the owner's permission risks being sued for copyright infringement in the SCO case, patent infringement in the hypothetical case put forward by this article. The article then goes on to stress how incredibly costly patent searches are and how this makes open source software very expensive to use.
Their web site did not offer any obvious references to who funded the piece. It's a bit sad they feel the need to smear by proxy.
It's a deamon, actually.
One example of what can happen can be found at http://www.hut.fi/~hynde/humor/DevilStory.html (it's also in Greg Lehey's CFBSD, btw)
The probability that SCO is actually able to show that IBM copied SCO code into linux is, mildly put, fairly low.
We do not know if IBM may have done other things which were incompatible with their contract with SCO, and SCO has not been willing to specify any other alleged breaches as far as I know.
I'm not too familiar with the US legal system, but isn't SCO required to state rather accurately what acts by IBM they consider breach of contract, and which parts of the contract are breached by these actions?
Or do the procedures of US lawsuits allow introducing fresh allegations as you go?
My advice is, either wait until the disk 1 image appears on your local mirror, or, if you're really impatient, download the floppy images and go from there. (if you've got the bandwidth to download iso images, network install via ftp or http should be just as feasible).
Disk 2 is a live filesystem disk of a base system install, mainly good for rescues etc. This means no extra packages etc, making the image smaller than disk 1, which contains installer, various packages such as kde etc. If the transfers started at the same time, it makes sense that the smaller file would be completely transferred first, appearing to the public before the larger disk 1 image.