Slashdot Mirror
Maintaining a Publicly Available Blacklist - Mechanisms and Principles
badger.foo writes "When you publicly assert that somebody sent spam, you need to ensure that your data is accurate. Your process needs to be simple and verifiable, and to compensate for any errors, you want your process to be transparent to the public with clear points of contact and line of responsibility. Here are some pointers from the operator of the bsdly.net greytrap-based blacklist."
Use greylisting as a first defense - easily configurable in postfix, and it reduces the amount of spam dramatically. This relies on the behavior of the sender, rather than someone else's opinion of them.
I want to delete my account but Slashdot doesn't allow it.
And while we're at it, some hints on using a public blacklist with regards spam. The correct way is not to trust the blacklist 100%. Instead, you use it as one part of a comprehensive scheme (part of this complete breakfast). So, you may use a dictionary, and for every word in the dictionary you add 10 points (viagra, v1agra, v14gr4, etc.). You can use SPF and if it doesn't match, then that's worth 50 points, and if it's not there, maybe 20 points. And if the domain or IP address is on a blacklist, maybe 40 points. You assign the points as you like. Then, if you hit 100 points, you mark the email as "probably spam".
But you never reject or mark an email spam just because it's on some blacklist. That's just stupid. Now I'm off to RTFA.
----
OK if you have your own blacklist (perhaps a list of domains or IP addresses that have sent email to a catch-all, or that have fallen into a honeytrap), then you do what you want. But you probably should date entries and remove old ones (if they do not misbehave again), in case a legitimate user is now at that location.
HELP MY ACCOUNT HAS BEEN HACKED BY AN ILLIBERAL ART STUDENT SET TO DESTROY THE INTERWEBZ!
AC is yelling profanities at himself again...
The nice thing about yelling at yourself is you can be pretty sure someone is listening, even if they don't agree with you.
. Your process needs to be simple and verifiable,
The process can't be simple because spammers are endlessly creative with how they try to get past the filters. And if it was verifiable, that would mean published -- and once published, becomes useless. Spammers can simply test their latest creation against your filter, and now you effectively have given them a way to bypass your entire process, making it worthless.
and to compensate for any errors, you want your process to be transparent to the public
The administrative process can be transparent, but the technical process, as outlined above, cannot.
with clear points of contact and line of responsibility.
The problem here is; how do you tell the liars from the rest? Responsibility is fine, clear points of contact are fine, but what's the criterion for delineating between 'spam' and 'marketing'? How about between 'spam' and 'opt-in' that the user no longer wants? How about between... you get the idea. There is some grey here, and odds are good you're going to find someone doing something with a legitimate and ethical reason, that by all appearances... isn't. And then you're going to make a decision based on those appearances (because what else can you go on?) and then you're going to burn a bridge down.
These problems can't be solved with a handwave and a post on an internet forum.
#fuckbeta #iamslashdot #dicemustdie
most of your spam problems will be solved by simply blocking all email from those countries except for your business partners
First off, because spam is so bad (80% of messages by some counts) just about ANYTHING that ANYONE does will reduce their spam (ignoring false positives).
Secondly, READ YOUR LOGS!
There are broad categories of how different groups use email (and their email infrastructure). So what works great for one group sucks for a different group.
So I recommend something like SpamAssassin where you can tweak the settings to what works for your specific circumstances (and the people/groups that you send/receive email with).
Greylisting is great, except when you try to greylist gmail servers. So know how the tools work and think about situations where they would fail and then adjust the knobs to deal with those potential failures.
And if you don't accept EVERY email sent to you (I don't) then make sure that you customize the rejection notice so that the SENDER can contact you if his server includes the rejection message (which most of them do). I include my phone number.
In my opinion, the more knobs that you can adjust the better it is.
in addition to making sure your data is accurate.
... though they are not publicly-accessible; only accessible to our customers. Here's how they work:
Using our reputation-collection protocol, we receive a constant stream of events from our customers. An "event" is something like "IPv4 address x.y.z.w sent to a nonexistent recipient" or "IPv6 address abcd::1234 sent something that a human voted as spam"
Currently, we have a database of just under two billion events. Once an hour, we go through our database and categorize IP addresses as:
The whole system is 99.99% automated. The only manual intervention is when some requests delisting. If it seems that someone was the victim of a compromise and has now cleaned up his/her machine, we delist it for 45 days which is long enough for all events from that IP to expire. Then it goes back into consideration for automatic listing.
This system works really well. We have about 3.75 million IPv4 and 3300 IPv6 addresses on our lists; those are machines for which we have confidence that there's enough data to categorize them.
You end up losing mail and who is it for someone else to filter what I can and can't see. There is a delete button for a reason. Use it.
Says the Anonymous Coward.... (Whom probably has an agenda, and is too afraid to post whom he is)....
Spam lists are not just for email. If you are forum admin (or a slashdot admin), spam posts are the bane of many a website.
A good, verifyable, public, and accountable spam list means admins of such sites can spend more time on being productive, rather than wasting time deleting hundreds and hundreds of useless posts and users.
Considering this is slashdot, and the majority of people here prefer technology to work for our benefit, a good spam blacklist makes a hell of a lot of sense !
Another board I frequent, using the Drupal blogging software, is currently being overwhelmed with spam.
.
Our beloved webmaster is experimenting with Mollum spam retarding software
This software does have its faults, as it is hindering the posting of links by some of our most informative posters. A blogsite's "good folk" need to be whitelisted so they can post links unhindered. More often than not, the most informative content of a post is a link.
Anyone else having a blogsite overrun with crap might want to look into this. I do not think its the ultimate solution, but its a start.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
Better solution: Stop trying to force email to be a reliable and concurrent source of information. It has never been reliable nor has it ever been concurrent protocol. Check the default settings for sending email - try every hour for up to 5 days before giving up. Wait one day before sending a trouble report.
That email now generally DOES deliver results in almost real time is no excuse to think it will ALWAYS deliver in real time. If your communication either critical and/or time sensitive, then email is the wrong tool to use.
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
What you describe is SpamAssassin. The scores are learned by feeding a lot of ham and spam and finding the right balance. Of course you can (re)train the scoring with your own ham & spam, and add your own rules, etc.
Perl Programmer for hire
0. Find a system that makes their blacklistings publicly available.
1. Send it SPAM.
3. See what gets through, send more of that from those IPs.
4. Tweak the stuff that didn't get through until it does.
2. V1AGR4 !!
5. Rotate IPs from your pool of thousands that aren't blacklisted.
6. Prophet.
7. GOTO 0.
Protip: Your public blacklist is part of the fucking problem, fool. Either use a whitelist if you can (+trust graphs), or if you can't then let those blacklisted contact you if they care.
No it doesn't. System administrators shouldn't be filtering the mail. It's not up to you do hand-choose what mail goes into your customers mail boxes.
Some people like myself always post AC. You don't have to have an agenda any more than anybody else. You clearly have an agenda and aren't posting AC. Your probably a system administrator who feels overwhelmed by financial constraints, work, bitching from end users, etc. Spam filtering makes you more productive only because there is no better system in place. Nobody said there can be a better system designed and implemented to eliminate spam. Its just the filtering part that is bad. How anybody can claim it isn't is beyond me. Filtering sucks- we shouldn't have to filter in the first place and if things were designed today these problems would be taken into consideration.
There should simply be a better system for authenticating anonymous users. We need a better system for authenticating anonymous users of forum posts and email users. That doesn't mean deanonymizing though. You can have an authentication system which makes it easy to eliminate mail from spamers. There is already such systems in place (I'm a new anonymous user, but joe trusts me, and saly trusts joe, so i can send saly an email, etc).
Botnets generally don't use IP addresses, but host-domain names instead: Why? For the purposes of "fastflux" botnet construction
So - what's that? Well, put it THIS way:
The "infamous they" (law enforcement or other authories online etc.) take 1 out?
Well, no big deal!
Just "jump" to another node on your botnet in some 'enslaved' system(s) you have in it! This is done @ the botnet C&C (command & control) server master level.
(Which of course, your botnet's infestors on clientrigs in it also has the ability to 'serve up' your bogus 'site(s)' from it & ANY ONE OF THEM...).
* Doing it THAT way's is a LOT tougher to "take out" than hardcoded IP addresses is why...
(Which as you yourself noted, are fairly EASY to blacklist out, & from a LOT higher levels than ISP's even)...
You MAY want to read more, here -> http://en.wikipedia.org/wiki/Fast_flux
I've been building my list since 1997, & see what gets used MOSTLY from 15 or so reputable sources for my data (and the rest comes from security articles from sources such as threatpost or sophos, among others).
Now - THAT bugged me to NO end, as to WHY they used host-domain names instead of IP addresses mostly, but once I got wind of that about a decade++ ago? It made sense...
APK
P.S.=> That answer anything for you? I hope so... & it's also WHY I use what I wrote here -> http://yro.slashdot.org/comments.pl?sid=3647643&cid=43447983 in custom hosts files (which work against bogus adbanners, maliciously coded sites/servers, or hosts-domains serving up the same or malwares even, and yes, spammers/phishers too)...
... apk
Your post advocates a
(X) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
(X) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
(X) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(X) Users of email will not put up with it
(X) Microsoft will not put up with it
( ) The police will not put up with it
(X) Requires too much cooperation from spammers
(X) Requires immediate total cooperation from everybody at once
(X) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
(X) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
(X) Lack of centrally controlling authority for email
(X) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(X) Asshats
(X) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
(X) Susceptibility of protocols other than SMTP to attack
(X) Willingness of users to install OS patches received by email
(X) Armies of worm riddled broadband-connected Windows boxes
(X) Eternal arms race involved in all filtering approaches
(X) Extreme profitability of spam
(X) Joe jobs and/or identity theft
(X) Technically illiterate politicians
(X) Extreme stupidity on the part of people who do business with spammers
(X) Dishonesty on the part of spammers themselves
(X) Bandwidth costs that are unaffected by client filtering
(X) Outlook
and the following philosophical objections may also apply:
(X) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(X) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
(X) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!
Six years ago, I wrote milter-greylist. At that time I thought some kind of distributed spam traps would be useful. I wrote software for a P2P network of mail servers that exchange signed information on messages reaching spam traps. The thing turned to be useless: greylisting alone was enough. Today, greylisting with variable delays depending on sender reputation from various DNSRBL is still enough, even is the DNSRBL information is not very reliable: an error just means an extra delay in delivery.
If email were supposed to be instant, nobody would have invented Instant Messaging. Email is designed to be reliable instead of instant. That's exactly why instant messaging was invented 15 years after email was, because email was not, is not, and is not designed to be instant. It's designed to be efficient and reliable. Read the protocols some time. Have a look at how send mail works. Queues to send, queues to relay, queues to receive.
I used to receive some 20K mails a day.
When unfiltered, the inflow is faster than my ability to manually delete mails.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
At least one thing positive:
(X) Microsoft will not put up with it
Isn't all that relevant anymore.
Let's update it;
(X) Google will not put up with it
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
blacklists are awesome and who gives a shit about the occasional lost mail every now and then? neurotic obsessed people, that's who.
Businesses, for one. You wouldn't like to lose a sale because you didn't get the request for a price offer, for instance.
Not all email is snail-social-network garbage; some people have useful things to say.
Though in reality, using a set of blacklists instead of a single one pretty much nullifies any mistakes in the individual lists.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
I'm running a physical server in a colo. Unfortunately about 6 months back my server sent a burst of about 5,000 spam messages. I was getting bounce messages on my admin account but with no information as to which account was breached in the bounce message, I'm scrambling about on my system, first shutting down mail, then trying to figure out if I was even sending it or just a victim of a one of the spam tricks. I did see entries in my logs, but I wasn't able to track it down to a specific account. During a second spam run about a month later, being suspicious, I had a copy of all outbound messages being backed up and discovered there was a script someone had uploaded into the root directory of a forum my wife has set up for her hobby. I tracked it down from one of the headers in the spooled messages. I found and killed the script, then did some further work to lock down the directory to keep it from happening again. So far, according to logs, nothing further has gone out.
Unfortunately now I'm on several (dozen?) dns blacklists with no apparent way to get back off. Some will let me query their records (and sure enough I'm in there; I'm not disputing that) but many others just leave it in place based on the bounces I'm getting from pretty much every mail service. So even though I've found the problem and killed it, it seems I can never send e-mail from this IP again.
I've investigated moving to a different ISP but none really give me the control I want (I have full root access to a physical system). And pricewise it's less expensive than a virtual machine at Amazon.
[John]
Shit better not happen!