Slashdot Mirror
University Capitulates, Switches Off Spam Filters
Heraklit writes "As reported on German news site Heise, the system administrators of the Technical University of Braunschweig have temporarily given up the fight against spam. Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!), they decided to switch off all filter mechanisms. Before, the 20 servers dedicated to processing e-mail alone had been breaking down under a load of 100000 unprocessed mail messages, ca. 98% of which had been spam or viruses. ... A similar e-mail jam occurred recently at the IT central of the German Federal Government.
Is this the beginning of the end of e-mail?" (The Fish may be useful.)
Site's a little slow -
Akamai Mirror.
Wait, don't tell me.
1: They refused to use blacklists to cut the load.
2: They refused to publish SPF records and use SPF to block all the email forged to look like it's from their domain, significantly cutting the spam load.
3: They used one of those "commercial-grade" virus/spam mail scanners that's designed to use entirely Bayesian scanning without ever setting time-outs on the generated rules, and which was written for "completeness", not speed.
4: They forgot to set up a honeypot machine to auto-block spam domains.
6: They underbudgeted for the servers to actually do the mail handling, forgetting to set up up appropriate MX records with good fallover behavior, so when any of their served domain's MX record listed machine blinked that entire domain went offline.
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
www.spamgourmet.com has always worked well for me. Give your adress to whom you want, receive just as much mail from them as you want.
MS Exchange servers. It's gotta be MS Exchange servers: no other SMTP server in the world could possibly require 20 servers to deal with only 100,000 emails an hour, even with only 1 GHz mail servers. Sendmail, Postfix, Qmail, all could handle 100,000 emails an hour on only 10 such machines, even running SpamAssassin and CRM114. Unless maybe they skimped on RAM and accepted vastly oversized mail messages, in which case they'd start swapping themselves to death at a lower than expected threshold.
I hope they find the idiot who selected their servers and software combinations and send them straight back to Redmond, in a box, along with the snipped off tie of the Microsoft person who sold them the bill of goods.
"All you have to do is be fragile and grateful. So stay the underdog." Chuck Palahniuk, Choke
Why dont these people start using reverse DNS to MX record verification? It checks to make sure the machine sending you email has a real reverse DNS that matches their MX record. If not, it disconnects. Combine that with the real time black hole list and you'll never see spam again! This mail package does it: Icewarp
Some university departments run on email. If you don't deliver reliably, you could create chaos in some classes.
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
Nah, it's sendmail:$ dig -t MX tu-bs.de
[...]
tu-bs.de. 172738 IN MX 10 rzcomm5.rz.tu-bs.de.
$ telnet rzcomm5.rz.tu-bs.de smtp
Trying 134.169.9.40...
Connected to rzcomm5.rz.tu-bs.de.
Escape character is '^]'.
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 04:00:51 +0200 (METDST)
GROGGS: alive and well and living in
Permanant Failure (5xx SMTP) codes are not safe either.
There are many cases where email is relayed before being sent to a system that does virus scanning. (Consider what happens when you use sendmail aliases and virtual domain entries that contain somthing on the order of "user: user@someotherhost.com".)
Your SMTP 5xx error will cause the relaying server to generate a bounce. The bounce will go to the person listed by the forged "To" headers, and will even include a copy of the Virus.
The proper way to deal with email worms is to quietly delete them.
I seem to recall the whitepaper about this method being posted on slashdot a while back. My free email provider (softhome.net) implements this and it works ok but it still lets quite a bit through. It seemed like a great idea in theory though.
"It is not how things are in the world that is mystical, but that it exists." -Ludwig Wittgenstein
I use Cashette for my email server. It's free, allows POP access, gives you the ability to activate its systems on other email accounts, and it works by using an authorization system. Basically, if someone isn't on your "authorized" list, then their mail gets put into a special folder. You can either review what's in that folder, or just forget about it. Here's the nifty part... If a spammer REALLY wants to get their message to you, they can pay you for delivery. You set the price, up to $300 for them to get their message to you.
You can get your own account at http://www.cashette.com/
Have fun!
-Phyre
"When the people fear their government, there is tyranny; when the government fears the people, there is liberty." -Thom
Simple problems have simple solutions.
You can increase the threshhold at which you declare spam to be spam. Allows for more misses, but reduces the false positives to, essentially, nothing.
Or, you can just tag likely spam with ***SPAM*** in the subject and let the user deal with it.
Or even better, you can direct likely spam into a specific IMAP folder on the server that the user's client can subscribe to and they can glance at their personal SPAM folder on the server whenever they want without having to download all the bodies.
As someone who personally uses postfix+procmail+spamassassin+razor and recieves 4,000 emails per day, I am currently filtering out 98% of the spam on the server and have had ZERO false positives in two years and 2.9million messages.
Statistically, you will eventually get some false positives - especially if you have a large userbase (as opposed to just one or two accounts). But if one out of every few million messages isn't acceptable, you can just use one of the previously suggested methods.
The worst you can do is nothing at all.
No, Sendmail:
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 06:46:39 +0200 (METDST)
This isn't even the beginning of the end of email. It's simply becomming less and less workable to run a single mail server system with a large amount of users. Small time mail servers aren't targeted by spammers. Universities are heavily targeted because there are lots of users all going to a common domain.
It's the same reason users of major ISPs are more likely to be probed for vulerabilities.
I've found the method of filtering based on the "Click-Me" domains to be the most effective with virtually no false positives (zero is a realistic number).
I've found that setting up a secure public mail system is cake. Mercury Mail is free and handles well. A single check box set by default is all it takes to keep it from being an open relay. Students of the university could probably do rather well offering their own e-mail services to students. Mercury Mail's filtering system is quite robust.
MM supports IMAP/POP3/SMTP and alternate ports as well as SSL on all them. Adding a web-based front end also isn't that difficult if you know what you're doing. There's actually one built in and a more robust version coming.
I already have a few hundred users on Indie-Mail and the amount of bandwidth used per day is pretty negligable.
Ben
Work Safe Porn
Make your boss happy, and block on these three DNS based lists: dsbl.org, spamhaus.org, dnsbl.org. Everything coming from IP addresses in these range is basically garantueed not to contain false positives. It'll clear your inbox quite effectively. (I'm one of the volunteers helping out dsbl.org, so feel free to mail me with questions)
-John
The measured UBEs over a 3 moth period were 172,887 - only for their top-25 most spammed employees!
Notepad specialist & FAT administrator, group training available
Two things:
The approach that we take is the following: We mark virus messages with a special header and deliver them in a dedicated folder in the user's mailbox. Most users simply delete all messages in this folder, but then it is their choice, we abide to all laws and do not generate bounce messages.
Sebastian
To quote the post directly above you ...
No, sendmail (Score:5, Informative)
by marnanel (98063) on Monday May 24, @12:04PM (#9234290)
(http://marnanel.org/)
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
Nah, it's sendmail:
$ dig -t MX tu-bs.de
[...]
tu-bs.de. 172738 IN MX 10 rzcomm5.rz.tu-bs.de.
$ telnet rzcomm5.rz.tu-bs.de smtp
Trying 134.169.9.40...
Connected to rzcomm5.rz.tu-bs.de.
Escape character is '^]'.
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 04:00:51 +0200 (METDST)
I don't know what kind of machine you are running but we have SA running on it's own machine, serving two mail servers. It handles over 300K messages a day with network tests enabled, and the standard scan time is sub 1 second.
If you are going to be running SA with any kind of volume you need to keep in mind...
1 - Run a local DNS caching server. dnscache works well, give it lot's of memory to play with
2 - Rsync and run as many of the RBL's locally as you can.
3 - Set the max number of children that SA is allowed to spawn, on our hardware that number is about 12.
4 - Lot's of memory! Depending on the number of max children, you might want 1 gig or even 2 gigs of memory
5 - Off load SA on to it's own dedicated machine, so if need be you can easily inject another server using hardware or dns round robin load sharing.
I don't know what kind of volume the Uni was handling but with 20 machines I think I could easily handle upwards of 20 million deliveries per day.