Slashdot Mirror
University Capitulates, Switches Off Spam Filters
Heraklit writes "As reported on German news site Heise, the system administrators of the Technical University of Braunschweig have temporarily given up the fight against spam. Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!), they decided to switch off all filter mechanisms. Before, the 20 servers dedicated to processing e-mail alone had been breaking down under a load of 100000 unprocessed mail messages, ca. 98% of which had been spam or viruses. ... A similar e-mail jam occurred recently at the IT central of the German Federal Government.
Is this the beginning of the end of e-mail?" (The Fish may be useful.)
Does anybody know the filtering methods they were using before they decided to toss everything to wind?
As Seen On TV's? Come back!!!
Just white list known good addresses. Hand out auth tokens (X-Not-Spam: md5 digest here) and white list those temporarily. And white list known good PGP keys.
Byebye, spam.
Byebye, email.
If you don't filter out any of the spam, then some mail server somewhere is gonna have to store all that junk mail. Even with a quota system I'd expect that there'd be a whole bunch of people just "giving up" on e-mail.
"they decided to switch off all filter mechanisms"
Finally, I can get my "male enhancement" emails again.
what sort of awful sound the servers made as soon as the filters were turned off? ...I imagine it would be akin to someone who 'just' made it in a mad dash to the bathroom.
Perhaps just disabling spam filters and leaving virus blocks in place would be a less drastic approach. Detecting spam is non-trivial, but detecting viruses is not. They are easily found and the email should be blocked. This is implemented by my ISP (Road Runner NYC). Emails containing viruses are replaced by a text message warning that a virus was sent to the email address.
Site's a little slow -
Akamai Mirror.
...is to inform the students how to install their own software, like Spam Assassin. That would distribute the processing to the people who actually would use it.
Read journal when you are not understand
> Is this the end of email?
:), so I think that responibility is the key to keeping email working. Adding some numbers (*sigh*) helps guard against random address guessing.
Yes. When one university decides to stop filtering SPAM the entire world's infrastructure has effectively been shut down. Oh wait... no.
My UIC account gets NO spam (because I don't give it to anyone
Anyway I don't see anyone stopping you from using your own SPAM filter. Let's not blow this out of proportion, please.
My other car is first.
Maybe they should just blacklist the most common spam and virus adresses by subnet then filter on a lower percentage. It would seem that if they got rid of china or some other area like what happend recently with spain, it would send a message to those networks to stop things and bear some of the weight.
Excuse me? One university gives up on spam filtering for questionable reasons and you declare death of email? Weird, I still do most of my communication via email. My servers all run spam marking services and my client filters out the junk as soon as it's retrieved.
:)
Of course more bandwidth is wasted on spam mails, but since I don't see much of it, it doesn't bother me so much.
What do you propose to use instead of email? instant messaging? Talk about waste of time
-- shortcut - the longest distance between two points.
Is this the beginning of the end of e-mail?
I seriously doubt e-mail will ever die. It's FAR too convenient to just give up on. Even if it comes to the worst case scenario where you have to whitelist everyone who wants to send you e-mail, it'll never go away.
Here at the university where I am a sysadmin, we get approx. 100K emails per day and we have no problem pushing them through spamassassin on a single server with dual 2.8 xeon processors. How in the world could this place possibly need 20 servers to process this much mail?!
Wait, don't tell me.
1: They refused to use blacklists to cut the load.
2: They refused to publish SPF records and use SPF to block all the email forged to look like it's from their domain, significantly cutting the spam load.
3: They used one of those "commercial-grade" virus/spam mail scanners that's designed to use entirely Bayesian scanning without ever setting time-outs on the generated rules, and which was written for "completeness", not speed.
4: They forgot to set up a honeypot machine to auto-block spam domains.
6: They underbudgeted for the servers to actually do the mail handling, forgetting to set up up appropriate MX records with good fallover behavior, so when any of their served domain's MX record listed machine blinked that entire domain went offline.
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
"Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!)"
Is it just me or is this another ridiculous law? The University is providing free email services to those that are students at this establishment and they obviously need to filter out spam in order to be able to offer this service with there current hardware requirements. Spam is a legitimate problem and people that are offering free email services should be allowed to attempt to filter it as it can be extremely taxing on a busy mailserver. They can filter the spam without being intrusive or breaking privacy laws so I see no reason that it should be prevented by law.
Personally, if it were my universtiry, I would prefer they started to use a RTBL. The fact of the matter is, if the likely spam isn't sorted out first, I have to try to discern the stuff entirely by hand. And although I can easily pick out Viagra ads, I have relatives and the occasional acquaintence who send mail that looks awfully like spam. Didn't want to type a subject. Used "hello" as the subject. Didn't configure their mail client properly, so their "replyto" looks crazy. Without some initialy spam filtering, I would miss at least some of these -- in fact, I'd probably miss more mail with no filtering than with a judicious blackhole in front of me.
Love or hate SPEWS and other kinder, gentler RTBLs, they're better than the present choice. It would certainly reduce the load of these email servers to where it could be more easily handled. And, if nothing else, they couldbe used to prioritize mail. Use Spam Assassin or something else to do some initial tag and filter so that mail coming from Asian IPs or originating from mail servers on cable/ADSL networks gets put into the "slow" processing queue while everything else gets sent down the faster pipe.
</spouting with little to no knowledge>
You like splinters in your crotch? -Jon Caldara
www.spamgourmet.com has always worked well for me. Give your adress to whom you want, receive just as much mail from them as you want.
No, but its one more nail in the coffin..
Something has to be done soon or email just wont be practical to have. Between Spam and viruii its overloading a lot of comanines network feed and servers..
And don't forget the cost of having to maintain antispam and antiviral solutions..
I know personally where I'm at, we are hitting over 2/3 of all email is spam/virus. ( i hear we drop 10k a day from the black hole list alone )
At home its 98%...
---- Booth was a patriot ----
One would think that even spammers would realize that if things go too far, businesses might not carry emails at all anymore.
I mean, even parasites usually try to not kill the host.
*sigh*
Failure is not an option. It comes bundled with your Microsoft product. -- Ferenc Mantfeld
I run Exim with an ACL extension called Exiscan, which runs SpamAssassin and virus checker during the SMTP dialogue.
Rejected mails thus don't generate any undeliverable bounce messages to fill up the local mail queue, and the sender gets an immediate response.
MS Exchange servers. It's gotta be MS Exchange servers: no other SMTP server in the world could possibly require 20 servers to deal with only 100,000 emails an hour, even with only 1 GHz mail servers. Sendmail, Postfix, Qmail, all could handle 100,000 emails an hour on only 10 such machines, even running SpamAssassin and CRM114. Unless maybe they skimped on RAM and accepted vastly oversized mail messages, in which case they'd start swapping themselves to death at a lower than expected threshold.
I hope they find the idiot who selected their servers and software combinations and send them straight back to Redmond, in a box, along with the snipped off tie of the Microsoft person who sold them the bill of goods.
University capitulates. /. visitors break down apache server. Oh .. i mean IIS server.
11 1101 1011111 0100 000 110 1011111 0101 10 01 1011111 101 1 011 1011111 0 1111 11 111 1011111 101
Why dont these people start using reverse DNS to MX record verification? It checks to make sure the machine sending you email has a real reverse DNS that matches their MX record. If not, it disconnects. Combine that with the real time black hole list and you'll never see spam again! This mail package does it: Icewarp
I go to Penn State, but since the university feels it has to protect dumb windows users from themselves, I cannot even send or recieve email with the subject, "Hi such-and-such"(Try explaining to a friend overseas who has almost never in her life touched a computer, in her language, why she can't send you mail with that subject) because it might contain the bagle virus. This is the same university that put in a firewall because supposedly too many people on campus had a butt-load of viruses and spyware.
Yet this same university loves to publish my email address on the web; ensuring I get tons of spam(some even in Chinese!)
I hate when the community at large has to pay for the transgressions of a few slimeballs and the idiocy of some(not even most) gullible windows users.
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
Nah, it's sendmail:$ dig -t MX tu-bs.de
[...]
tu-bs.de. 172738 IN MX 10 rzcomm5.rz.tu-bs.de.
$ telnet rzcomm5.rz.tu-bs.de smtp
Trying 134.169.9.40...
Connected to rzcomm5.rz.tu-bs.de.
Escape character is '^]'.
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 04:00:51 +0200 (METDST)
GROGGS: alive and well and living in
I would say this is probably not the end of email, nor is it the end of the Internet as a whole. However, it is probably the end of the protocols currently used to send and receive email.
I believe that spam is ultimately a security issue, because it slows down systems and creates problems for users and system administrators. Sometimes, security problems are caused by buffer overruns and other programming errors. However, in this case, I think the entire protocol is faulty. It may have worked wonderfully before spammers, but it's time to introduce something new that will make it extremely difficult to send spam.
I don't know exactly how the new protocol needs to look. But I have some ideas. Paying for "postage" is not one of them, as I think it is a very bad idea. Unless some payment system could be set up whereby the recipient of the mail receives the payment, not some 3rd party, like Microsoft, which would profit incredibly from garbage spam mails going all over the place. In fact, if that were the setup, then each recipient could state a price per email and/or per kilobyte of the mail message for receiving an email from a source, which the source would pay to the recipient as postage. A whitelist could be set up to allow certain senders, like one's friends, family, coworkers, etc., to send emails without paying the recipient. A blacklist could be set up to disallow all emails from specific senders and/or domains, as we have today, and if you read further in this post, you'll see my ideas for making sure that addresses are not spoofed. But I digress...
Perhaps first of all, the mail headers need to include digital signatures based on the source and destination domain names, email addresses, and other identifying information that is unique to each email sent. To avoid address spoofing, for example, people sending junk with a 'yahoo' or 'hotmail' address, when in fact it originates elsewhere, each such domain would have a private key, which upon sending, would be used in the computation. A valid signature could not be computed when the address is spoofed, and so all spammers would need to use their own valid domain name. Further, the need to make computations would make it more costly for spammers to send mail in high volumes. The algorithm should be designed so that recipients of email will have a much lower cost to verify the key. Further, the signature system could, should, and would be used to verify that each bit of the contents of the email, including all attachments, arrived correctly and without being tampered with or corrupted in transit.
The whole thing about them being legally obligated to deliver mail is the silliest thing I've ever heard. Leave it to the Germans to enact such a law.
Better to just not deliver ANY mail than to deal with that requirement.
the numbers dont add up, Loads of people have already raised the issue about the fact that 20 servers ( even decently mid spec single CPU machines ) will handle 100k emails an hour ( about 80 emails per min per machine is very achieveable ... ).
But there are some other issues you need to look at, with these emails not being scanned - do you know how much of storage you need to have online to have a mailstore this size and developing by the hour at 100k msgs ? not everyonce will use pop3 to get their emails, and not all the users will check email every day. Were talking about a very very large and very well setup Mail Store for this kind of volume. What about network bandwidth ?
A few basic things can reduce the work of those servers : Duplicacy level across these emails is going to be very high - all 100k emails per hour cannot be unique, there are going to be loads and loads of dupes, that dont even need to be scanned.
Creating a small database in-house with bad MailSender's list ( kind of like an in house RBL ), and flushing that list on an 6 hour interval will slow the inflow as well to quite an extent - in some tests done, i have seen it go down by almost 15 - 18% when there is a heavy load. Since most 'real' mailservers tend to retry, even if a genuine mailserver is blacklisted for 6 hours - it wont make much of a difference, however most 'hijacked PC's sending spam' dont have any retry or resending mechanism - and will just not be able to send into your server.
Another issue that helps stem the tide of bad email is to check for Virus infections before checking for spam. A lot of cases the tides of mail coming in can be virus infections ( which are easier and faster to check against - compared to rules + logic based spam checkers ).
However, all this is said and done without knowing of what system and what kind of a setup they use, there is no way anyone can really know what happened and why.
In the end, classic case for Linux and Unix based technologies to come into the frame I think.
Joking aside, it boils down to economics. Spam is profitable. If something is profitable, people will do it. Selling drugs is profitable, and the war on some drugs hasn't changed that. The answer to spam (and drugs) is not to try and stop them, but to make doing them unprofitable.
What makes spam profitable is the presence of people on the internet who are SO incredibly stupid that they fall for it. (See Junkie loves his spam) Remove them, and you shoot spam through it's purtid heart. I can think of several methods of doing do:
- If you respond to spam, you've probably got shitloads of viruses on your computer. Beyond any shadow of a doubt some of them are spamming people. If you ISP detects lots of mystery traffic from you on known virus ports, you're given one warning. Then you get kicked off without ceremony and not allowed to reconnect until you can prove to them that all computers using your connection are malware-free. No more malware, no more spam zombies.
- (You, the ISP) Send test-spams. Specify in the header that it is NOT a real spam so you don't get blacklisted. Anyone who responds to them loses all services except port 80 until you prove to a professional who visits your house that you know enough not to buy from spam. Do it again and you will never be allowed to use your ISP's mail servers again.
Neither of these can possibly be routed around or hacked by spammers, because they are not involved in any part of the process. If you are not in the habit of perpetuating malware with your computer, you needn't worry of getting caught up in it all. Neither of these requires a major invasion of your privacy> /dev/null.
Report that all emails are stored in an infinitismally small location that only future, advanced technologies will be able to restore email upon request. Requests will be queued until the technology has been developed.
spamd is a new approach to blocking spam. Its called greylisting. It rejects all email with a temporary failure notice in the hopes that the large volume spam senders don't have the resources to wait 30mins and send the same email again. Apperently this method works quite well and uses little resources.
The MTA's work is relatively light compared to what anti-spam software must do. This is especially true of SpamAssassin. While it does have some advantages over its competition, SpamAssassin is extremely resource intensive. Firstly, SpamAssassin is not written in fast C/C++ but Perl. Every email is sent through zillions of Perl regex rules. Then there is the Perl implementation of the Bayesian test, which really bogs down when an email auto-learned. Then there are the various (optional) network lookup tests: several RBLs, Pyzor/Razor/DCC ... each email can eat up a lot of resources even if you bypass the startup overhead by running spamd.
I have also seen situations where SpamAssassin was not correctly respecting the maximum child spawn limit. Since spamd is a fairly heavyweight process, the server started swapping and throughput plunged.
Such heavy overhead is not a essential part of anti-spam software. Something NOT written in Perl nor any "interpreted" language, something with a smaller footprint, will be much, much faster. I wonder how many people have switched to dspam for this reason?
Does Germany have a law that I'm not familiar with? Email is free not a paid service, why is there some obligation to deliver? Snail mail is normally Govt. run and delivery is what you pay for with a stamp.
No one has to or could guarantee anything for email. With the amount flowing because of SPAM the dropped packets must be astronomical.
Professional Politicians are not the solution, they ARE the problem.
I use Cashette for my email server. It's free, allows POP access, gives you the ability to activate its systems on other email accounts, and it works by using an authorization system. Basically, if someone isn't on your "authorized" list, then their mail gets put into a special folder. You can either review what's in that folder, or just forget about it. Here's the nifty part... If a spammer REALLY wants to get their message to you, they can pay you for delivery. You set the price, up to $300 for them to get their message to you.
You can get your own account at http://www.cashette.com/
Have fun!
-Phyre
"When the people fear their government, there is tyranny; when the government fears the people, there is liberty." -Thom
Being the person that blocks spam is a lose/lose situation. They don't understand how bad the problem is when you do your job right. They complain when spam gets through and complain when legit email gets blocked, but don't want you wasting all your time on it.
I predict that this school will be forced back into filtering spam by their students (customers).
[rant]See, 3 years ago, as spam was beginning to get bad, I began filtering spam on the email system I manage. Over 2.5 years, I developed a rather intensive filter, but since I knew I was not perfect, I had to scan blocked email for false positives. It got to the point I was spending 25% of my time scanning for false positives and the boss didn't like that. He also didn't want me to spend time trying to figure out how to set up Spam Assassin. (I'm not a Linux guru. Sorry!) The board didn't want to spend the money on a purchased system and didn't want me wasting my time with spam. They didn't think it was a problem so they told me to just stop blocking spam. My boss told them that spam was a BIG problem, but they never saw it so they didn't believe him. I asked my boss 10X "Are you sure you want me to stop blocking spam? They won't like the results." He confirmed. I stopped blocking spam and about 50,000 additional spams per week came flooding into the system. The 50,000 were what was being blocked previously. I was flooded with phone calls until everyone realized what was happening. Then, just 2 weeks ago, I was instructed by the board to go back to my filtering, but only spend 30 minutes a day on it. RIIIIGGGHHHHHTTTTT! Ever try scanning for legit email among the trash, adjusting filters to make it better and taking calls and emails from people that want you to be sure an email is blocked and only spend 30 minutes a day on it? I managed to put together a Spam Assassin box and it blocks 10,000 per week, but there's a lot that doesn't get blocked. I don't know enough about it to make it better.[/rant]
But why is the rum gone?
Plus we have a significant number of staff who are clueless who would be excluded from communicating effectively because they do not have the time or skills to learn how to train a spam filter. in such a situation, no-one could no-longer *rely* on email to contact/inform our staff, reducing its value as a tool.
True, I also work in a large international organisation, but our Spam/Ham ratio in "only" about 40%..
I am handling the Spam problem and we have been running SpamAssassin, as a pilot project, for the last year.
The SpamAssassin project almost got replaced by a commercial solution when people started asking themselves, "what good is it if we still deliver, the Spam to the users inboxes ?". Our users may be experts in other fields, but for many, computers are not their thing.
Some commercial solutions have "Quarantine" system where you can send a report once a day to the recipients, with a list of all spam they received the day before, with a link for each email the user can click if he wants it delivered to his inbox.
It took me 4 days, but I wrote my own Quarantine system that does exactly that, and got permission to release it under the GPL..
That way the Spam doesn't constantly flow in the user's inboxes and takes up the users time. (And, 'no' manually creating a filter rule for thousands of users is not an option)
echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
This isn't even the beginning of the end of email. It's simply becomming less and less workable to run a single mail server system with a large amount of users. Small time mail servers aren't targeted by spammers. Universities are heavily targeted because there are lots of users all going to a common domain.
It's the same reason users of major ISPs are more likely to be probed for vulerabilities.
I've found the method of filtering based on the "Click-Me" domains to be the most effective with virtually no false positives (zero is a realistic number).
I've found that setting up a secure public mail system is cake. Mercury Mail is free and handles well. A single check box set by default is all it takes to keep it from being an open relay. Students of the university could probably do rather well offering their own e-mail services to students. Mercury Mail's filtering system is quite robust.
MM supports IMAP/POP3/SMTP and alternate ports as well as SSL on all them. Adding a web-based front end also isn't that difficult if you know what you're doing. There's actually one built in and a more robust version coming.
I already have a few hundred users on Indie-Mail and the amount of bandwidth used per day is pretty negligable.
Ben
Work Safe Porn
Make your boss happy, and block on these three DNS based lists: dsbl.org, spamhaus.org, dnsbl.org. Everything coming from IP addresses in these range is basically garantueed not to contain false positives. It'll clear your inbox quite effectively. (I'm one of the volunteers helping out dsbl.org, so feel free to mail me with questions)
-John
If required they can also set a spam level on the mail server in a MySQL user/account database to automatically delete mail over the specified threshold (for accounts receiving oodles of obvious spam).
It has a nice balance between performance, security, and leaving most of the control in the hands of the users. We haven't faced extreme loads but it hasn't even raised an eyebrow over the load so far. Most importantly, no unhappy usres complaining of missing emails...
Q.
Insert Signature Here
The measured UBEs over a 3 moth period were 172,887 - only for their top-25 most spammed employees!
Notepad specialist & FAT administrator, group training available
How can you know you've had no false positives.
Have you personally reviewed the 2.9M messages which were filtered out... if you have then i'd question the value of your filtering.
I know i've occasionally had false positives and i get nowhere near your message volume. My personal favorite is the UK paypal-esque service NoChex which sends emails with the subject line "YOU'VE GOT CASH!!"...
I've seen this happen in my local University too.
Take a university that has thousands of people actively using email, and thousands of computers, probably a hundred of which function as mail server. Now, decide that "we need a central mail server to filter viruses and spam". Take a few useless machines lying in the computer center, and make them the mail server that's supposed to replace the hundred you had previously. Then slow down the new mail server by applying every concievable virus and spam filtering.
What do you get? Incredibly slow service (sometimes mails get stuck for hours or more in the queue), single point of failure, and officially-mandated false positives (noone in the university can avoid them). AND, you still get a lot of spam.
Computer centers must know that if they want to centralize a service that was previously decentralized (different departments and individual running their own mail servers and filters), they must be prepared. Prepared to handle the load (Google had to buy 100,000 machines to handle their load!), prepared to handle the humans who use their service, and prepared to handle exceptions (a person or department that doesn't want the centralized filtering). Often, these computer centers don't think of these issues in advance, causing things like described in this article.
Spamd and other means for "tarpitting" the calling SMTP are another great tool to be used in combination with RBLs and bayesian filters.
It's a strategy in layers:
To quote the post directly above you ...
No, sendmail (Score:5, Informative)
by marnanel (98063) on Monday May 24, @12:04PM (#9234290)
(http://marnanel.org/)
7: They're using MS Exchange SMTP servers, which bog down incredibly under load, especially if you run any separate service such as spam processing.
Nah, it's sendmail:
$ dig -t MX tu-bs.de
[...]
tu-bs.de. 172738 IN MX 10 rzcomm5.rz.tu-bs.de.
$ telnet rzcomm5.rz.tu-bs.de smtp
Trying 134.169.9.40...
Connected to rzcomm5.rz.tu-bs.de.
Escape character is '^]'.
220 rzcomm5.rz.tu-bs.de ESMTP Sendmail 8.11.1/8.11.1; Mon, 24 May 2004 04:00:51 +0200 (METDST)
Because once a solution becomes commercial, the spammers get hold of it and work out how to modify their spam so that it gets through.