Slashdot Mirror
University Capitulates, Switches Off Spam Filters
Heraklit writes "As reported on German news site Heise, the system administrators of the Technical University of Braunschweig have temporarily given up the fight against spam. Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!), they decided to switch off all filter mechanisms. Before, the 20 servers dedicated to processing e-mail alone had been breaking down under a load of 100000 unprocessed mail messages, ca. 98% of which had been spam or viruses. ... A similar e-mail jam occurred recently at the IT central of the German Federal Government.
Is this the beginning of the end of e-mail?" (The Fish may be useful.)
Does anybody know the filtering methods they were using before they decided to toss everything to wind?
As Seen On TV's? Come back!!!
Perhaps just disabling spam filters and leaving virus blocks in place would be a less drastic approach. Detecting spam is non-trivial, but detecting viruses is not. They are easily found and the email should be blocked. This is implemented by my ISP (Road Runner NYC). Emails containing viruses are replaced by a text message warning that a virus was sent to the email address.
...is to inform the students how to install their own software, like Spam Assassin. That would distribute the processing to the people who actually would use it.
Read journal when you are not understand
Is this the beginning of the end of e-mail?
I seriously doubt e-mail will ever die. It's FAR too convenient to just give up on. Even if it comes to the worst case scenario where you have to whitelist everyone who wants to send you e-mail, it'll never go away.
Personally, if it were my universtiry, I would prefer they started to use a RTBL. The fact of the matter is, if the likely spam isn't sorted out first, I have to try to discern the stuff entirely by hand. And although I can easily pick out Viagra ads, I have relatives and the occasional acquaintence who send mail that looks awfully like spam. Didn't want to type a subject. Used "hello" as the subject. Didn't configure their mail client properly, so their "replyto" looks crazy. Without some initialy spam filtering, I would miss at least some of these -- in fact, I'd probably miss more mail with no filtering than with a judicious blackhole in front of me.
Love or hate SPEWS and other kinder, gentler RTBLs, they're better than the present choice. It would certainly reduce the load of these email servers to where it could be more easily handled. And, if nothing else, they couldbe used to prioritize mail. Use Spam Assassin or something else to do some initial tag and filter so that mail coming from Asian IPs or originating from mail servers on cable/ADSL networks gets put into the "slow" processing queue while everything else gets sent down the faster pipe.
</spouting with little to no knowledge>
You like splinters in your crotch? -Jon Caldara
The students and other users of their mail system will just have to use their own spam filters now.
It's not the end of the world. There's a few good spam filters for outlook and outlook express, and some really awesome free ones for linux/unix.
- It's not the Macs I hate. It's Digg users. -
No, but its one more nail in the coffin..
Something has to be done soon or email just wont be practical to have. Between Spam and viruii its overloading a lot of comanines network feed and servers..
And don't forget the cost of having to maintain antispam and antiviral solutions..
I know personally where I'm at, we are hitting over 2/3 of all email is spam/virus. ( i hear we drop 10k a day from the black hole list alone )
At home its 98%...
---- Booth was a patriot ----
Are they providing free internet access? Or are the students paying for it, directly or indirectly? Because if they're paying for it, and legitimate mail gets lost due to the Universitys system, that's probably a basis for somebody to sue them. Failure to provide a service that was paid for. The Uni probably can't take the risk of legal action.
Just because you're paranoid doesn't mean there isn't an invisible demon about to eat your face
I don't think e-mail is dead, but e-mail as we know it, specificially the SMTP protocol, is long overdue for a retirement party.
Afterall, the "from" field is a total free-response section in SMTP with no need to authenticate that you're really associated with the address you claim to be. That and other weaknesses are why spam is so hard to kill in the first place.
We'd be in a much better place if our e-mail system at least had a trustworthy traceback facility so that we affirmatively know who sent the message by default.
We shut off our email filters too. No need for them now that we go through Postini (http://www.postini.com). They filter the spam before it hits your server, then give each user power to customize their filters and view caught messages.
If they only processed 100,000 messages, they would have processed them long ago and thrown out the mailservers. They would no longer be necessary, given that they'd processed all the mail they're ever going to get.
But they process 100,000 messages per hour. That's 24 times your 100,000 messages per day. Thus the need for over 20X the hardware you're using.
set up a SPAM filter; send all filtered SPAM to Senators/Representatives who voted *for* the CANSPAM bill.
Inform them that they can stop receiving your forwarded SPAM when they enact legislation which puts an effective stop to it.
It is a common misunderstanding. While most web server these spams are pointing to may be located overseas, most of spams are originated from US. Mostly likely from hijacked fast cable/DSL connected home machines.
You may think it is okay to block email from China or even the whole Asia because you don't know some Asians in person, but please check again where your RAM, mobo, anime etc come from... A lot of companies and university have collaborations overseas as well...
We don't really have much options left... Basically, you will have to blacklist all the high boardband provider's IP range (rr, earthlink etc)... Sorry, geeks, your email server will no longer work... It is not really an ideal solution. The other idea is kind of similar to secured DNS, ie, mail server retrieves "good IPs" from a central server. Email originated elsewhere are assigned with very low priority or filtered out altogether.
Everyone needs to be registered with their mail server with the governing body (similar to the domain name idea), say for $100 per IP. It is not that expensive if you really need that... But, prohibitive for spammer... Yes, it makes home run email server more expensive... But, you cannot get a domain name for free anyway. Why should we expect email server to be free? It may be the solution to get the economy of spamming right again.
One problem is: who will make this specification? MS? They certainly want to.
Once this new email2 protocol is invented, how long would it take to be implemented around the world by every admin?
What happens when that protocol gets hacked (probably by the spammers)?
I think its the right direction to make an email2 protocol but it wont be easy.
Can your karma go above being Excellent?
Even if the spammers band together and make a big organziation to self organize and police, spammers by almost by definition dishonest (no honor among theives!), and as soon as one realizes that he can make more money by ignoring the organzation (i.e. almost immediately), he will.
98% spam and virus's? Damn. Think that the mail is coming from campus.
Outside world:
Block direct contact to the mail servers, use an upstream MX record.
Inside world:
authenticated SMTP.
What's OK for you may not be OK for other people. Personally, I get about 200 spams a day, versus about 1-2 real e-mails. When the ratio of spam to good mail is 100:1, it gets hard to implement spam filtering that's accurate enough to do the job. And are you under the illusion that you aren't paying your ISP for the bandwidth they waste dealing with spam?
There are some basic problems here:
Find free books.
As someone who lives in China I get more than a little tired of being filtered out because of the continent I live in. (Especially since the vast majority of spam I get is selling products from America, regardless of what server they're sending them through.) And in this particular case, being a university it's very likely that they have a sizeable number of students from China, and many staff with academic links.
er, let's see ... 10,000 messages per hour, across 20 boxes ... that's what, 500 messages per hour, per box? I'd think pretty much *any* computer worthy of the name could swing that.
I would say this is probably not the end of email, nor is it the end of the Internet as a whole. However, it is probably the end of the protocols currently used to send and receive email.
I believe that spam is ultimately a security issue, because it slows down systems and creates problems for users and system administrators. Sometimes, security problems are caused by buffer overruns and other programming errors. However, in this case, I think the entire protocol is faulty. It may have worked wonderfully before spammers, but it's time to introduce something new that will make it extremely difficult to send spam.
I don't know exactly how the new protocol needs to look. But I have some ideas. Paying for "postage" is not one of them, as I think it is a very bad idea. Unless some payment system could be set up whereby the recipient of the mail receives the payment, not some 3rd party, like Microsoft, which would profit incredibly from garbage spam mails going all over the place. In fact, if that were the setup, then each recipient could state a price per email and/or per kilobyte of the mail message for receiving an email from a source, which the source would pay to the recipient as postage. A whitelist could be set up to allow certain senders, like one's friends, family, coworkers, etc., to send emails without paying the recipient. A blacklist could be set up to disallow all emails from specific senders and/or domains, as we have today, and if you read further in this post, you'll see my ideas for making sure that addresses are not spoofed. But I digress...
Perhaps first of all, the mail headers need to include digital signatures based on the source and destination domain names, email addresses, and other identifying information that is unique to each email sent. To avoid address spoofing, for example, people sending junk with a 'yahoo' or 'hotmail' address, when in fact it originates elsewhere, each such domain would have a private key, which upon sending, would be used in the computation. A valid signature could not be computed when the address is spoofed, and so all spammers would need to use their own valid domain name. Further, the need to make computations would make it more costly for spammers to send mail in high volumes. The algorithm should be designed so that recipients of email will have a much lower cost to verify the key. Further, the signature system could, should, and would be used to verify that each bit of the contents of the email, including all attachments, arrived correctly and without being tampered with or corrupted in transit.
The whole thing about them being legally obligated to deliver mail is the silliest thing I've ever heard. Leave it to the Germans to enact such a law.
Better to just not deliver ANY mail than to deal with that requirement.
the numbers dont add up, Loads of people have already raised the issue about the fact that 20 servers ( even decently mid spec single CPU machines ) will handle 100k emails an hour ( about 80 emails per min per machine is very achieveable ... ).
But there are some other issues you need to look at, with these emails not being scanned - do you know how much of storage you need to have online to have a mailstore this size and developing by the hour at 100k msgs ? not everyonce will use pop3 to get their emails, and not all the users will check email every day. Were talking about a very very large and very well setup Mail Store for this kind of volume. What about network bandwidth ?
A few basic things can reduce the work of those servers : Duplicacy level across these emails is going to be very high - all 100k emails per hour cannot be unique, there are going to be loads and loads of dupes, that dont even need to be scanned.
Creating a small database in-house with bad MailSender's list ( kind of like an in house RBL ), and flushing that list on an 6 hour interval will slow the inflow as well to quite an extent - in some tests done, i have seen it go down by almost 15 - 18% when there is a heavy load. Since most 'real' mailservers tend to retry, even if a genuine mailserver is blacklisted for 6 hours - it wont make much of a difference, however most 'hijacked PC's sending spam' dont have any retry or resending mechanism - and will just not be able to send into your server.
Another issue that helps stem the tide of bad email is to check for Virus infections before checking for spam. A lot of cases the tides of mail coming in can be virus infections ( which are easier and faster to check against - compared to rules + logic based spam checkers ).
However, all this is said and done without knowing of what system and what kind of a setup they use, there is no way anyone can really know what happened and why.
In the end, classic case for Linux and Unix based technologies to come into the frame I think.
I agree that SMTP needs a makeover, but what to replace it with is still very much an open question.
Mail? Put "slashdot" in the subject to pass the spam filters.
As someone who lives in China I get more than a little tired of being filtered out because of the continent I live in.
Then bitch at the Chinese ISPs who allowed the problem to exist in the first place.
STOP MISUSING APOSTROPHES, YOU MORONS!!!
That may not actually be the server handling the mail though. It's rather common to have a sendmail/postfix mail forwarder on the outside that forwards all mail to/from the Exchange server on the inside.
not to exploit the 'imagine a beowulf...' cliche.. but how would clustering email servers help this? has it already been tried?
So requiring multi-case is useless.If you make your users change their email addresses every couple of weeks, then I wouldn't want to be one of your users.
Imagine if everybody did this.
The number of SPAM messages would quickly be swamped by the number of change-of-address messages.
I'm certainly not going to take the time to update my address book every couple of weeks from someone who changes his/her email address that often.
That means that your users are SOL if they want me to stay in touch with them.
There is such a thing as going overboard.
You are going overboard.
Those who sacrifice security to condemn liberty deserve to repeat history or something. - Benjamin Santayana
Does Germany have a law that I'm not familiar with? Email is free not a paid service, why is there some obligation to deliver? Snail mail is normally Govt. run and delivery is what you pay for with a stamp.
No one has to or could guarantee anything for email. With the amount flowing because of SPAM the dropped packets must be astronomical.
Professional Politicians are not the solution, they ARE the problem.
No, you should be using a MAIL SERVER that has an A record anyways. It's not that you can't send mail, you just can't run the SMTP server on a machine that you can't do a reverse-lookup on.
----- Question authority, but not ours. Hate the man, but we're not him.
When my spam mailbox is full of things offering me credit cards, mortgages and such that are only available or sellable in the US. Same for most of the viagra and diet pills, if I follow the links I usually end up at an American company. A small percentage aren't, of course, mainly Nigerian scams and some local stuff, but 95% is.
This isn't just my opinion. See this in The Guardian: "There are really only 150 spammers doing 90% of all the spam we get in the US and Europe... at least 40 of them are in Boca Raton."
Your faith is touching. Was it Nixon who started the first "war on drugs"? How's that going?
How can you know you've had no false positives.
Have you personally reviewed the 2.9M messages which were filtered out... if you have then i'd question the value of your filtering.
I know i've occasionally had false positives and i get nowhere near your message volume. My personal favorite is the UK paypal-esque service NoChex which sends emails with the subject line "YOU'VE GOT CASH!!"...
I've seen this happen in my local University too.
Take a university that has thousands of people actively using email, and thousands of computers, probably a hundred of which function as mail server. Now, decide that "we need a central mail server to filter viruses and spam". Take a few useless machines lying in the computer center, and make them the mail server that's supposed to replace the hundred you had previously. Then slow down the new mail server by applying every concievable virus and spam filtering.
What do you get? Incredibly slow service (sometimes mails get stuck for hours or more in the queue), single point of failure, and officially-mandated false positives (noone in the university can avoid them). AND, you still get a lot of spam.
Computer centers must know that if they want to centralize a service that was previously decentralized (different departments and individual running their own mail servers and filters), they must be prepared. Prepared to handle the load (Google had to buy 100,000 machines to handle their load!), prepared to handle the humans who use their service, and prepared to handle exceptions (a person or department that doesn't want the centralized filtering). Often, these computer centers don't think of these issues in advance, causing things like described in this article.
Spamd and other means for "tarpitting" the calling SMTP are another great tool to be used in combination with RBLs and bayesian filters.
It's a strategy in layers:
Because once a solution becomes commercial, the spammers get hold of it and work out how to modify their spam so that it gets through.
OK, some stats:
My company receives about 3,000 e-mails per week, of which 2,600 or so are junk.
I recently installed a simple bayesian junk filter + whitelist on this, and it is catching about 2,500 of those 2,600 junk messages. Last week there were two false positives; the week before there were none. 99% of the false positives have come through mailing lists that add loads of shite to the bottom about how to unsubscribe. In the 2 months we've been using this filter, we have not had a single business-critical message filtered.
Previously we used a spam-assasin style points system, which I would spend about an hour a week fine tuning. We were letting through about twice as much junk, filtering about 5 times as many legitimate messages.
The message - try a bayesian filter (yes I know s.a. has a bayesian filter built in now, but IMO the other stuff it does just confuses the issue). Set up an IMAP folder for everyone to dump the junk that they receive into, one to put their false positives into, and one for their filtered messages to be delivered to. Instruct them clearly about what to do with them. Re-train every other week. You'll get much better results than you're getting now, by the sounds of it.
Why not sign email, at the mail routers and gateways.
Email from large organizations could then be given priority (you'd know who it was by the signiture).
If an organizations starts spamming remove there signiture from the trust list.
thank God the internet isn't a human right.
Strip all attachments?!? You're kidding, right? In a university or business setting, that is NOT a viable option for most people. They're still figuring out how to right-click under Windows and make things print correctly to the printer down the hall; forget teaching all of them how about FTP, SMB, NFS or some other file serving method.
And you've got to be kidding about blacklists being better than filters... talk about false positives, sheesh! Maybe the best blacklists are better than the worst filters, but that doesn't say much. Simple control lists (black or white) are not a long-term viable solution; if they were, none of us would ever get spam, would we? You really need something that makes your email trustworthy, like Zoemail.
$nice = $webHosting + $domainNames + $sslCerts