Slashdot Mirror
University Capitulates, Switches Off Spam Filters
Heraklit writes "As reported on German news site Heise, the system administrators of the Technical University of Braunschweig have temporarily given up the fight against spam. Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!), they decided to switch off all filter mechanisms. Before, the 20 servers dedicated to processing e-mail alone had been breaking down under a load of 100000 unprocessed mail messages, ca. 98% of which had been spam or viruses. ... A similar e-mail jam occurred recently at the IT central of the German Federal Government.
Is this the beginning of the end of e-mail?" (The Fish may be useful.)
Just white list known good addresses. Hand out auth tokens (X-Not-Spam: md5 digest here) and white list those temporarily. And white list known good PGP keys.
Byebye, spam.
Byebye, email.
If you don't filter out any of the spam, then some mail server somewhere is gonna have to store all that junk mail. Even with a quota system I'd expect that there'd be a whole bunch of people just "giving up" on e-mail.
> Is this the end of email?
:), so I think that responibility is the key to keeping email working. Adding some numbers (*sigh*) helps guard against random address guessing.
Yes. When one university decides to stop filtering SPAM the entire world's infrastructure has effectively been shut down. Oh wait... no.
My UIC account gets NO spam (because I don't give it to anyone
Anyway I don't see anyone stopping you from using your own SPAM filter. Let's not blow this out of proportion, please.
My other car is first.
Maybe they should just blacklist the most common spam and virus adresses by subnet then filter on a lower percentage. It would seem that if they got rid of china or some other area like what happend recently with spain, it would send a message to those networks to stop things and bear some of the weight.
Excuse me? One university gives up on spam filtering for questionable reasons and you declare death of email? Weird, I still do most of my communication via email. My servers all run spam marking services and my client filters out the junk as soon as it's retrieved.
:)
Of course more bandwidth is wasted on spam mails, but since I don't see much of it, it doesn't bother me so much.
What do you propose to use instead of email? instant messaging? Talk about waste of time
-- shortcut - the longest distance between two points.
Here at the university where I am a sysadmin, we get approx. 100K emails per day and we have no problem pushing them through spamassassin on a single server with dual 2.8 xeon processors. How in the world could this place possibly need 20 servers to process this much mail?!
Good question. I would think that 100,000 emails is really not a lot, even for 20 low-end PCs.
LOAD "SIG",8,1
"Because of the legal obligation to deliver all mail and of the delay time exceeding critical 5 days(!)"
Is it just me or is this another ridiculous law? The University is providing free email services to those that are students at this establishment and they obviously need to filter out spam in order to be able to offer this service with there current hardware requirements. Spam is a legitimate problem and people that are offering free email services should be allowed to attempt to filter it as it can be extremely taxing on a busy mailserver. They can filter the spam without being intrusive or breaking privacy laws so I see no reason that it should be prevented by law.
This kind of shutdown is only going to encourage spammers to send out even more junk mails.
Now they know that most servers will eventually not able to cope up with the traffic, they might as well send out randomly-generated '@domain.com' spams until the admin gave in.
Rock that crushes, Paper & Scissors that don't matter.
One would think that even spammers would realize that if things go too far, businesses might not carry emails at all anymore.
I mean, even parasites usually try to not kill the host.
*sigh*
Failure is not an option. It comes bundled with your Microsoft product. -- Ferenc Mantfeld
I run Exim with an ACL extension called Exiscan, which runs SpamAssassin and virus checker during the SMTP dialogue.
Rejected mails thus don't generate any undeliverable bounce messages to fill up the local mail queue, and the sender gets an immediate response.
What about some kind of seti@home like distributed filtering system on campus? There'd be privacy/security issues I guess, but with masking the recpients address, a whitelist system to bypass the filter, encryption, a well designed client,etc intercepting other people's mail might become hard enough to deter all but the most determined which would be fine by me if my inbox was clean -- let's face it email isn't really all that private to begin with.
"... drowning in information,
And that warning is so useful. Who do you send it to?
These messages are a waste of everyone's time. I get hundreds of worms daily...but I never see them, because they're easy to filter. What I do see are these damned "helpful" messages that "I" sent someone a virus. Those are much harder to filter.
Much better way: reject viruses in the SMTP transaction. The SMTP client is then responsible for notifying the sender. If that client is a virus or worm, it will do nothing; no one is bothered. If it's a false positive, the sender will get the bounce. Reliable, unobstrusive.
If you want to filter email politely, you must follow these rules. People who don't cause the rest of us constant headaches. The worst thing is that they don't even realize it.
Maybe there should be an n-monthly day on which spam-filters are disabled. That way the public may realize the extent of the spam problem. Can you expect that they know it when they only get a few spams because all the rest is blocked at the server?
Anyone know the specifications of the 20 servers they were using? 100,000 messages isn't that much. Five day delays? Did I read that right?
I go to Penn State, but since the university feels it has to protect dumb windows users from themselves, I cannot even send or recieve email with the subject, "Hi such-and-such"(Try explaining to a friend overseas who has almost never in her life touched a computer, in her language, why she can't send you mail with that subject) because it might contain the bagle virus. This is the same university that put in a firewall because supposedly too many people on campus had a butt-load of viruses and spyware.
Yet this same university loves to publish my email address on the web; ensuring I get tons of spam(some even in Chinese!)
I hate when the community at large has to pay for the transgressions of a few slimeballs and the idiocy of some(not even most) gullible windows users.
In addition, I have two accounts that I use regularly -- one that I give to everyone (web registration forms, etc) where I don't care about spam, and another one that is personal and I only give to close friends. Guess which one gets more spam? That's right. My personal account gets about 150 per day. My "don't care" account gets like 6 per day. They have both been active for many years.
...just my 2 gil.
Oh common, the 100s of daily "message has virus" emails I get are very useful. It makes me keep my Gentoo box win32 virus free!
I once confronted a sysop about this and they told me "if we don't email them back people won't know the message was rejected". Apparently the idea of checking while reading the message never crossed his mind.
As another poster suggested I just filter out all "warning" emails as junk which helps.
Tom
Someday, I'll have a real sig.
No one (sane) *manually* checks for false positives, just the end user. You do need manual personnel to follow up on end user inquiries, but it should be moot. If you have the right spamblocking service/setup, you're not going to get false positives...
There is no America. There is no democracy. There is only IBM and AT&T and DuPont, Dow, General Electric, and Exxon
Joking aside, it boils down to economics. Spam is profitable. If something is profitable, people will do it. Selling drugs is profitable, and the war on some drugs hasn't changed that. The answer to spam (and drugs) is not to try and stop them, but to make doing them unprofitable.
What makes spam profitable is the presence of people on the internet who are SO incredibly stupid that they fall for it. (See Junkie loves his spam) Remove them, and you shoot spam through it's purtid heart. I can think of several methods of doing do:
- If you respond to spam, you've probably got shitloads of viruses on your computer. Beyond any shadow of a doubt some of them are spamming people. If you ISP detects lots of mystery traffic from you on known virus ports, you're given one warning. Then you get kicked off without ceremony and not allowed to reconnect until you can prove to them that all computers using your connection are malware-free. No more malware, no more spam zombies.
- (You, the ISP) Send test-spams. Specify in the header that it is NOT a real spam so you don't get blacklisted. Anyone who responds to them loses all services except port 80 until you prove to a professional who visits your house that you know enough not to buy from spam. Do it again and you will never be allowed to use your ISP's mail servers again.
Neither of these can possibly be routed around or hacked by spammers, because they are not involved in any part of the process. If you are not in the habit of perpetuating malware with your computer, you needn't worry of getting caught up in it all. Neither of these requires a major invasion of your privacyspamd is a new approach to blocking spam. Its called greylisting. It rejects all email with a temporary failure notice in the hopes that the large volume spam senders don't have the resources to wait 30mins and send the same email again. Apperently this method works quite well and uses little resources.
The MTA's work is relatively light compared to what anti-spam software must do. This is especially true of SpamAssassin. While it does have some advantages over its competition, SpamAssassin is extremely resource intensive. Firstly, SpamAssassin is not written in fast C/C++ but Perl. Every email is sent through zillions of Perl regex rules. Then there is the Perl implementation of the Bayesian test, which really bogs down when an email auto-learned. Then there are the various (optional) network lookup tests: several RBLs, Pyzor/Razor/DCC ... each email can eat up a lot of resources even if you bypass the startup overhead by running spamd.
I have also seen situations where SpamAssassin was not correctly respecting the maximum child spawn limit. Since spamd is a fairly heavyweight process, the server started swapping and throughput plunged.
Such heavy overhead is not a essential part of anti-spam software. Something NOT written in Perl nor any "interpreted" language, something with a smaller footprint, will be much, much faster. I wonder how many people have switched to dspam for this reason?
OK, you could tell the end users to find their own tools and just cope.
However, I work in a large organisation, and with a 98% spam ratio, the mail infrastructre would need to be much larger (and more expensive!) than it actually neeeds to be. Let alone the (*&&^$@# junk traffic and bounces caused by auto-responses to forged addresses. Plus we have a significant number of staff who are clueless who would be excluded from communicating effectively because they do not have the time or skills to learn how to train a spam filter. in such a situation, no-one could no-longer *rely* on email to contact/inform our staff, reducing its value as a tool.
Our email infrastructure already groans under the load each time another Outlook virus arrives.
The hay-stack of spam is probably just as disruptive as false scanner positives.
Xix.
"Everything is adjustable, provided you have the right tools"
Being the person that blocks spam is a lose/lose situation. They don't understand how bad the problem is when you do your job right. They complain when spam gets through and complain when legit email gets blocked, but don't want you wasting all your time on it.
I predict that this school will be forced back into filtering spam by their students (customers).
[rant]See, 3 years ago, as spam was beginning to get bad, I began filtering spam on the email system I manage. Over 2.5 years, I developed a rather intensive filter, but since I knew I was not perfect, I had to scan blocked email for false positives. It got to the point I was spending 25% of my time scanning for false positives and the boss didn't like that. He also didn't want me to spend time trying to figure out how to set up Spam Assassin. (I'm not a Linux guru. Sorry!) The board didn't want to spend the money on a purchased system and didn't want me wasting my time with spam. They didn't think it was a problem so they told me to just stop blocking spam. My boss told them that spam was a BIG problem, but they never saw it so they didn't believe him. I asked my boss 10X "Are you sure you want me to stop blocking spam? They won't like the results." He confirmed. I stopped blocking spam and about 50,000 additional spams per week came flooding into the system. The 50,000 were what was being blocked previously. I was flooded with phone calls until everyone realized what was happening. Then, just 2 weeks ago, I was instructed by the board to go back to my filtering, but only spend 30 minutes a day on it. RIIIIGGGHHHHHTTTTT! Ever try scanning for legit email among the trash, adjusting filters to make it better and taking calls and emails from people that want you to be sure an email is blocked and only spend 30 minutes a day on it? I managed to put together a Spam Assassin box and it blocks 10,000 per week, but there's a lot that doesn't get blocked. I don't know enough about it to make it better.[/rant]
But why is the rum gone?
Dumb question, but someone mentioned the odd spellings these days... ba|\|a|\|a = banana. How many people spell that way? Why not tell the spam filter that more than one word using ^[A-Z] (for English language) has an increased likelihood of being spam?
"Sometimes a woman is a kind of religion, she can save your soul & set you free from all your sins" - Bad Examples
This shouldn't be a problem for mildly capable admins. Our company uses Sun's Sun ONE Messaging Server (aka Java Enterprise Systems Messaging Server, formerly iPlanet Messaging Server, formerly Netscape Messaging Server, formerly Netscape Mail Server) and we process three million messages a day with an almost empty mail queue at all times. Even a normal home PC should be able to process hundreds of thousands of messages in a day without much strain.
Their hardware is severely underpowered (thinking IO bottlekneck here) or their software is poorly tuned or they are using ancient hardware.
How they think refusing to process spam is going to help I have no idea. That's only going to increase the load. A decent system could filter out much of the spam at the SMTP level without incurring the extra IO of writing to disk and processing the message all the way.
If I understand this spam problem correctly, why do they (ISPs) filter incoming mail when logically they should be filtering outgoing mail? The way I see it is spammers hit unsuspecting network vendors (Chinese, Brazilian, Korean etc.) who are all to glad to have netted a hefty account until a week or two later they find themselves blacklisted all over the world. The damage is done and the spammer has already moved on to another ISP.
If email were channelled, filtered, throttled and who knows what else on its way out instead of in, spammers would be discouraged or at least slowed down to a snail's pace. A trustworthy registry of ISPs using this technique could be created and providers could choose to receive mail from this list only. Spam has become a world wide plague and requires a global effort. Does this make any sense, anyone?
Most viruses have a text line that start out: .exe. Simple solution is to hunt for that tag when the message comes in and kill any message that has it. Should you have a real person sending an exe attachment, they will get the bounce if you reject it while the SMTP connection is still active and there is no siletnly lost real mail. A patch for sendmail can cope with a few hundred thousand messages an hour on pc class servers so its no big deal but I've got a faster hack when it matters.
TVqQAAMAAAAEAAA
since they are mime encoded
Surely it is Bayesian classification which brings my Athlon 1700 to a screeching halt (spam assassin) (takes about 1 second to classify an e-mail). There are FPGA and DSP based Bayesian classification systems, they should really look into them.
Religion is a gateway psychosis. -- Dave Foley
If you really want to make spam unprofitable, don't prevent people from clicking on the links. Instead, make everyone do it.
Perhaps they're running Exchange or something over there. I would think that 100,000 emails, distributed to 20 different machines, would amount to 5,000 emails apiece. How could this possibly be a substantial increase in load over a period of _five days_ for just about anything?
... but Germany is a pretty modern place, and my desktop machine (an AMD 1800+ system, which cost about $600 total in parts when I first built it) could handle the kind of load their servers should be seeing. They should be able to get access to enough power to deal with the mail load they're getting, or they're running on ten year old machines and are refusing to spend a cent on upgrading.
Going a bit further, it would mean these machines were blowing at least 17 real seconds processing each email, if each machine were performing only one scan at a time. Whose filter takes 17 seconds to process and scan one mail? I know that DNS lookups can take time, but doing regexps on text documents certainly doesn't take much by comparison.
Now, it's possible that they only have three or four of these 20 machines acting as the front line MX servers for the system, with most of the others just storing subsets of the mail, and running POP/IMAP to deliver to users. It would mean five times the load would be on these machines, but geez... 5 x piddly is still near piddly.
I've worked for a company dealt with way, way more than 10,000 mails per hour (a tenth of what the uni is taking more than five days to handle) and delivery time for that network was under 30 seconds--with only four servers acting for both incoming and outgoing mail.
It sounds like they simply must have done something critically wrong somewhere in their network design that's acting as a severe bottleneck, or they are using REALLY old/slow machines to toss mail.
I can only come to the conclusion that they're just flat out refusing to spend money to upgrade the equipment because the thing that's overloading what they have is spam. The only way that will ever work for them is if spam mysteriously disappears entirely from their network--which isn't bloody likely to happen in the real world anytime soon.
There are graphics format exploits, including a
recent one for Internet Explorer using BMP files.
Lossless graphics re-compression is dirt cheap
compared to SpamAssasin, Bayesian filters, and
Apple's word vector thing.
Rule: Do the cheap and obvious filtering, plus
the filtering needed to protect Microsoft junk.
I get legit email with HTML tags and even images.
Often this lets me know the sender has no taste,
but sometimes it is justified. You used an italic
font to quote me; that doesn't work in plain text.
Bold, underlining, and fixed-width characters can
all be justified. The sort of formatting you'd see
in a man page is perfectly legit.
If required they can also set a spam level on the mail server in a MySQL user/account database to automatically delete mail over the specified threshold (for accounts receiving oodles of obvious spam).
It has a nice balance between performance, security, and leaving most of the control in the hands of the users. We haven't faced extreme loads but it hasn't even raised an eyebrow over the load so far. Most importantly, no unhappy usres complaining of missing emails...
Q.
Insert Signature Here
They could easly educate their students to use a throwaway yahoomail or gmail or even better the awesome spam.la service when they sign up for pr0n services or NYTimes spam lists.
Maybe you are living in a country where privacy laws are no longer enacted, but I prefer to have rather strict privacy laws over having someone spy on me.
There are simple solutions that allow to abide to the law while still providing Spam filtering. We add appropriate headers to Spam and Virus Mails and deliver them to certain subfolders of a users mailbox. He/She can then decide to delete the mails. Users who would click on attachments are also not capable of using IMAP instead of POP and thus won't get access to the messages.
Sebastian
The sysop of a local BBS grew his own spam filter, based on all sorts of header criteria, as observed in actual spam. It fails very rarely, maybe once or twice a year (either a false positive or a spam let through) and yes, he DOES hand-vet the results (did so every day for months, until he was absolutely sure it could be trusted, and still checks it on a regular basis).
Anyway, if an amateur could do that well, I'm sure close enough to 100% accuracy *can* be achieved by a professional solution. In fact, it's made me wonder why some solutions don't perform better than they do.
~REZ~ #43301. Who'd fake being me anyway?
It doesn't say breaking down after 100k emails a day. Everyone here knows most mail servers can do that on junk hardware in a day(yes, Even exchange can do it).
It sounds more like they are having problems when they start reaching 100k messages in the queue. Anyone who's dealt with tracking a large number of small files across a file system knows that there can be slow downs (not that there aren't solutions to those, but they may not have been able to spend the time to address the problem since they've been 'fighting fires'). When my incoming postfix/amavis/spamassassin systems get 100k or so mails in their queues on ext3 file systems, they start behaving badly too. We addressed the 'fire' problem by throwing more front end servers at it while we take time to rethink our file systems where the queues reside. We'll get the the luxury of a few weeks to address it with other hardware before we start getting unacceptable delivery delays again (for us, thats
Universities don't always have the money to throw hardware at a problem like this or are willing to give their often student supported IT administration the benefit of the doubt that 'we need $20k (euros, lira, beads, whatever) to buy some hardware to roll a better solution'.
Yes, I'd be surprised too if they mean '100k emails a day and we bog down' . . . I just seriously doubt that is what they mean. Maybe they are stopping their spam/virus processing just to clear their backlog. Maybe its not that they aren't receiving it & spam processing it fast enough; maybe its their backend server that is taking it all in just can't keep up. I mean, if they've got 20 spam/virus receivers that are getting the job done and trying to hand off to one fat exchange box that isn't keeping up then their queues are going to grow on those front ends and eventually kill them which makes it look like their spam/virus scanners are causing the delays.
Then again, they could be a bunch of retards and everyone is right that they don't know how to run even a low volume mail server . . . but somehow I doubt it . . .
I work at a UK university and we're introducing a new system to deal with spam. We've already got an in-house product, MailScanner which does the detection job pretty well, but our mail servers are quite loaded with junk.
We're about to offer a "delete at gateway" option, so our users don't have to filter their email and lessen the load on the mail servers at the same time. This service is optional, so our users can choose whether they want it, but we'll be strongly encouraging them to use it.
Additionally, they can set their spam threshold, so they can delete most spam, but review the borderline cases.
perl -e 'print "Just another Perl newbie\n";'
I don't suffer from spam as much as I do from emails bouncing back to my inbox from the sender saying "YOU'VE GOT A VIRUS!!!!!"
After checking headers, none of these have come from my server but they have my name and email address as the sender. It pisses me off no end when I get near enough a thousand of these a day when none of them are from me.
That's the reason so much useless traffic is on the net - bounced email reports pinging backwards and forwards and backwards and forwards and, well, you get the idea.
Would it really kill this software to check to make sure that the sender's domain and reported email address match?
PocketGamer.org - For the gamer on the go!
Why not, do tell.
Lets say that there are a few thousand trusted parties,(shouldn't be too hard to set up).
They are the top email servers (apart from spammers).
Any mail from the servers gets priority delivery. (you know that it's really is from the servers because they've signed the message).
Everything else (sorry all you who run sendmail/postfix at home), gets slow tracked, along with the spam.
If a trusted sender is found to be sending span there trust certificate is removed and they get slow tracked.
Known spammers could be put in the even slower mail delivery pool.
thank God the internet isn't a human right.
I would question quietly deleting such mails. Most of the worm/virus ridden mails that I get come from people who have infected systems and where I am in their address book. They need to know they have an infected system.
I quarantine all the worms/viruses sent to my system. I look through the quarantine directory about once a week. On ONE occacion (out of a few hundred virus laden messages) I was able to determine who was sending the virus. The vast majority of the time the viruses don't leak any information about the system, and they come from dynamic IP addreses. Delivering the virus, or a "user X sent you a virus" message to the user is useless. I've never once had a false positive (and I believe the chance of false positives is about zero).
Delivering the virus laden email is just stupid. The reasons deleting it, or quarantining it far outweigh the reasons for delivering it. I'm pretty good about being able to track where a virus came from and I was only able to track down one virus origin. End users are going to have zero ability, and zero interest in doing do. They'll actually send out false "you've got a virus" reports to their friends (who don't actually have a virus, the from address was just forged).
AccountKiller
My sister-in-law is now receiving over 2400 spam a day, and no longer even has the time to scan for false positives. For folks like her, email is definitely broken. She has no choice but to rely on spam filtering to make the right choice, even though a false positive could cost her small business a serious amount of revenue. Even the local processing on her PC to sort/filter the emails is keeping her machine busy.
For many of us, the problem isn't that bad. But we need to recognize that many others are dealing with an onerous problem.
Can You Say Linux? I Knew That You Could.