Slashdot Mirror
Revolutionary Spam Firewall Developed
psy writes "physorg has a story on a new spam firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
I have a simple algorithm to reject spam: spelling.
If you can't spell correctly, then I don't want your v1agr4.
Life is the leading cause of death in America.
filter out mesages from my x ;-)
another Roadkill on the Information Superhighway
I think Barracuda Networks would rather disagree with the idea that this is the "only true spam firewall in existence," considering that Barracuda's entire product line consists of spam firewalls.
Damn fine spam firewalls, too, I might add. They handle around 115 messages per second, and can run up to eight filtering steps (including Bayesian analysis, which is similarly efficient to SVM, which the one in the article uses). Plus Barracuda's can do virus scanning.
I'm not sure how this is revolutionary.
Sourceode would be nice....
What the hell is one of these? There seems no substance to this report, bar some TLAs as above and a load of hype. Where is the proof? How was it tested? Etc.
Although this is a great new technology, for a business setting, I don't know if even missing one e-mail is acceptable...
The words revolutionary and spam in the same phrase... frightens me.
This isn't a firewall as it doesn't filter based on addressing. Furthermore, the use of SVMs (support vector machines) to classify text is not new...
Fetchmail + SpamAssassin?
What am I missing here?
Doesn't save B/W: you need to run in INSIDE your network.
Don't care how fast it is: It's a dedicated server.
1/25,000 failure rate with no false positives: OK, that's good. But still not amazing.
How are their servers?
I would rather be ashes than dust!
Well, this certainly sounds like a good thing for many people, but because it's been described as "firewall" and not a "server-side filter", I certainly hope it wouldn't be set up at major ISPs to intercept all smtp traffic going through.
Hopefully their spam firewall is more robust than their web server.
It's easy to produce these kind of results in trials - you just tune the spam filter to handle a certain set of emails, then you feed it those emails again and you get a near 100% success rate.
Heck, why not do it with a million emails? Makes better headlines that way.
I don't see how this is any different to SpamAssassin (the term 'Mail Firewall' is pure marketing bullshit. It's a spam filter. Get over it.) except I bet it costs a hell of a lot more...
1 out of 25k is impressive, but what happens to these spam mails? Are they bounced back as an error "no user account found"? Or done like a blackhole where the spammer doesnt know if it reeached its intended recipiant? I like my SpamBayes :)
I know! Ciphertrust's Ironmail works the same way... It stops ALL mail inbound, runs it through about a dozen different detection queues, only letting legitimate stuff through. I'd really like to see how this new one is otherwise unique.
Ed R.Zahurak
You know, oblivion keeps looking better every day.
Until there is a 0% fail misclassification rate such a method is useless. Filtering was one thing, if you misfiltered a message you always had the oppertunity of occasionally scanning your SPAM box and making sure everything was about penis enlargement and not about the meeting you have next week. However, with this method email is stopped and never delivered, thus your misclassified email is now gone- forever.
I'd rather get 5 extra spam if it meant I also recieved every real email.
transmission_err
"...companies losing valuable employee time to deleting spam..."
/.ed, here:3
Maybe they should be working on a Slashdot-Firewall. Damn, I really should get back to work.
Oh, and since the linked article got
http://www.uq.edu.au/news/index.phtml?article=583
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
Heh. Does anyone else see that as a good way to downplay false positives?
"Oh, good point, Computer. That email from my boss actually was spam. I didn't realize that until you mentioned it."
Lack of eloquence does not denote lack of intelligence, though they often coincide.
I submitted this as an ask slashdot and was promptly rejected, so I'm going to put this here as a slightly on-topic post.
What I want to see is a software hard drive "firewall." If you're not sure what I mean, think of what a product like zone alarm does when spyware.exe tries to access the internet on your pc. It pops up a window saying "do you want to allow this program..." Now, why can't we have the same thing for hard drive access? So, I download fungame.exe, and when I go to run it, my "firewall" tells me fungame.exe is trying to write to fifteen different directories to install different spyware products. It could only give a popup on the first time a program tries to write to a given directory, and have an option to not show any new notices for this program, to limit the annoyance factor. I think this would be a great tool to help lessen spyware/trojan problems. If the program interacted with spybot or a similar product, it could even automatically prevent writing of files that are known to be adware. Is there anything like this out there? Anyone who would be willing to help make it?
Unconsciously Desired Email Industry (Our slogan: You opted in in your heart!), I'd like to strongly protest the continuing escalation of technology against us. We provide the opportunity for hundreds of thousands of people to spend freely on products unburdened by simple heuristics of "they work" or "they won't make you ill" or "we'll actually send them". Why are you so intent on interfering with the consumer ethos?
You mean it blocks all email, and the one ligitimate email among the 25000 is the "misclassed" one...
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
The idea is that the mail server keeps a whitelist of "allowed" addresses which are always accepted. If a mail comes from an address which is not known, the mail server will reply with a "server unavailable, try later" error message. All real mail servers will try to send the message a little later (I don't know the exact time, but it's probably less than an hour. Someone else might know better).
The second time the remote mail server tries to connect, the server accepts the mail and adds the address to the whitelist.
However, mass mailers for spam don't do this but simply go on to the next address in the list if this happens. This way the spam message is filtered out.
Note that this method doesn't require any analysis of the actual content of the messgae, nor does it involve any manual actions from neither the sender nor the receiever. Currently it's porbably the best spam blocking method that exists.
I'd guess that if you put the firewall up against your average email user, the average user would shitcan legitimate messages at a much higher rate than the firewall thanks to the fact that the user can get frustrated while the firewall can't. I know my boss accidentally deletes mail from me at least 3 times per week because he's careless while mass-deleting spam in the morning.
Since the firewall functions based upon code rather than emotion and intuition, the firewall's error rate is going to look better and better against human error as it handles more and more mail.
amidoacetic platymyoid granomerite nonacceptant dorsoposteriad uninclined unshocked zibet intercity lornness
You mean TMDA.
Not new. Nobody ever sends the replies. Mailing lists automatically ban users who run it (I know I do... if they didn't want email they shouldn't have frikkin registered, so I grant them their wish and ban them.).
people not considering their mail important enough
Well if you don't consider my email important enough to read it before assuming it's spam, I don't see why I should continue the conversation.... Sucks for you if I just sent you a job offer..
Thanks to spam, I have been able to remortgage my house online seventeen times to pay for diet pills, pirated software, false identity cards and bogus certificates proving I am a minister of religion.
Not to mention my enormous, permanently erect p3N1s.
Just say NO to spam-blocking!
I'm not wrong. You haven't thought about it hard enough.
For example, Mail Avenger allows you to filter spam based on network characteristics like SYN fingerprints and routes. It even integrates with the kernel firewall to filter out aggressive spammers and mail bombers. However, because it runs as an ordinary user-level process, it also has much more flexibility, for example allowing individual users to set different policies on different email addresses. What can a spam "firewall" do that you can't do with a system like Mail Avenger.
(Presuming that wasn't a troll) That's a horrible, horrible solution. Viruses fake sender addresses, which means the faked address gets *loads* of these 'Please confirm' emails, clogging up another innocent mail server. Get it wrong, and you'll have two servers sending 'Please confirm' messages to each other until one screws up into a little ball and dies. I'm all for the War Against Spam, but this isn't the way - it just doubles the amount of emails.
Here's a nice How-To that covers building an SMTP mail relay with SpamAssassin, Amavisd, DCC, Razor, and Clam AntiVirus all running chrooted on OpenBSD.
Once the relay determines a message is spam, it rejects and drops the message before it is transferred to the 'real' mail server. End users never even know the message was there...
We set up two of these about 6 months ago and eradicated most of our spam problems. (some still get through, on the order of 5 - 10 false negatives on a mailserver handling about 3k messages per day.)
Everyone will start to cheer when you put on your sailin' shoes.
This didn't make it through my bullshit filter. Oh - sorry, I mean bullshit firewall. It's like this new technology that rejects bullshit from the evil internet, so I never have to read it. Thank god, because if I'd read about this "revolutionary spam firewall" I would be forced to make a childish comment on slashdot and burn some karma.
One of the nice things about the Barracuda is that I can configure it as a spam filter or a firewall.[1] I can decide whether to have certain mails stopped at the border, or dumped in a special box, or passed through (and optionally tagged).
/. ad, so /. isn't a complete waste of time! 8^) has done a great job so far. For the first week, I put 1-2 hours in per day going through the list, training things. Then I dropped down to 1 hour a week for a couple of weeks. Now I spend very little time on it. It's great.
In fact, you can do all this with free software as well. It's just that the free software was freaking out on us, and requiring way too much handholding. We were losing email, and having huge delays.
The Barracuda (which we found through a
Is it perfect? No? But most of my complaints are niceties in the GUI, so it's still well ahead of where we were before trying to maintain things ourselves.
This may be a new, rockin' way to detect spam, but if so, they need to pitch it better. They're focusing on the wrong things, IMO. I have an enterprise to run, and marketing jive doesn't cut it.
[1] It's a dessert wax and a floor topping!
They are celebrating false positives?
That's not a firewall either - it's a sandbox (and not new, either)...
The guy is not asking for a sandbox. He is asking for the ability to give or deny individual processes write-access to the hard drive. That's something quite different from a sandbox.
I would also be interested is software that does this.
Kaa
Kaa's Law: In any sufficiently large group of people most are idiots.
Hellfire missiles into the offices of spammers. It's the only way to be sure.
--- Ban humanity.
Any sufficiently advanced spam is indistinguishable from ham.
Fenley's torment.
-John Fenley
All these phrasings automatically trigger my B.S. filter. Or should I say firewall.
You smell that vapor? Sounds like bullshit to me.
Someone has figured out how to build a "spam firewall" that is different from everything out there. Yeah right. No details to tell us exactly how it is different.
My guess is that they took a software based product using baysien filters and some other common anti-spam filtering technology and packaged it in hardware. Won't really improve the function of the machine but could possibly help with performance (process mail faster).
I won't believe it is anything else until I actually see it. Unfortunately, I don't think that will happen anytime soon.
your name is Dick? My father, whose name is Dick, has had endless trouble with spam filters blocking all of the messages he sends where he uses his own name, or when clients send him email using his name. It seems most filters and firewalls don't distinguish between "Dick" and "dicks," and this is a problem for businesses, where context is so important.
From the site: These three additions change the first equation to (3*13*17*4*3*17) variations, and boost the second equation to ( 192 x 3 x 192 x 13 x 192 x 17 x 192 x 4 x 192 x 3 x 192 x 17 x 192) = 1,300,925,111,156,286,160,896. Thanks Greg, Ryan and SR, you helped push the total into the SEXTILLIONS!
Please don't tell me I'm the only one who finds it ironic that the number of different ways to spell it comes out as sextillions...
Dealing with lawyers would be a lot less tedious if they all looked like Casey Novak.
Hell, there's even a product called the Mail Firewall that pops up if you google for mail firewall.
If the spammer gets a "try later" response, he tries later ONE TIME. Worst-case this doubles their bandwidth costs and delays everything by 4 hours.
Today, MOST bad addresses will get SOME OTHER reply, so the cost increase is 2x.
I agree that it's a GOOD stopgap measure but it will fail as soon as the spammers catch on.
On the other hand, spammers might catch on to the idea that "these people are likely to complain, so I don't want to mail them anyways." That would be a Very Good Thing.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Heuristic analysis - detects and blocks spam by various email characteristics
Black lists - checks if the sending server is in RBL (Realtime Blackhole List), dial-up or open-relay servers
DNS verification - checks if the sender is using a valid mail server
Keyword blocking - blocks spam according to keywords in subject and body
Anti-spoofing - blocks email masquerading as coming from within the organization - a common spam technique
Cookies/web beacons - blocks email cookies which help spammers identify the recipient as a "live" email
Header verifier - inspects various header signatures and blocks spam
Textual analysis - categorizes spam according to textual content like mortgages, pornography, dental care, etc
Spam signatures - an auto-updating spam database allows detection and blocking of spam according to smart signatures
Spam URL filtering - blocks email with links to spam sources and sponsors
Spam image filtering - blocks email containing spam associated images
Auto-updating database - local or remote spam blocking database based on thousands of Spam collecting bots and web crawlers
http://www.esafe.com/esafe/anti-spam.aspeSafe
Here's a hypothetical:
1) I get a spam "from" you and forward it to you with a note saying "did you send this." You want to get this type of email. Since you might get such a message from anyone at any time, traditional "is he in my mailing list" filters aren't suitable.
2) I'm a spammer and malware writer, and I write a virus that sends mail from my victim's machine that looks identical to #1. Even though the message is malware-free, you definately do NOT want this message.
No human recipient can tell the two apart, by looking ONLY at the received email.
Of course, no computer can identify "friend or foe" by simply looking at the message either.
So, if you are looking for the perfect filter, it doesn't exist.
If you are looking for a filter that's better than a person, I recommend Yahoo for web-based mail and a number of good solutions for your own system.
In the above scenario, there are solutions. One requires analyzing multiple copies of the message to spot patterns, something big houses like AOL and Yahoo can do but small shops that may only get 1 copy of the message cannot. You can also use RBL lists that track zombied machines, but that won't trigger if the machine in question isn't RBL'd yet. Delay-try-again-later tactics like those mentioned elsewhere in this thread can help here, but are ruinous if you want legitimate complaints ASAP. "Man in the loop" solutions like sending a confirmation message might help, but many people ignore such requests.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Yes:
Almost forgot:
4.
5. Profit
Eh, never mind...
www.mxlogic.com
Any incoming email that spamassassin detects as spam I record the IP for. If that IP has more than 2 infractions in a given amount of time I execute an ssh command to add an iptables rule to my firewall to block that IP. Problem solved.
The firewall I use does exactly what this company is claiming their new product does. I've been running it for years. It's Open Source to boot. It's called messagewall, and I think it's great. My (other) mail server receives between 100 and 700 spams a day, out of which I actually receive 1 or 2. I like it because it rejects the mail if it is spam before the sending server can actually send it.
The down side, you have to load, compile, and build it. It's not too bad, even for a non programmer like me.
CC
Let's configure all SMTP servers to drop mis-spelled email. Then not merely will we have ended the scourge of spam, but also cleared the internet of dumb people. This is not a bug!
You should re-run your study, and correlate against average IQ before and after...