Slashdot Mirror
Revolutionary Spam Firewall Developed
psy writes "physorg has a story on a new spam firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
I have a simple algorithm to reject spam: spelling.
If you can't spell correctly, then I don't want your v1agr4.
Life is the leading cause of death in America.
I think Barracuda Networks would rather disagree with the idea that this is the "only true spam firewall in existence," considering that Barracuda's entire product line consists of spam firewalls.
Damn fine spam firewalls, too, I might add. They handle around 115 messages per second, and can run up to eight filtering steps (including Bayesian analysis, which is similarly efficient to SVM, which the one in the article uses). Plus Barracuda's can do virus scanning.
I'm not sure how this is revolutionary.
This isn't a firewall as it doesn't filter based on addressing. Furthermore, the use of SVMs (support vector machines) to classify text is not new...
Fetchmail + SpamAssassin?
What am I missing here?
Doesn't save B/W: you need to run in INSIDE your network.
Don't care how fast it is: It's a dedicated server.
1/25,000 failure rate with no false positives: OK, that's good. But still not amazing.
How are their servers?
I would rather be ashes than dust!
It's easy to produce these kind of results in trials - you just tune the spam filter to handle a certain set of emails, then you feed it those emails again and you get a near 100% success rate.
Heck, why not do it with a million emails? Makes better headlines that way.
I don't see how this is any different to SpamAssassin (the term 'Mail Firewall' is pure marketing bullshit. It's a spam filter. Get over it.) except I bet it costs a hell of a lot more...
1 out of 25k is impressive, but what happens to these spam mails? Are they bounced back as an error "no user account found"? Or done like a blackhole where the spammer doesnt know if it reeached its intended recipiant? I like my SpamBayes :)
I know! Ciphertrust's Ironmail works the same way... It stops ALL mail inbound, runs it through about a dozen different detection queues, only letting legitimate stuff through. I'd really like to see how this new one is otherwise unique.
Ed R.Zahurak
You know, oblivion keeps looking better every day.
Although this is a great new technology, for a business setting, I don't know if even missing one e-mail is acceptable...
That's what everybody says but what's the other option? Letting all the SPAM come in? Do you really think that fed-up employee who gets hundreds of SPAMs a day is really going to do a better job of just mashing down the delete key then a SPAM filter with a 1/25000 error rate?
Of course I doubt this technology would perform that well but the point still stands -- if you don't have a computer flagging them then chances are you have a human flagging them. Who do you trust more?
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Most users of email are now treating it as a lossy messaging system, and the users themselves accept that some messages simply don't make it. Critical business is always followed up with a call.
-Adam
If you are sending something so critical then you shouldn't be using email. FedEx with signature required delivery and certified/return-receipt USPS mail exist for a reason.
"...companies losing valuable employee time to deleting spam..."
/.ed, here:3
Maybe they should be working on a Slashdot-Firewall. Damn, I really should get back to work.
Oh, and since the linked article got
http://www.uq.edu.au/news/index.phtml?article=583
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
Heh. Does anyone else see that as a good way to downplay false positives?
"Oh, good point, Computer. That email from my boss actually was spam. I didn't realize that until you mentioned it."
Lack of eloquence does not denote lack of intelligence, though they often coincide.
You shouldn't be exposing port 6000 anyway.
In Soviet America the banks rob you!
I submitted this as an ask slashdot and was promptly rejected, so I'm going to put this here as a slightly on-topic post.
What I want to see is a software hard drive "firewall." If you're not sure what I mean, think of what a product like zone alarm does when spyware.exe tries to access the internet on your pc. It pops up a window saying "do you want to allow this program..." Now, why can't we have the same thing for hard drive access? So, I download fungame.exe, and when I go to run it, my "firewall" tells me fungame.exe is trying to write to fifteen different directories to install different spyware products. It could only give a popup on the first time a program tries to write to a given directory, and have an option to not show any new notices for this program, to limit the annoyance factor. I think this would be a great tool to help lessen spyware/trojan problems. If the program interacted with spybot or a similar product, it could even automatically prevent writing of files that are known to be adware. Is there anything like this out there? Anyone who would be willing to help make it?
and if we missed 4 legit client emails a day... that would be lost business, and that's just unacceptable no matter how you look at it.
Well... how much money would it take to have the staff necessary to do the filtering manually (at a better rate - even humans are fallible), and how much would the potential business loss cost you? Assuming that the business was very profitable, and that the senders wouldn't call or send a follow-up email of course.
You're special forces then? That's great! I just love your olympics!
Unconsciously Desired Email Industry (Our slogan: You opted in in your heart!), I'd like to strongly protest the continuing escalation of technology against us. We provide the opportunity for hundreds of thousands of people to spend freely on products unburdened by simple heuristics of "they work" or "they won't make you ill" or "we'll actually send them". Why are you so intent on interfering with the consumer ethos?
You mean it blocks all email, and the one ligitimate email among the 25000 is the "misclassed" one...
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
One of two conditions exists in this case.
1) The e-mail is vitally important and your business will be seriously damaged by its failed delivery.
2) The e-mail was somewhat important, but not something large enough to materially change your revenue/profits.
If the first is the case, you probably shouldn't be using e-mail in the first place and/or whoever sent it is probably going to follow up with a FedEx or phone call.
In the case of number 2 (ha ha, number two), you've saved so much time not having to wade through spam that the losses are negated.
The idea is that the mail server keeps a whitelist of "allowed" addresses which are always accepted. If a mail comes from an address which is not known, the mail server will reply with a "server unavailable, try later" error message. All real mail servers will try to send the message a little later (I don't know the exact time, but it's probably less than an hour. Someone else might know better).
The second time the remote mail server tries to connect, the server accepts the mail and adds the address to the whitelist.
However, mass mailers for spam don't do this but simply go on to the next address in the list if this happens. This way the spam message is filtered out.
Note that this method doesn't require any analysis of the actual content of the messgae, nor does it involve any manual actions from neither the sender nor the receiever. Currently it's porbably the best spam blocking method that exists.
amidoacetic platymyoid granomerite nonacceptant dorsoposteriad uninclined unshocked zibet intercity lornness
1/25000 is significantly better than a human being. If you use no automatic spam filtering at all, and you get a typical geek's email load (about 100 spam a day with 10 legitimate emails a day), you will still delete mail as spam when it wasn't spam.
That's why I use SpamAssassin - it does a good job, and is no worse at making false positives than I am. If I'm just as liable to make a false positive than an automatic filter, I'm better off saving my time.
Oolite: Elite-like game. For Mac, Linux and Windows
Thanks to spam, I have been able to remortgage my house online seventeen times to pay for diet pills, pirated software, false identity cards and bogus certificates proving I am a minister of religion.
Not to mention my enormous, permanently erect p3N1s.
Just say NO to spam-blocking!
I'm not wrong. You haven't thought about it hard enough.
For example, Mail Avenger allows you to filter spam based on network characteristics like SYN fingerprints and routes. It even integrates with the kernel firewall to filter out aggressive spammers and mail bombers. However, because it runs as an ordinary user-level process, it also has much more flexibility, for example allowing individual users to set different policies on different email addresses. What can a spam "firewall" do that you can't do with a system like Mail Avenger.
(Presuming that wasn't a troll) That's a horrible, horrible solution. Viruses fake sender addresses, which means the faked address gets *loads* of these 'Please confirm' emails, clogging up another innocent mail server. Get it wrong, and you'll have two servers sending 'Please confirm' messages to each other until one screws up into a little ball and dies. I'm all for the War Against Spam, but this isn't the way - it just doubles the amount of emails.
This didn't make it through my bullshit filter. Oh - sorry, I mean bullshit firewall. It's like this new technology that rejects bullshit from the evil internet, so I never have to read it. Thank god, because if I'd read about this "revolutionary spam firewall" I would be forced to make a childish comment on slashdot and burn some karma.
Then you're stuffed anyway, because internet e-mail is not guaranteed.
It is difficult. We're swatting away a million of the damn things a week and still our users complain. They also complain when we get false positives. And when, next week, we turn on the system that lets them see what we have blocked that was addressed to them, they'll complain too.
I think the one solution they would find acceptable is for me to personally read every one of those million messages and mark it as good or bad. I hope our VP doens't read slashdot....
~~~~~ BigLig2? You mean there's another one of me?
Hellfire missiles into the offices of spammers. It's the only way to be sure.
--- Ban humanity.
Any sufficiently advanced spam is indistinguishable from ham.
Fenley's torment.
-John Fenley
I support among other people, a marketing staff. When people are interested in buying things, they may only send one email. That one email is all you are going to get, and not getting it is the same as not getting the sale. I know the marketing staff is extremely skeptical about any sort of spam filtering, as they are always concerned about missing important emails that may lead to sales, and ultimately, revinue. I don't know how this fits in with spam filtering, but suggesting that all important email is followed up with a call is not true. And ask any CEO--sales are the most important thing to a company. It doesn't matter if you have the best thing in the world, if you can't sell it, it isn't worth anything.
If missing one email is not acceptible to your business, then your business should not be using email ever anyway - email is not, nor has it ever been, a guaranteed delivery mechanism.
At our company, current just over 50% of all inbound email is detected as spam. Thus more than 50% of all our inbound email is spam, and the true figure (allowing for the false negatives which slip through) is probably in excess of 60% (and rising)
With a failure rate of 1 in 25,000, AND assuming that means a false positive rather than a false negative, then for our company taking into acount the volume of spam we receive it means 1 email in > 55,000 is wrongly identified.
I can assure you that our business is capable of coping with 1 missed email in > 55,000.
We certainly do not to business-threatening-essential transactions via insecure, non-guaranteeded publicly-transported email, and nor shoudl your business!
People should not be afraid of their governments - Governments should be afraid of their people.
From the site: These three additions change the first equation to (3*13*17*4*3*17) variations, and boost the second equation to ( 192 x 3 x 192 x 13 x 192 x 17 x 192 x 4 x 192 x 3 x 192 x 17 x 192) = 1,300,925,111,156,286,160,896. Thanks Greg, Ryan and SR, you helped push the total into the SEXTILLIONS!
Please don't tell me I'm the only one who finds it ironic that the number of different ways to spell it comes out as sextillions...
Dealing with lawyers would be a lot less tedious if they all looked like Casey Novak.
It is in principle possible to produce a reliable email system, but only if a receipt is returned to the sender when the recipient actually reads the mail, not when it arrives at his ISP for example.
Sadly some businesses do rely implicitly on things that usually, but not always, work, such as mobile phones, pagers, and text messaging. It may have been the same with pigeons, a predator might get the bird! Businesses should set up foolproof systems if they want to do well, a quick phone call to confirm receipt of critical items, for example. The occasional email, even now, takes many hours or even several days to arrive, there is no guarantee whatsoever of time of arrival, but again some seem to think it is "instant", because it very often is. Managers should be aware of these issues, sadly some are not.
But I hope this anti-spam firewall is a brilliant success, and that if it has minor shortcomings there will be satisfactory work-arounds. I am sick of spam, but the ultimate answer must be to ensure that it does not pay, i.e. that the probability of being caught multiplied by the fine greatly exceeds the potential profit. That requires legislation worldwide and some conceptually simple additions to existing mail servers, with care taken to protect the privacy of normal users. Given the political will, and some competent leaders (not Dubya or B. Liar, for a start) it should be easy.
Heuristic analysis - detects and blocks spam by various email characteristics
Black lists - checks if the sending server is in RBL (Realtime Blackhole List), dial-up or open-relay servers
DNS verification - checks if the sender is using a valid mail server
Keyword blocking - blocks spam according to keywords in subject and body
Anti-spoofing - blocks email masquerading as coming from within the organization - a common spam technique
Cookies/web beacons - blocks email cookies which help spammers identify the recipient as a "live" email
Header verifier - inspects various header signatures and blocks spam
Textual analysis - categorizes spam according to textual content like mortgages, pornography, dental care, etc
Spam signatures - an auto-updating spam database allows detection and blocking of spam according to smart signatures
Spam URL filtering - blocks email with links to spam sources and sponsors
Spam image filtering - blocks email containing spam associated images
Auto-updating database - local or remote spam blocking database based on thousands of Spam collecting bots and web crawlers
http://www.esafe.com/esafe/anti-spam.aspeSafe