Slashdot Mirror
Revolutionary Spam Firewall Developed
psy writes "physorg has a story on a new spam firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
I have a simple algorithm to reject spam: spelling.
If you can't spell correctly, then I don't want your v1agr4.
Life is the leading cause of death in America.
filter out mesages from my x ;-)
another Roadkill on the Information Superhighway
I think Barracuda Networks would rather disagree with the idea that this is the "only true spam firewall in existence," considering that Barracuda's entire product line consists of spam firewalls.
Damn fine spam firewalls, too, I might add. They handle around 115 messages per second, and can run up to eight filtering steps (including Bayesian analysis, which is similarly efficient to SVM, which the one in the article uses). Plus Barracuda's can do virus scanning.
I'm not sure how this is revolutionary.
Sourceode would be nice....
What the hell is one of these? There seems no substance to this report, bar some TLAs as above and a load of hype. Where is the proof? How was it tested? Etc.
When they actually start publishign details on how I can do it, I may care.
Right now, it doesn't sound that much more effective than running SpamAssassin on messages as a receipt time, and rejecting them if they score too highly.
Although this is a great new technology, for a business setting, I don't know if even missing one e-mail is acceptable...
The words revolutionary and spam in the same phrase... frightens me.
This isn't a firewall as it doesn't filter based on addressing. Furthermore, the use of SVMs (support vector machines) to classify text is not new...
I'll believe it when I see it...
Remember, CRM114 was supposed to be the sh*t, too..
Fetchmail + SpamAssassin?
What am I missing here?
Doesn't save B/W: you need to run in INSIDE your network.
Don't care how fast it is: It's a dedicated server.
1/25,000 failure rate with no false positives: OK, that's good. But still not amazing.
How are their servers?
I would rather be ashes than dust!
Well, this certainly sounds like a good thing for many people, but because it's been described as "firewall" and not a "server-side filter", I certainly hope it wouldn't be set up at major ISPs to intercept all smtp traffic going through.
Hopefully their spam firewall is more robust than their web server.
It's easy to produce these kind of results in trials - you just tune the spam filter to handle a certain set of emails, then you feed it those emails again and you get a near 100% success rate.
Heck, why not do it with a million emails? Makes better headlines that way.
I don't see how this is any different to SpamAssassin (the term 'Mail Firewall' is pure marketing bullshit. It's a spam filter. Get over it.) except I bet it costs a hell of a lot more...
1 out of 25k is impressive, but what happens to these spam mails? Are they bounced back as an error "no user account found"? Or done like a blackhole where the spammer doesnt know if it reeached its intended recipiant? I like my SpamBayes :)
There were 3 comments when I first tried to load the article, but alas ... The server was /. --ed already ..
echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
I know! Ciphertrust's Ironmail works the same way... It stops ALL mail inbound, runs it through about a dozen different detection queues, only letting legitimate stuff through. I'd really like to see how this new one is otherwise unique.
Ed R.Zahurak
You know, oblivion keeps looking better every day.
Until there is a 0% fail misclassification rate such a method is useless. Filtering was one thing, if you misfiltered a message you always had the oppertunity of occasionally scanning your SPAM box and making sure everything was about penis enlargement and not about the meeting you have next week. However, with this method email is stopped and never delivered, thus your misclassified email is now gone- forever.
I'd rather get 5 extra spam if it meant I also recieved every real email.
transmission_err
"...companies losing valuable employee time to deleting spam..."
/.ed, here:3
Maybe they should be working on a Slashdot-Firewall. Damn, I really should get back to work.
Oh, and since the linked article got
http://www.uq.edu.au/news/index.phtml?article=583
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
Heh. Does anyone else see that as a good way to downplay false positives?
"Oh, good point, Computer. That email from my boss actually was spam. I didn't realize that until you mentioned it."
Lack of eloquence does not denote lack of intelligence, though they often coincide.
Could it be count as spam? In that case, will users behind that spam firewall receive it by mail?
By their definition, qmailscanner is a firewall too. It stops (quarantines) spam and only lets legitimate email through.
Semantics.
Feed the need: Digitaladdiction.net
I submitted this as an ask slashdot and was promptly rejected, so I'm going to put this here as a slightly on-topic post.
What I want to see is a software hard drive "firewall." If you're not sure what I mean, think of what a product like zone alarm does when spyware.exe tries to access the internet on your pc. It pops up a window saying "do you want to allow this program..." Now, why can't we have the same thing for hard drive access? So, I download fungame.exe, and when I go to run it, my "firewall" tells me fungame.exe is trying to write to fifteen different directories to install different spyware products. It could only give a popup on the first time a program tries to write to a given directory, and have an option to not show any new notices for this program, to limit the annoyance factor. I think this would be a great tool to help lessen spyware/trojan problems. If the program interacted with spybot or a similar product, it could even automatically prevent writing of files that are known to be adware. Is there anything like this out there? Anyone who would be willing to help make it?
For those who belive this software actually can do this well in real-life environment, I have this bridge that might interest you ...
echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
Now.
Sincerely,
Your Boss
It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails. Who cares if a few get through incorrectly. The interesting statistic is how it does on incorrectly labelling legitimate mail as spam.
I guess I don't usually associate the term "firewall" with spam filtering. The article only touches on their use of the terminology in the quote that you've selected. Otherwise, it's a general discussion of filtering techniques and the effects of spam on the internet.
If they're maintaining that they filter out spam prior to it hitting the email server, or well before it hits the email client, then they really need to get out more before making the claim that they're the only one to do it. My personal fav these days is GFI MailEssentials, which stops spam at the server level by examining the incoming SMTP traffic.
Their SPAM firewall may work but their webserver seems to need a Slashdot firewall installed. The site is a burning mash of hardware now. Guess they don't have to worry about SPAM now.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through"
Slashdotting has made it impossible to check for more meaning in the article, so can anyone tell me what the difference is supposed to be here. How does stopping mail and then allowing non-spam through differer from a spam filter? It sounds like pretty much what the qmail/spamassassin boxes I've set up as mail gateways do.
"Are you being weird, or sarcastic?" said Emma. I said I didn't know because I get the two feelings mixed up.
pretty simple.. filtering out html email (99.9% of which for me is spam) and then all the pen1s and v1agr4 (misspelled words, particularly in small concentrations) combined with a URL.
one bad thing about all the misspellings is that the spam poetry project got messed up..
anime+manga together at last.. in real time.
Unconsciously Desired Email Industry (Our slogan: You opted in in your heart!), I'd like to strongly protest the continuing escalation of technology against us. We provide the opportunity for hundreds of thousands of people to spend freely on products unburdened by simple heuristics of "they work" or "they won't make you ill" or "we'll actually send them". Why are you so intent on interfering with the consumer ethos?
You mean it blocks all email, and the one ligitimate email among the 25000 is the "misclassed" one...
If you disagree with me on social issues, then it's pretty clear that you are a narrow-minded bigot.
Does spam even exist anymo... oooh, you mean the e-mail spam! got'cha ;)
Yeah, seems neat.
I'm a signature virus. Please copy me to your signature so I can replicate.
Does this bridge filter traffic as well?
I remember a swedish guy explaining me his solution to SPAM. Each sendern, which isn't registered in the server whitelist will get a notification back like in many mailing list registrations. After replying the mail goes through and is entered into the whitelist. A nice side effect is that this not only filters out all the faked senders, but also people not considering their mail important enough to acknowledge they sent it.
That should have read ..
"filter out html and junk words."
heh..
anime+manga together at last.. in real time.
Put any website up and it is automatically filtered out of existence.
Aren't these guys (psyorg) the same ones that showed us the 100TB storage disk with the cheesy animated gif a little while ago?
That's not a firewall either - it's a sandbox (and not new, either)
That's why I put the quotes around the word firewall, and I would have never thought to google the term sandbox to find such a product. Do you have any suggestions for good sandboxes, now that I know what it's called?
The idea is that the mail server keeps a whitelist of "allowed" addresses which are always accepted. If a mail comes from an address which is not known, the mail server will reply with a "server unavailable, try later" error message. All real mail servers will try to send the message a little later (I don't know the exact time, but it's probably less than an hour. Someone else might know better).
The second time the remote mail server tries to connect, the server accepts the mail and adds the address to the whitelist.
However, mass mailers for spam don't do this but simply go on to the next address in the list if this happens. This way the spam message is filtered out.
Note that this method doesn't require any analysis of the actual content of the messgae, nor does it involve any manual actions from neither the sender nor the receiever. Currently it's porbably the best spam blocking method that exists.
I'd guess that if you put the firewall up against your average email user, the average user would shitcan legitimate messages at a much higher rate than the firewall thanks to the fact that the user can get frustrated while the firewall can't. I know my boss accidentally deletes mail from me at least 3 times per week because he's careless while mass-deleting spam in the morning.
Since the firewall functions based upon code rather than emotion and intuition, the firewall's error rate is going to look better and better against human error as it handles more and more mail.
We keep adjusting the frequency of the shields and they keep adjusting the frequency of the phasers. So to speak.
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
E-mail ALWAYS (sorry for the yelling) was a lossy messaging system. Initially, it did not have confirmation receipts or anything.
It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
The food of choice for Che Guevara.
this can definitely be done. .NET had an example that shipped with it that logged each r/w to the filesystem. also, GoBack must use a similar mechanism of the win32 api.
amidoacetic platymyoid granomerite nonacceptant dorsoposteriad uninclined unshocked zibet intercity lornness
Why is it anytime a filter is discussed, everyone starts yammering about "1 is too many" and in reality, a 1000 would still be fine.
email is an unreliable system, so dont expect it to deliver every message flawlessly to begin with.
i think people get all antsy about it, because they like to think their email is just soo damned important, arctic winds will freeze the entire planet if they dont get whatever lame useless email from their spouse/manager/cousin.
if it were that critical that the person absolutely must know that information, it's called a fucking telephone.
over inflated self importance.
Thanks to spam, I have been able to remortgage my house online seventeen times to pay for diet pills, pirated software, false identity cards and bogus certificates proving I am a minister of religion.
Not to mention my enormous, permanently erect p3N1s.
Just say NO to spam-blocking!
I'm not wrong. You haven't thought about it hard enough.
For example, Mail Avenger allows you to filter spam based on network characteristics like SYN fingerprints and routes. It even integrates with the kernel firewall to filter out aggressive spammers and mail bombers. However, because it runs as an ordinary user-level process, it also has much more flexibility, for example allowing individual users to set different policies on different email addresses. What can a spam "firewall" do that you can't do with a system like Mail Avenger.
Here's a nice How-To that covers building an SMTP mail relay with SpamAssassin, Amavisd, DCC, Razor, and Clam AntiVirus all running chrooted on OpenBSD.
Once the relay determines a message is spam, it rejects and drops the message before it is transferred to the 'real' mail server. End users never even know the message was there...
We set up two of these about 6 months ago and eradicated most of our spam problems. (some still get through, on the order of 5 - 10 false negatives on a mailserver handling about 3k messages per day.)
Everyone will start to cheer when you put on your sailin' shoes.
I don't know if this is still being
updated...
Try googling for a program called
"In Control" - inctrl. It's
not quite as automated and slick as one
might like, but it will tell you what
the installer is doing.
Otherwise, thanks for the million bounces I got that week some [redacted] was forging my domain.
This didn't make it through my bullshit filter. Oh - sorry, I mean bullshit firewall. It's like this new technology that rejects bullshit from the evil internet, so I never have to read it. Thank god, because if I'd read about this "revolutionary spam firewall" I would be forced to make a childish comment on slashdot and burn some karma.
Now, why can't we have the same thing for hard drive access? So, I download fungame.exe, and when I go to run it, my "firewall" tells me fungame.exe is trying to write to fifteen different directories to install different spyware products.
Don't run software as root.
Oh, you're using Windows. Try using an account with non-Administrative priveleges and see if you can get by with runas.exe to run installers, and ensure that %WINDIR% and anything in your path is not writeable by your normal user account.
What the OP is asking is something outside of any application, so that no matter what you're running, the firewall kicks in. Also, the OP isn't looking for a logger that tells you what happened after the fact (after your password file's been sent to their web server), but something that stops unwanted disk access before it happens.
I've never seen anything that does what you want, but there's definitely programs that log drive access. FileMon at sysinternals.com (http://www.sysinternals.com/ntw2k/source/filemon. shtml) will do it.
I don't know exactly what API it's using and if it could "reject" those accesses though. It's not a bad idea.
On a *nix box - man chroot. :)
retrorocket.o not found, launch anyway?
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
Then you are nitwits.
Ummm... This exists. In part, anyway.
After it was discovered that web pages could autorun/autoinstall softmware, (I don't know the technical details, but anyway) Apple instituted this feature as a security device. Any app that wasn't specificall requested by the user and not a known system app, would the first time, pop up a message stating that this program (insert app name) is trying to run, if you want it to run, click OK, if not, or if appears malicious, click cancel.
Of course, this is a fairly new security addition to the MacOS X environment, and not for all you main stream windows ppl. But it's possible in one place, so it should be possible elswhere. Deffinitly a good security feature.
If expanded to auto block known spyware apps it could be even better.
Z
I'm not talking about programs screwing up windows so much as installers coming with a bunch of extra crap. For example, when kazaa first came out, not many people knew it installed gator. If you had a progrm that pops up a big ugly window saying "foo.exe is trying to write to c:\program files\gator\" you would become suspicious much more quickly. Restricting access to WINDIR wouldn't help because if I'm running an installer, it would need access to program files, and hence could install any other useless crap it wants.
There are far more effective solutions available here and here. In fact, you could get ahead of the curve for when people start trying to write spyware for Linux. Do a front-end to LIDS. Install it with a restrictuve ruleset, and then the front-end monitors the warning logs. If it detects something then it pops up a box saying "blah just tried to write to directory foo, do you wish to authorise this?". If the user clicks yes then add a new rule and restart LIDS. Obviously this isn't perfect as you would then have to re-run the command. It would be better to write hooks into LIDs itself for this purpose.
Phillip.
Property for sale in Nice, France
One of the nice things about the Barracuda is that I can configure it as a spam filter or a firewall.[1] I can decide whether to have certain mails stopped at the border, or dumped in a special box, or passed through (and optionally tagged).
/. ad, so /. isn't a complete waste of time! 8^) has done a great job so far. For the first week, I put 1-2 hours in per day going through the list, training things. Then I dropped down to 1 hour a week for a couple of weeks. Now I spend very little time on it. It's great.
In fact, you can do all this with free software as well. It's just that the free software was freaking out on us, and requiring way too much handholding. We were losing email, and having huge delays.
The Barracuda (which we found through a
Is it perfect? No? But most of my complaints are niceties in the GUI, so it's still well ahead of where we were before trying to maintain things ourselves.
This may be a new, rockin' way to detect spam, but if so, they need to pitch it better. They're focusing on the wrong things, IMO. I have an enterprise to run, and marketing jive doesn't cut it.
[1] It's a dessert wax and a floor topping!
I run Gentoo both at work and on my xbox, did my first Debian install via ftp over a 14.4 modem years ago. I know linux solves the spyware issue (for now) but I am not a zealot and run several different OSes and like to find the best way to utilize each.
They are celebrating false positives?
That's not a firewall either - it's a sandbox (and not new, either)...
The guy is not asking for a sandbox. He is asking for the ability to give or deny individual processes write-access to the hard drive. That's something quite different from a sandbox.
I would also be interested is software that does this.
Kaa
Kaa's Law: In any sufficiently large group of people most are idiots.
Bwahahahahaha!
This already exists in the form of directory security: assign permissions accordingly and run untrusted software as a very low-privileged user.
Trust me, though: you don't want to know how many I/O operations a program can perform. Try running FileMon (it's freeware) once or twice to see what I mean. Do you REALLY want to be prompted every time I/O is performed? Most users would automatically hit whatever button dismissed this kind of warning because they see it too often.
What I want to see is a software hard drive "firewall."
It sounds like what you want is a filesystem driver that warns you when a "sandboxed" app. is trying to write a file to disk and allows you to prevent it. This does seem like an excellent idea, and though I don't know of any products that address this specific need, the latest versions of many personal firewalls have similar "application protection" features where they will warn you if an app. tries to write something outside of it's directory.
On the filesystem side, specifically, you may want to try FileMon from SysInternals. It's free, although not OSS, but they link to some great books and articles describing the Windows' filesystems. (You may also want to look into the IFSKit from M$, though you have to pay to use it.) I bet you could get a good start by looking at FileMon and trying to fiture out what it does and how it implements those capabilities...
OK, now tell me this.
If I'm filtering 500 bad "please respond to get on the white list" messages a day sent ostensibly from my hosts/addresses, how am I supposed to tell the different between white list requests that are genuine?
And thanks Barracuda for amplifying a 1k spam into a 5k bloated HTML white list response!!
There may be some legitimacy to the application you're looking for, but I've got an easier solution :
/Program Files.
;)
1) Don't run apps as Administrator.
2) Don't have your user account in the Administrator's group.
Why should an application have to babysit an account with more privs than it needs? If you run funprog.exe as a user, there are very limited places you should be able to write (in a bug / design problem free world anyawys.)
Of course, there's the issue of the setup application for funprog.exe, but that's MS's problem. Either a user should have a sandbox where it can install it's own private apps, so that they don't need admin to install. Or applications should not be allowed to install to anything outside of
Or, just run Linux
Hellfire missiles into the offices of spammers. It's the only way to be sure.
--- Ban humanity.
Any sufficiently advanced spam is indistinguishable from ham.
Fenley's torment.
-John Fenley
Funny, zone alarm doesn't prompt me every single time there's network traffic, but it sure warns me when a program I haven't authorized for network access tries.
need lots of bandwidth accept incoming mail, and send a email to the return address, if it fails, delete it :) of course the sender would get a email if it was passed, but then other servers could filter that email, as long as it makes it
I use a Barracuda here at work, it handles the SPAM and anti-virus checking before it gets to my mail server. I would class this as a spam firewall. By the way the Barracuda works crazy good!
I think Tiny Personal Firewall already does that.
0 22 904225965A0&offer=&pg=content&an=Windows_Security_ 1
http://www.tinysoftware.com/home/tiny2?s=494940
It's quite a good firewall program, with some nice options like notifying you when an application attempts to spawn a child proccess, access files outside its working directory, change registry settings, etc.
I guess they forgot to read any technological publication sin the last year or so. Ever here of a Barracuda Spam Firewall. Same thing as they think they just invented, and it has been out for a year or so. They at least have a track record with hundreds if not thosands of customers and a years worth of data to back up their claims.
It is the physorg website that is /.ed not the University of Queensland.
:o
Here is the article at the university, which is still up. (for the moment)
Not very much extra information though.
All these phrasings automatically trigger my B.S. filter. Or should I say firewall.
You smell that vapor? Sounds like bullshit to me.
Someone has figured out how to build a "spam firewall" that is different from everything out there. Yeah right. No details to tell us exactly how it is different.
My guess is that they took a software based product using baysien filters and some other common anti-spam filtering technology and packaged it in hardware. Won't really improve the function of the machine but could possibly help with performance (process mail faster).
I won't believe it is anything else until I actually see it. Unfortunately, I don't think that will happen anytime soon.
I've sent lots of emails to people while I'm talking to them on the phone. That way you can be sure that they received it.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan, the man who invented the term "spam firewall."
Secession is the right of all sentient beings.
web pages that make your eyes want to pop out of their sockets?
To catch those chinese and korean spam before they make it to your smtp: This geoip firewall filter for iptables drop mail coming from incriminated countries. This tool gets 50% of the spam I should receive. Combined with dspam, I do not receive anything but genuine mails. Enjoy !
Didn't an old version of Norton Utilities have this? IIRC, it was in the old DOS days. It would pop up and ask you for permission every time a program would try to write to the disk.
Perhaps one of the developers is a balding male who has a small penis with erectile dysfunction and bad credit?
Being stuck calling SA from procmail (and therefore being a "filter" instead of a "firewall" kinda sucks, but it allows for greater flexibility.
I for one welcome our new spam fighting firewalls!
CVB
free ipod and free gmail!
As soon as enough people use this firewall spam senders will also use it to check in advance if theire spam will get through.
Doesn't mean it won't help for a while and we just hope a better firewill will come out as soon as spam got around this one.
I get over 100,000 spam emails a month.
Beleive it or not, I have better things to do with my time then sift through THOUSANDS of garbage spam mails a day.
It's extremely time consuming. Well, was... I gave up on filtering the crap out about a week ago. I shut down the mail account and removed the MX record from my DNS server so I would have to see the damn processes running.
Yeah, losing 1 in 25,000 would have been acceptable to me...
And now, I'll go through and update all my NIC records with my new email and in a few months I'll get the same crap again, I'm sure. But, a few months without having to deal with it is worth it.
This is the second time I've switched my primary email address. Just like the last I had to knock the MX record out for the domain name. It's that or sit and watch the qmail processes run by the thousands as it accepts the spam essentially to just delete it.
Now, if only we could get everyone to redirect their spam to a congressman or something.
Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan. Now mind you I didn't RTFA but it sounds the same to me. I will go RTFA now.
I'm using CRM114. It may not get 99.98% of my spam, but it gets at least 95% (and I haven't trained it all summer), and hasn't misclassifed a good message in a long time. I've done one add-on, though, a mod that helps it find certain forms of dictionary salad.
The sh*t, perhaps not, but still darn good.
I hereby place the above post in the public domain.
http://www.spampal.org/ checks mail on several RBLs.. personal black/whitelists, and it's free.
Astaro Security Linux (www.astaro.com) does this as well, stopping all messages and running them through a gauntelt of anti-spam, anti-virus, domain lookups, callouts, realtime blackhole lists, file extension filtering, keyword filtering etc...BEFORE sending the message through to the recipient.
You can also choose to blackhole, quarantine, reject, or pass a failed check with only a subject line warning.
Best part? It's free for home use!
Click here for his picture, you'll know why I go through the Jesus pharmacy...
If carrots got you drunk, rabbits would be fucked up. - Comedian Mitch Hedberg R.I.P. 03/30/68-2/24/05
A useful sandbox has levels of permissions and write access to the hdd is a common one such...
it seems to me that this is simply an evolution of spam fighting; moving it from the MTA off to the Firewall. I think it's a good move, and should allow of less resources on the MTA going to filtering spam, and shift it to a more appropriate place.
still, I think all spam filters need to do is to: ckek speeling in the emaaills and dteermine if an emaiil has towo mani missplleded wordss.
CVB
free ipod and free gmail!
There is a pproduct called messagewall, which I have used for over a year. It does exactly this. Does the filtering before feeding it to the MTA.
your name is Dick? My father, whose name is Dick, has had endless trouble with spam filters blocking all of the messages he sends where he uses his own name, or when clients send him email using his name. It seems most filters and firewalls don't distinguish between "Dick" and "dicks," and this is a problem for businesses, where context is so important.
August 23, 2004
The email spam nightmare could be halted in cyberspace by a groundbreaking firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
A Specialist Systems Programmer at The University of Queensland, Mr Sullivan worked on the spam firewall concept largely in his spare time, only coming together this year to work on the project with Guy Di Mattina, a recent UQ Engineering honours graduate, and Dr Kevin Gates, a UQ mathematics lecturer.
Pivotal to the trio's spam firewall is the unique method of using a Support Vector Machine (SVM) to categorise emails. The only anti-spam software that analyses emails as a whole picture, rather than based solely on components such as key words or phrases, said Mr Sullivan.
"Using a SVM, we can train our spam firewall to accurately recognise legitimate emails to the extent that it can tell the difference between a pharmaceutical bulletin on Viagra and someone trying to sell Viagra," he said.
UQ's main commercialisation company, UniQuest, has formed a start-up company based on the technology and is seeking investment to take the spam firewall to market.
UniQuest Managing Director, David Henderson said the global cost of spam was estimated by the Radicati Group in 2003 to be $20.5 billion or $49 per user mailbox.
"With spam escalating and companies losing valuable employee time to deleting spam, UniQuest hopes to get this revolutionary spam firewall technology on the market quickly but it just depends on the level of funding we receive," said Mr Henderson.
Source: University of Queensland
What are you guys talking about.... what spam?
Simpy
it gives me something to blame when the VP's mail from Pfizer about selling viagra doesn't get through
For one thing, a decent spam filter will allow IT to whitelist the employees, especially management.
Since the topic of spam has come up again, I blogged about this in January and would welcome feedback on my idea. In summary, it is my belief that we should simply make users accountable for the content of their e-mail and accept everything. Read on...
RP
Gee, I hope they use *different* technology on their SPAM Firewall than they do on their webserver... I'm sure that the scumbag spammers could concot a DDOS attack stronger than a /.'ing...
Here's the message
PhysOrg is temporarily unavailable.
We are currently working to resolve the problem.
Please try again later.
Please accept our apologies for any inconvenience caused by our Web Services.
Borderwarea point
Elron/Zix
IronPort
ISS
McAfee
Mir
Proofpoint (which also uses SVM, by the way!)
Sophos(?)
Symantec/Brightmail
Tumbleweed (which actually has a PATENT on "Email Firewall")
By the way, why does everyone always mention Barracuda when these threads come up? CipherTrust, McAfee, and Tumbleweed (et al) had these concepts as actual products long before Barracuda put SpamAssassin in $300 bargin basement hardware and called it a "firewall".
Someone is WRONG on the Internet!
We just hired Kathy Lee's old employees to review our emails.
Works like a champ and unlike these firewalls and filters all it takes is a stick for them to learn.
As an endnote, after a few months of my account cripple with spam, I went through and followed the unsubcribe links for each mail that I got. Deleted them and repeated until after about 2 weeks I was no longer receiving spam(aside from the university's student announements which I considered to be the worst perpetrator of spam in existence).
I guess my point is that for the most part repeat spam shouldn't be a problem because it can be stopped. Now learning how to not subject yourself to new spam is a valuable thing. And way more useful than some filter/firewall bloat.
take this example of how a filter cannot determine whether something is spam or not -- what if you're a network administrator, writing an email to a colleague about a new spam message that has appeared. You forward the message, with subject and text.
How in the world can a spam filter understand that this is not spam? How can any filter understand the intent of a message?
Wasn't making a great revelation that many of such things do exist (sorry if the wording sounded aggressive - I was annoyed by the article, not your suggestion), nor that they're configured in the way you suggest (which I think is spot on).
Using the term I suggest at least finds some discussion of this idea, like the following article:
http://www.nwfusion.com/newsletters/sec/0913sec2.From the release notes.
spamd(8) gains greylisting support. This allows greylisting (a very powerful spam reduction technique) to be done on a firewall for many mail hosts, no matter what MTA is being used.
Even humans can't classify spam/ham with 0% failure rate. People get bored, hit the wrong keys, etc. all the time.
From the site: These three additions change the first equation to (3*13*17*4*3*17) variations, and boost the second equation to ( 192 x 3 x 192 x 13 x 192 x 17 x 192 x 4 x 192 x 3 x 192 x 17 x 192) = 1,300,925,111,156,286,160,896. Thanks Greg, Ryan and SR, you helped push the total into the SEXTILLIONS!
Please don't tell me I'm the only one who finds it ironic that the number of different ways to spell it comes out as sextillions...
Dealing with lawyers would be a lot less tedious if they all looked like Casey Novak.
Hell, there's even a product called the Mail Firewall that pops up if you google for mail firewall.
I'd settle for a log of the day's disk write activity. Just log the process that accessed the disk, what time it did it, and what file it wrote or changed.
Almost all the spam I get these days is a bitmap image, no text, so how can any filter tell the difference between this spam and a relative or friend sending me a photo?
wake up and hold your nose
yeeessss..
"Oh not ours isn't a spam filter, it;s the worlds first and only Spam Firewall"
"So what's it do then?"
"well, when mail comes in, it classifies it as either spam, or non-spam.."
"and this differs from a spam filter because....?"
Yet another spam filter. Move along.
If you have a sandbox on top of a secure, properly designed OS, then your problems would be minimal. But you would not of course be using either Windoze or Incompetent Exploder, but you could be using almost any other modern OS.
Any sufficiently simple magic can be passed off as mere advanced technology.
Barracuda sells SpamAssassin with a bunch of plugins, installed on Linux, installed on sweat-shop-special PC hardware. They aren't "hardware devices" with an ASIC and real firmware.
They call their Linux OS "firmware", but that doesn't change the fact that it's installed on a hard disk drive and the internals of the box is no different from a 1U or 2U rackmount server that you'd get from Dell, IBM, HP, etc (except that the name-brand hardware is probably 10 times more reliable than the no-name crap that Barracuda uses and has real field service people).
Go ahead, order one of their trial units and open it up.
By the way, anyone familiar with the performance of SpamAssassin and Bayesian will immediately notice that Barracuda's throughput claims are a total farce. Not even the IronPort boxes which run on high-speed hardware (name-brand, SCSI, striped RAID) on a hacked Qmail on a hacked (to the point of unreliability) FS claim the speed that Barracuda claims, and IronPort is widely regarded as the fastest e-mail appliance in existence.
If a pure spam cannon doesn't even claim to process messages as fast as a stock Linux box loaded down with SpamAssassin, how much credibility are you going to give to Barracuda?
Someone is WRONG on the Internet!
If the spammer gets a "try later" response, he tries later ONE TIME. Worst-case this doubles their bandwidth costs and delays everything by 4 hours.
Today, MOST bad addresses will get SOME OTHER reply, so the cost increase is 2x.
I agree that it's a GOOD stopgap measure but it will fail as soon as the spammers catch on.
On the other hand, spammers might catch on to the idea that "these people are likely to complain, so I don't want to mail them anyways." That would be a Very Good Thing.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Heuristic analysis - detects and blocks spam by various email characteristics
Black lists - checks if the sending server is in RBL (Realtime Blackhole List), dial-up or open-relay servers
DNS verification - checks if the sender is using a valid mail server
Keyword blocking - blocks spam according to keywords in subject and body
Anti-spoofing - blocks email masquerading as coming from within the organization - a common spam technique
Cookies/web beacons - blocks email cookies which help spammers identify the recipient as a "live" email
Header verifier - inspects various header signatures and blocks spam
Textual analysis - categorizes spam according to textual content like mortgages, pornography, dental care, etc
Spam signatures - an auto-updating spam database allows detection and blocking of spam according to smart signatures
Spam URL filtering - blocks email with links to spam sources and sponsors
Spam image filtering - blocks email containing spam associated images
Auto-updating database - local or remote spam blocking database based on thousands of Spam collecting bots and web crawlers
http://www.esafe.com/esafe/anti-spam.aspeSafe
...25,000 extra spam for that one legit email? :P
"People" using "unnecessary" quotes should be "shot".
I run Gentoo both at work and on my xbox, did my first Debian install via ftp over a 14.4 modem years ago.
How long is a modem year?
Not everything is analogous to cars. Car analogies rarely work.
"I think this is the point of contention - either they are talking bullshit about it being a 'firewall' or they are talking bullshit about being the 'first'."
And since it must be one or the other, then why trust ANYTHING in that press release.
A real "spam firewall" would be able to drop connections from spam sources instead of receiving all the messages from them and processing them. Now THAT would be revolutionary (provided it worked correctly and wasn't completely vulnerable to spoofing/DoS).
Here's a hypothetical:
1) I get a spam "from" you and forward it to you with a note saying "did you send this." You want to get this type of email. Since you might get such a message from anyone at any time, traditional "is he in my mailing list" filters aren't suitable.
2) I'm a spammer and malware writer, and I write a virus that sends mail from my victim's machine that looks identical to #1. Even though the message is malware-free, you definately do NOT want this message.
No human recipient can tell the two apart, by looking ONLY at the received email.
Of course, no computer can identify "friend or foe" by simply looking at the message either.
So, if you are looking for the perfect filter, it doesn't exist.
If you are looking for a filter that's better than a person, I recommend Yahoo for web-based mail and a number of good solutions for your own system.
In the above scenario, there are solutions. One requires analyzing multiple copies of the message to spot patterns, something big houses like AOL and Yahoo can do but small shops that may only get 1 copy of the message cannot. You can also use RBL lists that track zombied machines, but that won't trigger if the machine in question isn't RBL'd yet. Delay-try-again-later tactics like those mentioned elsewhere in this thread can help here, but are ruinous if you want legitimate complaints ASAP. "Man in the loop" solutions like sending a confirmation message might help, but many people ignore such requests.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
"UQ's main commercialisation company, UniQuest, has formed a start-up company based on the technology and is seeking investment to take the spam firewall to market"
/. go quick and buy it and be rich fast!
Damn and they managed to spam
Anyone running relay blacklists (IPlists, spamcop, spamhaus, etc.) has been running a "spam firewall" for years. It's a very effective way to stop spam, but it's nothing new nor revolutionary.
A corporation will not accept even a very LOW risk of a false positive because that could be the million dollar email.
Nick Powers
Encryption: I may not agree with what you say, but I will defend your right to encrypt it...
script: :(
cat dict.dat | grep "$x"
where $x is ur mail body.
that should take care except of the nouns
Yes:
Almost forgot:
4.
5. Profit
Eh, never mind...
A false-negative is very bad (calling email from my business partner spam), but a false-positive is alright once in a while. I can take, at most, 2 V1Agr4 emails a day. This seems like it would just be a matter of tweaking the engine.
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird. --Nietzsche
it is this: I have a technology that has a 99.9999999% hit rate but requires someone to sit at the console of the spam filter and manually release false positives, manually train the filter to false negatives, manually whitelist incoming emails based on what people are sending outbound, manually authenticate inbound emails by calling the sending party on the phone, and manually update the RBL.
We're losing the battle, you know.
=^..^= all your rodent are belong to us
The parent poster enjoys sucking cock. However, his post should also be modded informative, not offtopic.
Don't these devices already exists? What makes this one so special?
Mistake.
Directions for doing this with amavisd here: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC
Was at the smtp level too.
Eval of the classification system is here:
http://reeltwo.com/products.html
At one point in time I was only getting 60% of my postal mail. If you complain, your local postmaster will tell you he'll look into it and promise to call you back. When he fails to call you back, you can get ahold of the regional customer service number, promise to look into it and call you back. They of course will not call you back. Then of course you will call their national customer service number. They will open a trouble ticket, promise to call you back, but won't. When you call back later, they will inform you that your ticket will have been mysteriously closed without reason.
The United States Postal Service has the worst customer service of any company I've ever dealt with as well as an extremely high failure rate. Email is a far better way to communicate, even if some messages never make it to you.
www.mxlogic.com
www.surbl.org nuff said?
I wasn't too encouraged when I read the physorg.com spam filter story, noticing that two of their top 5 news stories in the sidebar were "Researchers say Tunguska Event was an UFO Crash: Debris of Alien Spaceship found", and "Tunguska Event: New Details and Sensational Theory". Too bad the links pointed to subscriber-only pages. Has The Enquirer spun off science/tech reporting to a new site?
Luke, help me take this mask off
Any incoming email that spamassassin detects as spam I record the IP for. If that IP has more than 2 infractions in a given amount of time I execute an ssh command to add an iptables rule to my firewall to block that IP. Problem solved.
Still, even one in 25000 is too much. It means that this software cannot be trusted to fully automatically classify my email, and so I will still have to babysit it, wasting time.
Also, regarding: "the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails", how the hell does a HUMAN misclassify spam as legitimate? Well, I suppose that after manually processing 25000 emails in a row, you're bound to slip...
So... mail is stopped at this 'firewall' and only valid mail is allowed through... sounds good but how exactly is that different from an mta with spam filter?
I was actually wondering what to do with http://www.virtual-girlfriends.co.uk.....
seriously. spamassassin has had such feature for years.
They're definitely not the first to use SVM for blocking spam, unless they've been sitting on this thing while developing a full blown set of products and companies to market them... ProofPoint has had this in their spam appliances for a while, and Aladdin has been using it in their eSafe service since at least their press release in June of 2003...
Why don't people listen to me!!!? I keep saying that we need to have a DNS like service for the following applications:
1. Spell checking dictionary (multiple languages witha "root server" for each dictionary)
2. Application to data type association (cross platform and non-profit so that all applications and file types could be included)
But do they listen? No.
Un-news
Until there is something to evaluate or, less exciting, purchase, it's all just vaporware. Where is the open source distribution?
TallGreen CMS hosting
You said yourself that they aren't doing anything unique. Other solutions employ connection blocking as well. Their claim is to process a certain number of messages per second/hour/day/whatever, not connections. There is no possible way you can process as many messages as they say, using SpamAssassin and a bunch of PERL plug-ins.
Compare all the stuff you mentioned as what Barracuda does, vs. what the widely acknnowledged "fastest" SMTP appliance does... IronPort (which, by the way is a competitor to the company I work for) just has a totally speed optomized MTA with a little speedbump of a Brightmail filter (which is itself fairly fast, but then IronPort took out a bunch of Brightmail's filter to make it even faster).
Now an IronPort box will send out about 600,000 messages per hour in spam cannon mode. Depending on the model, and the creative license taken by their sales rep, they claim to process between 100,000 to 300,000 messages per hour inbound. Keep in mind this is with an MTA written strictly for speed and one of the faster spam filters (certainly it runs rings around SpamAssassin, I've seen both in action).
Flat out, Barracuda are lying when they say an individual box can handle n million messages per day. Fantastic claims like 115 messages per second are absolutely ludicrous. That works out to 414,000 messages per hour. You'd be lucky to get a Beowolf cluster of ____ to process that many messages per hour through a SpamAssassin filter (or for that matter, anything written in PERL).
As for doing it better, cheaper, etc probably 5-10 other companies do spam-blocking better than Barracuda, and most of them have the same degree of maintanence (some significantly less).
As for cheaper, there's no way any company can do it that cheaply, including Barracuda. I gaurantee they're taking a big loss on what they're shipping today, but they don't have a real company's business model, they're trying to get acquired. Barracuda are only aiming for market share, that's it. If they had to feed themselves by their sales rather than their funding, they would starve.
The other competitors aren't charging more because they're price-gouging, they're charging that much because they need to in order to sustain a business (in fact some of them, like IronPort are actually burning their money very quickly).
Using Open Source doesn't magically mean they have zero overhead. All the other companies selling e-mail security appliances use a substantial amount of Open Source code, although most of them aren't dumb enough to use SpamAssassin (with the exception of McAfee and Sophos).
That's OK, though. The Slashdot army can continue to delude themselves into thinking that Open Source automatically means software so cheap that any individual person can buy enough software to support a large enterprise. I wonder who Joe will work for that will pay him that much if all the software is free, though?
Someone is WRONG on the Internet!
The firewall I use does exactly what this company is claiming their new product does. I've been running it for years. It's Open Source to boot. It's called messagewall, and I think it's great. My (other) mail server receives between 100 and 700 spams a day, out of which I actually receive 1 or 2. I like it because it rejects the mail if it is spam before the sending server can actually send it.
The down side, you have to load, compile, and build it. It's not too bad, even for a non programmer like me.
CC
100 Megs is roughly equivalent to 1 modem year... 8*)
(Yes - I DID download a 100 MB file over a 56K modem a few years back - took about a week with download resume).
In this particular case, since you know that all your critical emails show up with a certain type of file attached, you'd just tell the filter never to ditch anything with that attachment type. Presto, no false positives on the important stuff. (Theoretically you'd start getting false negatives too, but not many; how often does a spam contain a PDF attachment?)
More generally, you'll want to make sure your clients know email is unreliable. No matter what precautions you take, there is ALWAYS a positive chance that some server error causes the message to be lost. Tell your clients that if they don't receive a human acknowledgement within N business days, they need to assume the mail never arrived.
If these "very wealthy individuals" insisted on sending you important legal documents by USPS without proof of delivery, you'd think they were nuts. Why should email be considered any different?
Let's configure all SMTP servers to drop mis-spelled email. Then not merely will we have ended the scourge of spam, but also cleared the internet of dumb people. This is not a bug!
You should re-run your study, and correlate against average IQ before and after...
Hi All,
/.'d, but I will say a few words about it (I'm Matthew Sullivan if you hadn't guessed).
;-)
Well I'm surprised to see this
What makes our firewall a firewall is a fundamental differnce in the way we handle the mail. Products such as the Baracuda Spam Firewall are filters - they accept the mail and looks for spam to reject. Our software looks for real mail to accept and rejects everything else - the difference being real mail does not find new ways to get around filters, spam is continually doing it.
Currently the software is ALPHA code, and the test results are from that. The one FP it had was predicable and will be solved before going to beta.
Now flame all you want
I will not likely be able to read comments, so if you want to talk to me mail me via the form on: http://www.dnsbl.sorbs.net/
Thanks
Matthew
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through,"
In other words it's a filter. Sheesh.
Filtering is not fighting spam, it is an accomodation with spam.
So no these researchers did not make the company per se.
Actually I like the idea quite a lot: what is SVM? support vector machine. :)
What is support vector (SV): just an instance of the class which happened to be valuable in definition of the separating boundary between spam and non-spam. This reduces amount of 'valuable' information a lot. On other hand you achive good generalization on future classifications. Idea is simple and straightforward
The premise of this article is entirely false and not well researched. There are commericial vendors already offering spam firewalls and leveraging SVM (Support Vector Machine) algorithims. A simple search turns up the likes of Proofpoint (http://www.proofpoint.com).
There will almost certainly be too many individual cases for this to work.
its been written by University coders - will the work be marketted and sold or will it be released as OS for the common good of all mankind?
How long is a modem year?
A very very very long time. Long enough that I waited a month before deciding to install X because I knew my dialup provider was going to crap out while downloading the packages.
It allows the user to select the level of filtering desired. All email containing content unwanted by the user is treated as spam. At SpamByte code 0 (which is displayed along with your email address and a notice that 'all email content containing unwanted email will be summarily deleted or reported as spam'), the only spam that gets through will look something like this....
It is spam that got past my program's filtering routine but is inconvenient for the user to use. Because it is written like this to evade the filters (in spite of the email sender warning above), the sender must be a spammer and the message can be reported as spam and deleted without further thought. Once this task becomes overwhelming, 'close' your inbox for a while then 'reopen' it later--Let the spammers deal with the bounces of 'unavailable' mailboxes.
And anyway, the one misclassified message mentioned in the article could have been a real email treated as spam, unacceptiable performance in a business or otherwise mission-critical environment.
It was press released on 2004-08-16, one week before the 2004-08-23 date in the article
An earlier version of one of the software programs using a different (now discarded) approach was submitted as a Slashdot story but was rejected.