Slashdot Mirror
Microsoft Researchers on Stopping Spam
TheBackBencher writes "Scientific American today has a very interesting article about "Stopping
Spam" by Joshua
Goodman, David
Hackerman and Robert Rounthwaite from Microsoft Research. They talk about different types of spam -- spam with emails, spam on IMs, spamlinks
on web pages and image based spam. They mention different techniques for
spam filtering mainly fingerprinting matching techniques, n grams model,
naive bayesian approach, optical character recognition, challenge/response systems and Human Interacted Proofs (HIP) in a very lucid style. They however do not mention fingerprinting approach of using Nilsimsa Hash to
tackle addition of random words by spammers in emails or hypertextus interruptus technique used
by spammers of splitting words using HTML comments, pairs of zero width tags,
or bogus tags. Also, Spam-Research is reporting the
SplitFit
Technique that Spammers are using to fool Yahoo! Mail SpamGuard."
Creating your own spamming division, use illegal tactics to undercut your spamming competition, put them out of business, then stop spamming.
If it was developed it can be reversed engineered. Sorry to say but spam is here to stay unless of course someday the internet becomes regulated somehow.
Spam is like porn: hard to define but you know what it when you see it. That can be hard to program I would think. But, who knows.
http://www.busyweather.com/
The ebay.com link showed up at the bottom of the browser, but was replaced with some kind of javascript mouseon event. This is probably not new.
Instead of random text to fool Bayesian filters, it had hidden recent news article summaries (bracketed by html comment tags) that would be similar to what you might post to a friend.
Spam filters will probably be upgraded to catch this soon, but it was the first time I had seen it. And of course as mentioned in the article, the ebay specifics where obfuscated by html tags between letters.
Letter To Iran
Of course, one 200MB update from Microsoft would kill this idea. Or how about a 500MB game demo download? Thats legitimately free. Or better yet, what if I need to download a linux distro or a television episode?
I would hate to have to explain all my actions to my ISP. Espically with the way media is driving the internet nowadays. 200MB is way too small of a limit.
Now, you can monitor how many e-mails are sent by a host. That would be a better way. At least there could be a filter on the "to:" line. If that list includes over say, 1000+ users, consistantly, then at least there could be some flags raised.
give spammers a 9 year prison sentence.
That'd probably be the best thing M$ could do to help reduce spam.
Maybe you didn't quite understand what I was talking about.
This would be completely done server side. Just like when you sent an e-mail to a host, and you get returned mail because you somehow typed in the address improperly. There would be no difference between that message and one that was sent to a user and then flagged as spam. It would be impossible to tell the difference if the user was a valid address or not.
Thats what I am getting at.
I shudder to think on what you mean by a "bounce" feature. Most likely sending a "bounce" reply to the forged sender address? That's part of the problem, not the solution.
One line blog. I hear that they're called Twitters now.
So, does Microsoft Research plan on combating Spam with a Bob-like approach, or the more refined Clippy approach?
:).
Or are they going to come up with an entirely new file system to combat it, hype it up for every Windows release, but then delay its release a few more years?
Oops, pardon me while I reminisce about all the great advances Microsoft Research has given me
my blog
Interesting idea, however invalid address responses are sent within 5 minutes of the original mail. If the response is sent over a day after the original mail is sent, the spammer could just discard it.
If we respond instantly to all email with invalid address mails, it will be overused and spammers will ignore ALL of them. This is much like antibiotics, we use them too much, the bacteria accommodates for it, and antibiotics become obsolete.
So far, bayesion filtering and/or whitelisting email are the only good systems of blocking spam at the moment.
Don't you mean, Microsoft Mergers & Acquisitions?
"Who says nothing is impossible? Some people do it every day!" - Alfred E. Neuman
1) It is a form of communication
all email is communication
2) The communication is unwanted
"wanted" is a subjective property of the recipient - the computer has no programmable decision procedure for wantedness.
3) The source of the communication is hidden
There may be some system of authenticating sender ID, and will be as easy as getting ppl to use pk encryption.
4) In recieving the communication, you use your bandwith or incur a cost
again a property of all emaiil.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
I don't undertsand this. On one hand, you have the police saying they can't track spammers. Spammers use drones, they remain hidden, they hide their tracks. On the other hand, if you unsubscribe, they know your email is a real one, and you get more spam. That tells me whoever runs the unsubscribe service is in cahoots with the spammer and is just as guilty. They have to know where to send their lists? Just track them as part of the war on spam.
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
Here's a more interesting idea...
Authenticate SMTP with public key signing. -- Then use a trust network to only accept email from trusted companies.
Why it won't work:
It involves effort and cost.
Baah, the internet should be unregulated, if they can get rid of SPAM then whats to stop them getting rid of porn, anti-government information etc. There's a road we all want to go down.
Don't buy it and Get over it(tm).
Also, Spam-Research is reporting the SplitFit Technique that Spammers are using to fool Yahoo! Mail SpamGuard."
How much credence should we put into an analysis from a guy who goes to the spammer's web site to unsubscribe?
I thought the name David Hackerman was a bit too good to be true, and it turns out it was. Following the link shows that his name is David Heckerman . Note to /. eds: please proofread your posts. It's not like they're very long...
Dera Blcraays Mbmeer, Thsi eamil was stne by the Barclays serevr to vreify yuor emial adsserd. You mtsu competel thsi pssecor by ccilking on the likn bewol and entireng in the smlal wiodnw yoru Braclays Membership nrebmu, passcedo and meelbarom word. Tsih is doen for yruo proteoitcn - buacese semo of our mrebmes no lonegr haev assecc to theri emlia adserdses and we muts virefy it. To vyfire yruo eiaml arddess and accses yruo bnak anuocct , cilck on the lnik bolew:"
That email is extremely difficult to filter out because the only 'real' words are no, of, our, and, etc. Simple words that occur so many times in legitimate emails that most spam filters practically ignore them. But I have to wonder.. who would actualy 'cilck on the lnik bolew' anyway? I hate to use the term 'you get what you deserve', but if you are naive enough to click the link, then the problem isn't your spam filter, it's you.
Plus, with the postal service, there are 1000's of laws in place. If I send you an offer through the mail designed to rip you off, that is a federal offense. You can't use the US Postal Service for illegal activities, if you do you get caught.
Remember the movie The Firm? They did not convict the lawyers for tax evasion or any other crime. They convicted them for mail fraud. And if you let the worst spammers know that each and every time they send a message that is spam, each instance will incur a penalty, that might stop them.
Rosco: "If brains were gunpowder, Enos couldn't blow his nose."
I assume you're being facetious. Why does stopping someone from sending you garbage equate to not being able to find something. That argument sounds like those nitwit "censorship" whiners who think Freedom of Speech means Freedom to be Heard and that all content must be available at all times in all places.
Anarchy never worked in the real world, how could it work in the electronic world?
You are in a maze of twisty little passages, all alike.
The article barely mentions economics, and only in terms of the real costs of email--which only shows how much room there is for a real economic model with real business, real email, and *NO* spam.
I really wish one of the major email players would offer an option for prepaid email. That would be an absolutely spam-proof system. It doesn't matter if the postage is two cents, the spammers can't afford it. Two cents against 50,000,000 spams turns out to be *REAL* money. Any email via that address would be at least some kind of real thing.
Freedom = (Meaningful - Coerced) Choice != (Speech | Beer^2), and sad sock puppets' bad mods avail them naught.
Here's my solution to the greater unwanted communication Anti-spam paper submitted to Conference on Email and Anti-Spam
Error: Id10t detected
Each year they will announce that This is the Year of No More Spam on the Desktop (of course this never happens).
Or they will invent a brilliant new way to stop spam but as it requires the user to recompile all their OS and apps every 3 days it never gets used.
Or they just tell the end users "Why dont YOU code some anti-spam software?"
Or they produce an anti-spam system but the user must install 3 desktops and window managers, requires a 10,000 line config file that must be written by hand, comes with either missing or misleading documentation depending on the version you download and randomly purges any non-free software from the hard drive.
Legally, this is promising. First, there's no free speech issue. Second, in most jurisdictions, it's illegal to operate an anonymous business. So most spammers are criminals. Third, laundering transactions through intermediaries is usually a crime, too.
The problem for law enforcement is that following the money is difficult. Additional technical support for that would be a big help.
A good starting point would be to get a credit card issuing bank to cooperate in a scheme where, when one of their credit cards is used, full transaction details, including the payee's full identity, are immediately returned to the cardholder, using encrypted E-mail or some other secure means. That would make "following the money" much easier. This only requires one cooperating bank. That bank's credit cards might become popular with heavy Internet users. Especially if this works for prepaid credit cards, so you can find out who's behind a web site by using some disposable credit card.
The next step is to crack down on "credit card intermediaries". Non-bank credit card intermediaries that handle spammer transactions should be stuck with the legal liability of the spammer. Legally, they're the "merchant". They shouldn't be allowed to pass the buck to some other party. This will make "cheap merchant accounts" harder to get, which is probably a good thing.
Why not use a hidden markov model to filter spam that use random digits as filler?
A very basic filter will work this way:
Train a network of say, 30 to 40 units, with any english text. The training text doesn't just have to be limited to letters and numbers, it can include other ascii characters as well, because the hidden markov model will create distributions for them as well.
Now, for each new email that comes in, grab random chunks of text (maybe random 30-character strings) and see how probable the text would be in this hidden markov model. If it turns out not very likely, then scrap it.
Any thoughts?
First, I guess you didn't see the guy in VA who just got something like 9 years in jail.
That said, spam doesn't obey jurisdictional boundaries. Any single country can only solve a small part of the problem, and any spam incident often involves over 3 jusrisdictions that may be in separate countries (sender, spambot, recipient, etc). That's a logistical nightmare that isn't soluble outside of a dream world.
Also, force all ISP's to monitor how much bandwith a source has. If you get too much usage per day, say 200 megabytes or more, then that person has to explain why they need that much bandwith. If someone gets the RIAA on board, with their lobbyists, that should pass very quickly.
That's fantastic. Trade a bad problem for one that's much worse. Get the RIAA to legitimize their practices by using a guise of stopping spam? Let's not.
Also, force all email to have some element which identifies the source. Not just a header that can be forged, but something that can't be hacked.
Now by force, what do you do if they don't? Enforement issues again here.
Ultimately, legislative solutions for spam DO NOT and CAN NOT work for much but a small part of the problem. It's satistfying when some moron is clumsy enough to get caught (as with the guy in VA), but mostly these days the spammers aren't that stupid. Technological solutions work far better.
Is there really such a thing as a solution to spam? For every new technique that is developed, the spammers will find a way to circumvent it. Spam is a multi-million dollar business. I'd go so far as to say that it's a science. At least, the spammers seem to have it down to a science.
Trying to find a solution to spam is an idea in the eyes of experts and analysts. But to spammers, it's a road block that they must work around to stay in business.
Spamming techniques will no doubt end up as signatures in spam filters that are not unlike those signatures used by IDS and virus scanners. The experts don't seem to understand that if there's a will, there's a way. And the spam will just keep coming in another form or by some other technique. All that can be done is to keep up with changing techniques and patterns and treat spam for what is truly is -- an attack vector.
Well they weren't really a spam company, they sold software that allowed you to generate spam messages. I was going to do some telephone sales for them, cold call their market (I know, it's evil but I was calling corporations, not individuals, and I needed some cash) but after I got a copy of their software and became familiar with it's capabilities I felt icky, like I stepped in something, I couldn't in good conscience work for them. It had been presented to me as a customer contact software package - but it had too many little sneaky features that marked it to me as spam software, (built in SMTP server, throttle control on smtp activity so your ISP didn't get mad at you, and a bunch of message generation/tracking options) or at least there was nothing stopping customers from using it in that way, no matter how the company described their product.
The rock, the vulture, and the chain
Legislation ultimately runs into international borders and places where U.S. law cannot go. It can help, but honestly I am not sure how to craft a good law that will keep up with the pace of technology. Also, a law does not guarantee effective enforcement.
/. many times that if there were no money for spammers, there will be no spam. When spam becomes an issue which decides where money goes (who wins and who looses), the economics will take over. We need to convince people and businesses of simple ways to stop spam.
A better strategy, IMO, is to work on the commercial level. It has been said here on
Forcing monitoring is counter-productive. ISP's need to voluntarily enact monitoring schemes for their own benefit and that of other parties. When an ISP is convinced that they can contribute to stopping spam and that this is in their best interests, their efforts are more likely to be aimed at succeeding not simply complying.
Also consumers need to get involved, but not with lobbying Congress (on this particular matter). ISP's and webhosts need to believe that consumers will factor spam tolerance into their decisionmaking. Consumers (and other buyers) need to follow up and practice this a little - at least a vocal plurality.
On the community side, black-lists need to be scrapped in favor of informative lists of known, proven spam havens and spammers. What host's are the real problem? That is what buyers need to know. Block them if you want, but that is counter to how the Internet works and will not ultimately succeed. Instead, inform buyers who is responsible for letting spam through. Who should you not do business with? Do not be condescending or militant - be simple and clear. "So-and-so sends spam to your inbox."
I agree, technical work needs to be done. But beyond protocols, formats, and other standards this is problem which can be solved through small changes in behavior across many groups. It cannot be centralized and squashed.
I see all this time and money being invested into research to block spam. But we need to rethink our premises: does spam even need to be blocked? Is it actually a problem?
What you call "spam", I call "emails that help me learn about the latest products, websites, and business models". You want less of it? I want MORE of it. "Spam" keeps me informed about the world. And the fact is, consumers LIKE spam. Why do you think spam is profitable? Because people buy the products advertised! Studies show that 3 in 5 people who dislike "spam" have actually bought something online. So frankly, you need to be real careful about how you define "spam" because you could be targeting something you LIKE.
Between SpamAssassin, procmail, and MUA filtering rules, I rarely get to see spam anymore. The spam which does slip through is so absurd and surreal that it's more hilarious than annoying.
If everybody did this, the volume of spam would quickly dry up. Because when people don't see the spam, they can't respond to it, and when they don't respond to it, the spammer doesn't have a business.
Educate the people around you and help them reduce the spam that gets to their inbox. Don't support solutions which effectively render nodes at the network periphery to second-class status.
They blocked the block function for microsoft messages in hotmail.
Interesting idea, however invalid address responses are sent within 5 minutes of the original mail. If the response is sent over a day after the original mail is sent, the spammer could just discard it.
The thing is, I don't belive spammers ever remove an address due to an error. I had a domain that received a ton of spam, and that domain expired. Two years later (fighting with Network Solutions) I got the domain back, and immediately started receiving a ton of spam. Two years of spammers sending spam to invalid addresses (no DNS on the domain) and they still continued.
Why?
Simple: the spammers don't receive bounce messages, and the spam-servers (which could be static servers, or compromised zombie machines) don't provide accurate return information. Much like how telemarketers often show invalid or "Unknown" caller-ID info. It costs nearly nothing to send a spam message to an address, whether that address is valid or not. It costs much more to weed out invalid or unreachable addresses from your list by intercepting bounce messages etc.
And spammers don't give a shit. Most of the time, they are using someone else's machine (a zombie'd Windows box, or an open relay) so they don't need to care. So this trick simply doesn't work. It's cheaper to just continue sending to invalid addresses. Not to mention, many newbie spammers get their lists from less-than-legit sources who are selling large lists; they don't care (and are usually fully aware) that many of the addresses they are selling are bogus or no longer valid...
In short, simple tricks like this don't work, when dealing with an "industry" that doesn't give a shit...
NGWave - Fast Sound Editor for Windows
Legislate against spam. As long as spam is legal, or the penalties against it are too low, or it is too easy to do, people will continue to try and make a quick buck.
I don't see that helping. Legislate in what jurisdiction? In which countries can it be enforced? Note that one can simply lease servers in a country immune to such legislation, or outsource to a company in such a country.
Besides, FAX spam has been illegal for years, yet it continues to happen pretty constantly.
Also, force all ISP's to monitor how much bandwith a source has. If you get too much usage per day, say 200 megabytes or more, then that person has to explain why they need that much bandwith.
My DSL provider seems to have recently blocked port 25 outbound on me. Thanks to spammers I'm sure. So now I'm forced to use SBC's mail servers, or use a different port on my own servers.
Which is not fair at all. Neither would a bandwidth cap, when I'm paying for "unlimited" usage regardless of what port(s) the traffic may travel on.
Also, force all email to have some element which identifies the source. Not just a header that can be forged, but something that can't be hacked. And if a source can not be found, but it is selling a product an identifiable site, charge that site just as if they were the ones sending the spam.
I can deal with the first part of this: if everyone can agree on some authentication/validation standard, some verification can be good. As long as it doesn't cost the sending server operator anything other than the time taken to verify who they are.
The second part, though, won't fly. Forging the sender's address and/or IP is entirely too simple. And I've seend spam promoting a completely unaffiliated site, in the interest of getting a competing site shut down. In other words, send anonymous (forged headers) spam promoting your competitor, getting them shut down. Unless it can be proved beyond reasonable doubt that the company in question is in fact responsible for the spam, you can't convict or punish them...
NGWave - Fast Sound Editor for Windows
People don't send spam from their ISP's account.
Very true. They use a botnet.
They send it straight through their computer.
Not they don't. It's the easy to be on a RBL.
Now, you could put outbound filtering on port 25, and require everyone to send mail through the ISP's servers (with authenticated SMTP of some sort), though there will be some legitimate traffic surpressed if that happens...
The botnet is used to send just a few e-mails from each bot. Get an unfiltered inbox. Check the multiple copies of SPAM you get from diffrent senders. Check the headers. Identical SPAM arriving from many domains typicaly hit my inbox within a half hour of each other. This is the teltale sign of a botnet sending SPAM.
The truth shall set you free!
The overwhelming majority of spam filter deceiving techniques relies on HTML. If you block messages containing HTML on the mail server, the spam that gets through is near 100% identifiable as spam using bayesian filters.
So why on earth do people still use HTML in their email? Email should be plain text only anyway.
the macintosh asterisk mailing list http://www.astm
1. Most civilized countries are sick and tired of SPAM too. E.g., most European countries. So there is enough scope for a spam free zone, if the USA does want to get its act together and cooperate. It's not like you're alone against the world on the SPAM issue, except for the fact that:
2. It's mostly your spam that's dumped upon the rest of the world. USA is currently _the_ biggest source of spam, followed by... offshored operations paid for by someone from the USA.
So on one hand, the USA could halve the SPAM traffic on its own, without even needing much international cooperation, if it actually got its act together. And on the other hand, hey, there's a lot of incentive for a lot of other countries to cooperate. Just show us where to sign, if it means we'll stop getting your crap in our inboxes.
3. Once you have secured an EU+North America treaty on that issue, the rest of the world should IMHO be actually pretty easy.
We're talking some major combined economic power there. Any country who doesn't want to play ball with that kind of a behemoth can be whacked into submission in a variety of ways, ranging from economic sanctions to just disconnecting them from the Internet.
Makes that country unattractive to spammers too in the process. See, I don't think spammers want to target the local citizens of Elbonia with their operations. You disconnect them from the rich targets, you've killed that operation. So any country which thought it'll get rich by sheltering spammers, will quickly lose that investment too and be left with just the other disadvantages.
So I think they'll play ball.
4. But I don't think the spammers want to move to Elbonia or East Bumfuckistan and run their operation from there anyway. They might pay some local 5 bucks to run a server for them there, but they don't want to go live in a third world country. Those countries aren't that much fun.
You may see that even for legitimate operations, IBM might offshore their tech support to India or China, but you won't see the CEO of IBM moving there. (And those are already developping countries, not third world ones.)
A polar bear is a cartesian bear after a coordinate transform.