Slashdot Mirror
Microsoft Researching Anti-Spam Technique
Tim C writes "Microsoft's Research group are working on a technique to combat spam. Dubbed the 'Penny Black project', it involves making email senders perform a computation taking around 10 seconds, which their recipients can then check for. This delay would limit bulk emailing speeds to around 8000 a day, meaning that to spam all of those 'fresh, guaranteed 25 million addresses' would take approximately 8.5 years." We've reported on this before.
How do you "make" senders do anything?
"Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it." -- Linus Torvalds
Well actually yeah they did. At Crypto'03 a method for memory bound HC was presented.
So while MSFT didn't invent the original HashCash concept MSFT did improve upon it. So before anyone gets the bright idea of flaming MSFT ignorantly.... know your facts!
Tom
Someday, I'll have a real sig.
Typical. Delay the time it takes to send an email to make email less profitable. Ever notice that whenever Microsoft says, "1 minute remaining" you end up waiting for about three?
This is not a solution... as *I* still have to check for something on my end, and then discard if that condition is not met... my bandwidth and time are still wasted.
---
Programming is like sex... Make one mistake and support it the rest of your life.
Comment removed based on user account deletion
Problem is, if it takes 10 seconds on a modern computer, it takes three minutes for Aunt Edna to send you photos of her dog, and a distributed spamming network will still churn out spam. I think real cash is the only cost that makes sense if you want to go that route.
Even today, the most annoying spammers are not using their own computers, but insteady they are bouncing e-mail off virus infected and trojaned PCs.
So 8,000 emails / day is fine, if you have a couple thousands relays to pick from.
---- join dshield.org Distributed Intrusion Detec
Count on Microsoft's "cure" to be worse than the disease itself. You would think for $40 billion they could buy just a little more intelligence than that.
SMTP needs to be redesigned. Not by Microsoft, who will use any change in the protocol to tighten their monopoly grip, locking in their customers (and locking out the non-Microsoft world), but by the IETF.
Spammers having to do a computation before delivering email isn't going to limit them to 8000 pieces of mail a day, it simply means they're going to cluster all of those Windoze boxes their custom worms have infected, and let those millions of PCs do the work for them in parallel. SPAM won't decrease one bit, but the load and toll it places on those who use the net will go up significantly.
The solution isn't to increase the cost of email (computationally, bandwidth-wise, or financial), the solution is to repair the design flaws in SMTP (and, for that matter, USENET, something that remains the most useful medium on the 'net despite its widespread abuse) that make SPAM a viable methodology.
The Future of Human Evolution: Autonomy
Making e-mails "expensive" to send is stupid. There are many ways to fight spam effectively without doing that.
We could start by adding sender e-mail address verification to smtp - the recipient looks up the e-mail address's MX record, and asks if that specific e-mail was sent from that mail server. If not, it's probably spam.
The more server that implement this scheme, the more points will be given to those e-mails (by spamassassin etc.) that do not have this sender verification set up. Within a year or two, all serious mail providers, companies etc. will have sender address verification.
Combined with law enforcement, blacklists etc., this can become extremely effective.
Dybdahl
If this works as stated, then I can see issues.. For instance, large mailing lists. Would they have to be white-listed? 3000 seconds of computation is a heavy tax on a community based program like the Linux Kernel Mailing List, which averages 300 messages to my inbox a day. Also, there's the issue of viral spammers.. Those that send out viruses to do the spamming for them. If you infect enough, 8000 mails per day per computer can still be quite a bit.
Personally, my whole take on spam is that everything needs to be done on the user end. Laws have loopholes in every situation (foreign spammers being a large one,) server restrictions are either too restrictive on small servers, or can be defeated with distributed computing.. I say we stick with Bayesian filtering. It works _wonders_ for me, and I'd love to see more people use it.
This statement is false.
I don't want spammers to pay to have the right to send spam... I want them to stop sending spam!!
I seriously don't think this will work as (a) spammer won't use Microsoft products to send their wares or (b) because they will find a way to crack the security of this system (I mean, come on, this is Microsoft we are talking about here!).
The right to offend is far more important than the right not to be offended. (Rowan Atkinson)
Microsoft Research is no different from other industrial research labs: IBM, Bell Labs, etc. They hire the same kinds of people and get the same kinds of inventions out of them. One can't expect any more or less from any big company with a lot of money to spend. However, so far, MSR has not had much positive impact when it comes to driving innovation into the marketplace.
If Penny Black is all there is, it doesn't look like that's going to change. It will probably be decades before we know whether MSR will have had lasting impact. By that time, Microsoft will probably be a benign, lumbering giant, just like its monopolistic predecessors, AT&T and IBM.
If it takes a long time to send out bulk email, what about all the mailinglists people subscribe to? How would lkml or sourceforge lists continue to operate?
I am a viral sig. Please help me spread.
It's an attack on Open Source development. If SourceForge was limited to that few emails a day it would kill many projects run by mailing lists. Worse, think about LKML - it would take years for the latest BK patches to be distributed via email. Wait, maybe this is Larry McVoy's subterfuge and not Microsoft's...or they're in cahoots...after all, they're both on the dark side (i.e., non-open or closed) of the source.
I was just wondering (and I hate to play the Devil's Advocate but ....) what it would take to spawn multiple independent processes on one computer each running its own email client ... I know something like this should be easy with *nix ...
The nub of using memory is that it is question of "time." You can't fit "generated time" serially as the day is only 24 hours, but you can fit the "generated time" by putting it in parallel to fit within 24 hours with multiple processes ... and the parallel processes ONLY have to run the lightweight email client and nothing much else.
To see a world in a grain of sand, and then to step back and see the beach where the sand lies
This seems to be a "let's fix this by limiting what technology can do" case.
Instead, they should focus on adding more functionality to the smtp protocol. For instance, they could add sender e-mail address verification. You can't check the actual e-mail address, but you can make a "dial-back" TCP connection to check, if the e-mail is known by the mail-server that belongs to the sender e-mail address.
Combined with law enforcement, blacklists etc., this is extremely effective.
So this would have the effect of making legitimate high-volume, high-subscribership mailing lists expensive to operate (unless subscribers configured their MTAs to accept "unstamped" messages from the list, which is annoying and error-prone -- and has an obvious "workaround" for the spammers).
<tinfoilhat mode="on">Ha! Now we see Microsoft's *real* goal... to slow Linux development by shutting down the kernel mailing list!</tinfoilhat>
Seriously, though, any attempt to make e-mail expensive hampers those who have a legitimate need to send lots of e-mail.
Plus, there are obvious workarounds that will be developed in short order. A hardware stamp-generator could probably cut the stamp generation time to practically nothing, particularly since their approach somehow depends on memory/CPU latencies rather than processing time. You might be able to make a much faster stamp generator by running it on your graphics card, and custom-built hardware could certainly do it.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Before you chuck the entire protocol, do you have a solution for a better one?
Until you know how you're going to repair the problem, let's not get too excited about scrapping a protocol that still has a lot of flexibility. I've learned a lot about SMTP in the last few months, if there was universal agreeement as to WHAT to do, we could probably accomplish it in place.
What are the options? Whitelists, blacklists, red lists, gray lists, hash cash, filters, etc. No one can agree HOW to combat the problem. A new protocol would accomplish nothing without a planned solution that makes palpable the limitations of SMTP. Til then, let's not get hasty about blowing it off.
The programmer who works next to me used to be a construction worker. Every so often, I come up for an idea for some kind of home project, explain it to him, and he tells me a way to accomplish it that is much simpler and more reliable.
This MS solution is almost a caricature of one of my own over-done home improvement ideas. Why bother with some elaborate cryptographic system to delay inbound emails? Why not just have the receiving SMTP process call sleep(10) at the beginning of the SMTP session? You get the same desired slowdown, and all you have to change is the SMTP server software. There's no need to modify MTAs, promulgate new standards, or fit yourself more tightly into the MS monopoly noose.
Proud member of the Weirdo-American community.
Something that the Redmond Empire conveniently neglects to mention is that an awful lot of the spam is due to virus-compromised systems running -- you guessed it -- Microsoft Windows! I've lost count of the number of broadband IP ranges, notably from Shaw Cable and Comcast, that I've had to dump into our domain's local 'Reject' list thanks to their endless attempts to propagate Swen, SoBig, or whatever the latest spammer-zombie trojan is.
Perhaps, if Steve 'Uncle Fester' Ballmer and his cronies had paid more attention to basic security to begin with, or had taken the trouble to actually try and educate their customers about the most basic computing security steps, there wouldn't be such a huge problem now.
This 'Penny Black' nonsense looks like nothing more than a means for them to make money off a mess that they created in the first place.
Bruce Lane, KC7GR,
Blue Feather Technologies
This delay would limit bulk emailing speeds to around 8000 a day, meaning that to spam all of those 'fresh, guaranteed 25 million addresses' would take approximately 8.5 years. ;-)
Yeah, because they did not hear of parallel processing yet
ato
I guess we could combine this with distributed computing so if you send out an e-mail you are helping solving one of the puzzles like for example RC5, OGR or ECC2. And make the world better.
But I think microsoft is intending to create a complete new business model for e-mail providers (and ofcourse for microsoft's hotmail.com) by selling the computing power to companies who need it.
I actively subscribe to a lot of tech sites that have tens of thousands of subscribers. Slashdot is one of those sites. How many people have Slashdot e-mail their mail to them? How are legitimate bulk mailers (of their own content, not ads) supposed to send out newsletters, etc.)? If a retail outlet with a legitimate opt-in newsletter needs to send it to 50,000 or 100,000 people, what kind of hardware upgrades are they going to be looking at. I mean, I can add them to a trusted senders list on my side, but that doesn't tell them that they no longer have to run the computations. "If I don't know you, I have to prove to you that I have spent a little bit of time in resources to send you that e-mail. How do you know whether you "know" me or not? Does the user's mail client alert the sending server that it approves of mail from that SMTP server? Once senders have proved they have solved the required "puzzle", they can be added to a "safe list" of senders. Whose list? My personal list that is part of my mail client? My mail service's white list? Microsoft's special white list?
If you mod me down, I shall become less powerful than you could possibly imagine.
Having read the article, I was impressed by how clever their proposed solution was, though since I don't have a CS background, I don't understand how a mathematical computation can be essentially bottlenecked by memory latency -- I'd love it if someone could give an explanation of how that works.I'm guessing that some cryptographic hash needs to be held in memory, such that the nature of the data structure and physical access to it proves a bottleneck. This is probably way off.
But having read the /. comments, it becomes clearer to me that this solution, and many other proposed solutions face problems insofar as they "break" the assumed contract under which email has worked for so many years. To me, this seems to boil down to a challenge / response system (allbeit one that increases the overhead of the transaction signifigantly). The problem with these systems is that for a time, email will be broken for certain people, or broken when trying to communicate with certain people depending on whether or not one has migrated to the proposed system. I'd worry that this would have the effect of segmenting email users into little fiefdoms determined by which email system they are using.
I don't think a migration can happen unless there is some "benevolent dictator" who can force everyone to migrate to such-and-such a new email model and system, and frankly, I wouldn't want that forced on us.
It seems that the challenge to any such spam-reduction system is that migration must be immediate and non-backwards-compatible, and universal, otherwise for a time email users will be segmented into little fiefdoms based on whether they've migrated, and solution to which they've migrated.
All one has to do is hit the right keys at the right time and the instrument plays itself. - Johann Sebastian Bach
> The email is sent and the server runs it through
...their email would go to someone else's
...and they would just trash it...
> the scoring process. If the message scores more
> than 6/10 the server sends the sender an
> authentication message, asking to validate the
> email.
So you are one of those resposible for bomabarding me with those damn things.
> This would require spammers to manually
> intervene and waste tons of their time. if they
> forged the sender email...
They always do. My domain is a favorite.
>
> email...
Yes. Mine.
>
Isn't that what the spammers say? "If you don't want it, just delete it. What's the big deal?"
The big deal is that about a quarter of my email is bogus bounces and useless "confirmation" message from systems such as yours.
_NEVER_ _REPLY_ _TO_ _SPAM_
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
It's ironic that your complaint about worst-case users and grandmas is tied to mention of industry.
Anything that produces an end product for a userbase must adapt to suit the needs of that userbase at the time that the product is being produced. If the end user is so egregiously stupid that they can't even handle e-mail without someone holding their hand, then rather than evolving toward the next great technological advance, usability must be made the next branch for improvement.
Think about it in relation to industry once. If automakers had blazed trails toward the next great evolution in automobiles, we could have cars that run a 1/4 mile in 4 seconds at nearly 200mph. Oh wait! We do! They're called funny cars! And nobody except a particular niche knows how to use and maintain them, and they're exceptionally dangerous machines. They are not refined for the general public, they are not safe, and when something goes wrong, it's often disastrous. Neutered cars like Corvettes and, for a few adventurous souls, Vipers, are fed to the public as top-of-the-line even though they're not. They're safe, (relatively) easy to use, and, for the most part, attractive to the buying public because, even if they break down it's just an inconvenience, they don't generally erupt into a fireball the size of a small house.
The computer industry will continue to evolve in much the same way. Crippled, blighted, and weak but generally consumer friendly software will drive the marketplace. In the meantime, hobbyists (Vipers and backyard mechanics) and hardcore computer geeks (funny cars and track techs) will continue to use the cutting edge workhorses that are far less refined, but far more advanced.
Alito: A vote for Alito is a punch in the eye to put that bitch back in her place!
So the solution is for spammers to set up compute farms of cheap old hardware with an open soure version of the mailer. Since memory latency matters, and not processor speed, the solution is to have access to more than one computer. A farm of 10 machines then sends out 80,000 messages a day. A real super computer farm funded by a spammer alliance could get back to shipping millions of spam messages a day. What was the cheapest supercomputer cluster mentioned on Slashdot, something like $30,000? Is that really all that much money when you consider that a group of spammers could split that and amortize over many years? Remember, age of the hardware is not a consideration, just CPUs with access to memory segments. How about a very large system with hundreds of virtual 386 processes running 128k memory segments?
I think in the long run only something more expensive will deter most spam, but will not succeed completely. Case in point is all the junk mail we still get in our real mailbox. Someone out there is paying for postage to send that crap, yet they still ship it to me so that I can place it in my trash can.
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
Instead of hitting the delete button I started putting spam in a folder for later analysis. What I found is that spammers use affiliate programs. For example, I recently got a porn spam with an image from
://www.silverstate.co.sy@click.com-click.com.ph/cl ick.php?id=sicosyl
1 &c ampid=
http://gallery7.withsex.com/
All I do is block withsex.com with an expression filter and all spam that's afilitated with that site goes away. Spammers can't ofuscate an URL otherwise it won't work. The image linked from the same site is 28KB. If that spam was sent out to 25 million people and all of them looked at it once that cost the spammers 667GB of transfer. On a standard DSL line it would take about 6 months to transfer that. These companies need a dedicated host to allow them that kind of bandwidth. The company may have a number of domains for the site but spammers aren't going to be using random ones to advertise it like they use random from e-mail addresses. They also have to keep the domains functional or all that spam goes to waste.
Not many hosts would allow that kind of bandwidth transfer without charging up the nose for it. Which limits the number of hosts that spammers will use for images. 2004Hosting.org/.net is a big one for the cable filter and "banned CD." 530000x.net is also affiliated with those spams.
http
click-net and click-com are what spammers use to get paid. If you click on a spam link, most likely it goes through a common domain to log the referal to calculate how much the spammer gets paid. Block the referal site and all spam that uses that referer to get paid is gone.
For example
http://www.xswcde.biz/index.php?id=173&affid=56
342
Is a big e-bay spammer site. I block xswcde.biz with an expression filter and all e-bay spam from that company goes away.
It basically boils down to blocking the company and not the spammer. My spam count went from about a dozen a day to 1 or 2 and they also have obvious tells. If possible I also block the domain in the from address. Using a web-form cut down on spam quite a bit as well.
Ben
Work Safe Porn
No, if it takes 10 seconds for a spammer with the latest dual Xeon CPU (or hacked into a superfast company computer), it will take several minutes for the average user, and hours for my mother on her old P200 (which is more than good enough for sending email), or days for myself on my 20MHz PDA.
Of course, this will incite people to buy new PC's, which comes with a new operating system, made by guess who?
Nah, I'm not cynical. It's probably worse.
Regards,
--
*Art
The only problem with your analogy is the fact that you don't have to drive a viper. This scheme would mean that you do.
I don't think this is a good idea.
First, it would kill legitimate mailing lists. Imagine what the perl5-porters list or the Linux kernel list or any of the other high traffic mailing lists would have to do to keep operational. Large mailing lists already have problems with lag. This would just add to that.
Also, there does not seem to be anything that would stop them from doing these operations in background and just contact multiple sites while working on the problem. They would just multi-thread the mail spammer or just hijack more machines to use as their slaves.
This technique requires replacing every mail program out there to support the protocol. Of course, they will just make it a condition to connect to exchange. Might be a way of getting people away from having to talk to compromised Windows mail servers.
This is a bad solution for a big problem.
"Something must be done! This is something, therefore we must do it!"
"Trademarks are the heraldry of the new feudalism."
M$ should be spending the time and money preventing their mail servers from becoming compromised and finding ways for its desktops to not get so easily owned and that would prevent the majority of spam that comes to my systems.
/. Bill should pay for tech support if he wants to own the code.
This "spam filter" stuff when performed by M$ is an insult when it does little to address the problem which it has a contributed to.
---
Please stop discussing M$ fixes on