Slashdot Mirror
Yahoo and Unilateral Anti-Spam Technology?
EatenByAGrue asks: "According to this Business Week article, Yahoo is planning on distributing a toolkit for Sendmail and other mail daemons that adds an encrypted source domain key to email headers to verify where they came from. However, critics are concerned that the scheme will be easily bypassed and that it ignores standards bodies. What does the Slashdot community (representing countless email admins, I'm sure) think of this proposal? On one hand, its a commercial enterprise dictating standard technology, on the other hand, the standards bodies have proven themselves helpless and hopeless when it comes to providing solutions."
easy email tracking system will be gladly welcomed by police and other agencies...
This Is Not a Sig
I try to be as standards compliant with my mail servers as is humanly possible. Even with numerous spam filters, I get about 10 legitimate email messages a day and 100 spams. Something has got to change.
Whether it is this technology, or another, something has got to be done. I'll implement this and hope that other admins do the same.
-sirket
I think this is a good move on Yahoo!'s part. As a developer I think a solution that is available and 50% effective is better than a solution that no one has implemented yet.
Lets get the implementations out there in the wild and use the feedback to create real solutions!
These days I can't even open by inbox, it is so overflowing with spam. I'm exaggerating, but at some point email is going to become completely useless because of spam. I do a lot of business over telephone (the way I used to do it before email) and have an ftp site to which customers can copy shared files.
It's slower, but not as slow as deleted emails that I never see and can't respond to.
I have been pwned because my
It's important for standards organizations to be taken seriously if people want to actually see careful and appropriate change made. We could, I suppose, say that the W3C is completely useless because Microsoft essentially dictates what will and will not be a standard on the majority of platforms but that doesn't make the W3C any more useless. Actually, it makes it much more important to look for a body that can develop RFC's and such so that we can all look at the proposed solutions and say yes or no. When a corporation decides on something it just happens and all we have to fall upon to stop the adoption of a (potentially) damaging standard is the free market system. However, in this situation that wouldn't have much of a bearing on a system that doesn't technically bring Yahoo! any more revenue.
Web folk always moan about MSIE's poor standards complience, for instance, but forget that CSS/Text came from them -- Netscape was pushing CSS/JavaScript at the time. Now, one of those is a standard, and the other is dead.
Ultimitely, either people will like Yahoo's idea and adopt it and it will eventually become a new standard, or it will be ignored by everyone else and forgotten. Only time will tell.
The extra key could be used by anybody who wants to, and ignored by the rest. And their implementation is open-source, so it doesn't look like a way of making an end-run past other ISPs. And since many spam messages come from fake Yahoo email id's, this would be a great way to immediately filter out those ones: if it says Yahoo but doesn't carry a key-->SPAM bin
I like the idea of a major player getting on with it and DOING something.
Would we rather have MS dictating an anti-spam standard? You can be sure such a beast would be a lot less benign than Yahoo's proposal
"From" address from what your SMTP server is, in which case I don't see how it could work for you.
This may put a lot of travellers out in the cold.
A solution is badly needed, but it has to work for everybody.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
...de facto standards emerge. One need look no further than POSIX/SUS and GNU/Linux for an example.
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
Doesn't sound like this will be too effective in stopping spam for
Yahoo users, and Yahoo is already a pain
to work with.
I setup a proxy and was a spam relay (unknowingly of course) for just
under a week. I got blacklisted on a couple of email sites, my ISP
bitched and I fixed it. So sorry.
So I'm now off every blacklist I know of, and everyone loves me again.
That is except Yahoo, the evil nazi bastards. I've filled out their
stupid, "fill this out to get
un-blacklisted" form at least 30 times (twice a day normally).
It must go into a black hole because they still are rejecting my mail.
Everyone else lets me through but stupid Yahoo, who seem to have NO
admins, no technical people, and a violate once banned for life reject
policy. Grrr. So I guess, if this new system lets them drop their damn
overbearing blacklists, I'm all for it.
It would be much simpler to add a record type to DNS servers to identify **outgoing** mail servers. Email proxies, where 60% of all spam comes from, would be immediately eliminated. Spammers with fixed servers and addresses are easily taken care of by the RBLs. Why introduce something that is more complicated and less reliable?
Yes but we will never have a social solution when all it takes is 0.000002% of the worlds population to be spammers.
There's always going to be pricks who will do anything for a buck.
You can't expect to wield supreme executive power, just because some watery tart threw a sword at you
"...on the other hand, the standards bodies have proven themselves helpless and hopeless when it comes to providing solutions."
E-mail is supposed to do a certain job, and it does that job well, at least from a technical standpoint. The problems with spam are identical to similar problems in every other arena, it's just that they seem worse because of the level of automation. Even if it wasn't automated, spam would still be a problem. With idiots knocking on my door every other week with a hard sale for everything from oil changes to chinese food, I'm starting to almost regret the do-not-call list, because I didn't have to worry as much about these degenerates (if you don't take "No" for an answer and walk away immmediately, you are a degenerate in my book, and very door-to-door jerkwad so far has been one) giving my wife a hard time.
Standards bodies can't do anything to fix human behavior, unfortunately.
A far beter approach (which I think I saw on Slashdot but can't remember) is to use an extension which says whether IP addresses are allowed to use a domain.
This extension was based on DNS and basically allowed the mail server to query whether the IP address of the mailer was allowed to send on behalf of the domain.
Yes - this would be open to IP spoofing. Perhaps this DNS extension should be combined with the Yahoo method. If Yahoo, Hotmail and a couple of other providers adopted it could have massive effect.
To intially put live perhaps they could have an authenticated vs non-authenticated flag/filter in their web-mail client.
There were alot of vital ascpects to this point made in the previous article some of which are quite thought provoking!
If you missed the previous thread, I hgihly recommended reading or even reading it.
Never try to beat a professional at his own game!
I mostly disagree with the parent.
I agree that spam is a social problem, but you need to qualify what you mean a little more. Technology is the enabling mechanism to this problem (that some people are willing to be jerks and abuse a medium). Computers are exceedingly good at cranking out spam, day and night, and the medium of email is exceedingly weak against protecting against this kind of abuse. The same kind of social problem exists in all communications mediums, but you don't see just anyone wardialing people to sell viagra and penis pills. Calling a million people is expensive and time consuming, spamming is not. Therefore, this is a technologically exagerated (sp?) manifestation of a very minor social problem, making your point all but useless when trying to solve it. You've got to solve the problem in this situation, which is the enabler - technology.
$45 per U Colocation Special
That way, there's no question where the email came from, and exactly which account sent it. Plus traffic goes way down by not passing the content all over the place.
In addition millions of copies of the same email would not have to be held on recipient's servers, they would just sit on the originating server until received or until some time limit expired.
I guess this would prohibit using a (ISP's) email server as a repository, you would have to download everything you wanted to keep, but hey, no more email size limits! - send me the world - if I want it, I'll come and get it!
Could this help in the spam wars?
Development of a workable solution, that is.
There have been a few times in the past where an entrenched technology has hit a wall in functionality, but because it was entrenched no one really did anything about it.
Then, someone said "Fuck standards - I have to DO something about this!" and started pushing thier solution. Other saw that someone was willing to take the first step, and took a step themselves. After some shakeouts, a new, more functional standard emerged.
My hope is that Yahoo has started the "SPAM proof MTA" development war for real this time. I want my e-mail system back.
Soko
"Depression is merely anger without enthusiasm." - Anonymous
You mean like "reverse MX" records... google for RMX, SMTP+SPF, DRIP, DMX. (SPF seems to have momentum at the moment)
However, reverse-MX solutions will not kill off spam (a common mis-conception). The goal of reverse-MX proposals is to stop domain forgery where spammers are able to, with complete impunity, to tack on any old domain name to their spams. Which means that the unfortunate organization who is forged gets to deal with the thousands of e-mail bounces and the irate phone calls / e-mails from people who think that the organization was the source of the spam. As a mail admin, I'm able to control which servers handle inbound e-mail for my domain through specifying MX records. Reverse MX allows me to have the same amount of control over outbound e-mail from my domain.
What will happen instead, once reverse-MX systems (or Yahoo!'s system or other sender-authentication systems) come into play. Spammers will have to change tactics and resort to either forging one of the remaining domains that don't have reverse-MX information published, or they will register throw-away domains by the hundreds. It will drive up their costs a tiny bit (much like the impact of bayesian and other filters requiring them to use randomization techniques).
But the real nice side-effect of reverse-MX, etc., is that you'll be able to more reliable whitelist based on domain name. And your bayesian filters will be able to assign high ham values to domain names.
It also puts a crimp in e-mail worms that attempt to use a built-in SMTP engine to avoid detection. Unless the worm forges a domain with no reverse-MX info published, the worm won't spread (most MTAs will drop the connection). Instead, the worm will have to route through the user domain's SMTP server, where the mail admin is more likely to catch the traffic (virus scanner on the SMTP server, or rate limiters).
Wolde you bothe eate your cake, and have your cake?
Pain is a powerful motivator...
Reverse MX and Yahoo!'s proposal, however, don't require widespread adoption at the start. In fact, the tipping point is probably only a few percentage points of the domain namespace.
After all, for just a few minutes worth of work (more if you don't already provide SMTP AUTH, or require users to VPN in to send e-mail already), you protect your domain against joe jobs and forged e-mail bounces. So there's a low cost-of-entry. (Yahoo!'s proposal requires more work then the simpler, less CPU-intensive SPF proposal.)
What happens next is that domain admins that publish keys/SPF information find that they're no longer getting joe-jobbed and they're able to block a higher percentage of spam then they used to. Word gets out and more folks sign on (second wave adopters).
Sometime after that, the big ISPs require your mail servers to publish SPF/keys if you want your e-mail to be delivered to their users. (FYI, this is very similar to AOL's whitelisting program, which is essential a privately-administered reverse-MX system where you tell AOL what IPs your e-mail is allowed to originate from.)
As a WAG about rate of pickup, early adopters have started, second wave folks will probably sign on in the spring/summer, and I wouldn't be surprised to see ISP-blocking by the end of the year.
Wolde you bothe eate your cake, and have your cake?
This comment isn't insightful, it's stupid.
So if spam is a social problem, what about auto theft? Should that also be solved by economics and/or behavior? Do you think that people shouldn't lock their cars or have alarm systems? Or that they should have push-button starters with no key required? If you believe this, you're a fool.
How about hacking? Should that also be solved by economics and/or behavior? Should remotely-accessible computer systems not be password protected? Instead of having user accounts with passwords to keep hackers out, should we just let anyone log in who wants to, and use other means to punish people who abuse this? How about we connect our military systems to the internet in this way? Again, if you believe this, you're a fool.
Any time a technological measure can be employed to minimize a social problem, it should be, because relying on society to proactively halt the activities of those who prey on weaknesses in the society is foolhardy because society only acts in a reactionary manner.
Nothing new needs to be invented here. What we should all be pushing for is signed email. There are many advantages to signed email, but here are the most relevant:
(A) Signed email signs not just the message headers, but also the message body. No chance of header substitution.
(B) Signed email associates signatures with some certificate chain and, presumably, a CRL (Certificate Revocation List). Abuses can lead to certificates being revoked.
(C) Because of the certificate chain, there is a chain of trust. There is always SOMEONE to sue!
(D) It is a simple measure to simply throw out any email that is not signed.
(E) Because of esign legislation, signed emails can be considered legally binding. In other words, lies, misrepresentations, libel, etc... in signed emails provides you with grounds for prosecution in courts of law - as if the signer wrote you the document and signed his name at the bottom (and yes, they can also be used for legally binding contracts and whatnot).
There is an issue with "Crossing the chasm" with signed email, of course. It would require a body such as AOL and/or Yahoo rising up and providing signature filters on incoming email to force such a solution into the mainstream. But once this is done, SPAM will practically dissappear. And any SPAM that comes in through signed channels can be dealt with in a satisfactory way.
I do not believe this harms any of us, btw...
You want privacy? The same techniques that allow you to sign email also allows you to encrypt email to your destination.
Worried about anonymity? Certificates can be issued that authenticate an email address without full disclosure of the owner of that address (but this may not be satisfactory for stopping abuses). Anonymity and stopping SPAM may, unfortunately, be mutually exclusive goals.... Any thoughts?
The reason that it can be true that 1+1 > 2 is that very peculiar nonzero value of the + operator
Especially considering how promising the OSS model is, why can't we create a solution? We talk about the complexity of the problem, the importance of not breaking standards, etc. Who FUCKING cares if I can't check my email because it totally FUCKING BURIED in unsolicited junk...
I don't mean to come off as the thundering asshole, but this situation has grown so slowly its like watching a car crash spread out over the past 15 YEARS.
Please, experiment. Break things. I don't give a shit, but don't let us sit here moaning like helpless children while spammers sit back (laugh) and rake in MILLIONS.
Get fucking aggressive.
And if I hear one more idiot talk about how you have to cut spammers off by not buying their products I'm going to cut him off at the knees! If that would work you and Noah could be shooting dice right now and we'd have a hell of a lot less to worry about.
Programers still know how to experiment, right?
Quack, quack.
Anyone with experience with these standardization bodies knows that all of the complaining has to do with who's ideas win and who's name ends up on the standards documents. It's a particularly virulent form of academic arrogance. Solutions for signed email to stop SPAM are almost as old as email. Trust me, nothing is ever going to happen if one of the big guys doesn't put their ass on the line.
While the guys at the IETF fight for who has the biggest, ahem..., pen, the known email universe is collapsing under the weight of SPAM.
Let Yahoo hack and slash their way to a solution that works and then the standardization megalomaniacs can claim credit for inventing that idea 15 years ago while undergraduates at Stanford, Cambridge and MIT...
In the meantime, maybe we can have some peace...
The reason that it can be true that 1+1 > 2 is that very peculiar nonzero value of the + operator
But Spam is more about an inappropriate use of technology. SMTP was designed on the assumption that the community at large using it would not be interested in abusing it. This was the case back when the Internet was not yet commercialized, and I remember it pretty well.
I think the only thing that will resolve the spam issue is abandonment of SMTP as we know it, and an adoption of a new protocol that enforces traceability. This is nothing new or scary - the IP numbers are all tracked and the BGP tables that run the internet all provide traceability to the source. Even though your average Joe might not be able to do it, but ISP's cooperate and exchange this info all the time on NOC-to-NOC basis to identify sources of trouble.
A similar system will need to exist for mail, that will require some sort of a registration and compliance to join the "mail provider" network, whatever that will be. As soon as the e-mail becomes traceable to the source, perhaps even if not with 100% accuracy, there will be a drastic reduction in spam.
Second problem is all those exploitable zombie Windows boxen out there, but I don't know what to suggest here...
In all seriousness. How much spam can you possibly be getting?
I keep hearing horror stories about people getting 100+ spam emails per day. This leaves me with the question, HOW IS YOUR EMAIL ADDRESS GETTING INTO THEIR HANDS!?!?
I don't sign up for every "free" offer that I come across. I don't have business cards made up with my email address. I have two email addresses, I might receive 10 spams per week between them.
WTF are all of you doing to get on so many spammers' lists?
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
It's a value judgement... and according to my values, I think this is not a great idea.
First, I think the benefits of having free and semi-anonymous e-mail outweigh the disadvantages of having to use and maintain spam filters. Obviously, many people disagree with me here, and more all the time.
(Here's a conspiracy for ya: what if some Big Brother is trying to kill the free exchange of ideas in e-mail by burying the whole system with spam? I don't believe it's true, but it's worth wondering about before jumping to non-free solutions!)
Second, even if I thought that killing spam was worth the cost of crippling some of e-mail's better and more distinctive features, I think going about it in a non-standards-based way is likely to be a road to chaos.
The best solution, I think, would be to supplant e-mail with something new that works in a more trusted and accountable way. If someone really hates spam, they can use only the new system; if they want anonymity and freedom at the cost of spam, they can use the current mail system. The systems could coexist much like Usenet and the Web; each is useful for different things.
With reasonable men I will reason; with humane men I will plead; but to tyrants I will give no quarter. -- William Lloyd
I don't mind downloading the spam because I have broadband. Getting mail is no big deal, but sorting it is.
The solution I use requires that one owns a domain. Simply provide specific addresses to people/places/things depending on your expectation for spam. Filter on the client name based on the to: field and most of the crap drops into the crap folder where it belongs.
This combined with a bayesian filter keeps the spam to a very reasonable level. One added bonus:
You can know who sold you out and pass the word to others.
I use gandi.net for this. They provide e-mail redirection for free with a grab bag for unspecified addresses. 12 euros per year with nice online admin tools combined with very reasonable legal terms makes the service well worth it.
As for the e-mail problem, it is going to come down to trusted mail servers. I believe we all should be able to run mail out of our homes, because that is part of being peers on the Internet.
So, anyone can send mail, but if you expect anyone to actually read it, you need to be trusted by at least someone
Blogging because I can...
If you can send an e-mail anonymously, so can spammers. If spammers can't send e-mail anonymously, neither can you.
The price of spam doesn't come anywhere near the value of privacy and freedom of speech. I happen to like the idea that should a need arise I can easily send an untrackable e-mail. I'm sure plenty of people in more intrusive countries already enjoy this ability.
Click on the link in my sig for my method of dealing with spam which is highly effective that doesn't destroy the privacy of the sender or cost money.
Ben
Work Safe Porn
Maybe Yahoo's idea will work, though it seems to be quite porous and more of a surveillance tool than an antispam measure.. in fact it is quite plausible that this is Homeland Security's wet dream and is being sold by Yahoo on their request (though that is more paranoid than we have to be).
I have a concrete proposal at the end of this post so please read on.
Anyway someone mentioned the tipping point and I am reading this after cleaning a thousand spams out of my mail folder so I am ready to consider lots of things.
But one thing is definite about all this. If these guys were terrorists planning some horror and not just an army of rotten people bent on selling viagra and insurance, they would be shut down in a heartbeat. You can follow the money! (As many people have.)
Note these datapoints:
- Telemarketers don't like getting phone bombed, as Dave Barry launched retaliation against an association of them.
- Spammers are in it for the money
- Their clients pay because they want to sell something.
- Their clients are living in meatspace and are allergic to publicity.
- Spam is by definition, easy to get since so many are sent from each machine. (In fact I get too many to even reply with "unsubscribe" to them all).
- We all see spam, but can't stop it because the spammers are laughing at us by endlessly transforming their campaigns. The helpless feeling I suppose is similar to terrorism in that there is a feeling of a nebulous enemy profiting by your openness, there is nothing to grab hold of.
- People are willing to pay money to stop spam.
- Homeland security (probably) and the NSA and similar national organizations (definitely), and telcos and isps (of course) are sitting in front of the big routers around the world. This information can be coordinated.
- Some big organization wants a steganography analyzer built quickly (recent slashdot story)
From this and a bit of blue skying and paranoia, I get:
1. Spam, which is subtly personalized and includes photos and hyperlinks, could be used as a communications network by terrorists, so definitely falls under the national security bailiwick. Ditto for viruses and worms, though they are maybe too visible.
2. Though maybe it is better to unlock the messages than to stop spam, from a security standpoint.
3. Certainly it is possible to make transparent who exactly is sending spam, and how the money flows from their clients. Both by surveillance and of course just trying to buy some of their services.
4. If it isn't illegal, they can't be put out of business and so long as they have clients, it is a "business opportunity".
5. But by focussing the anger of thousands of people on each client and detected spammer, this lucrative business can be turned into a financially losing proposition.
6. Finally, if we make it impossible for their clients to sell their wares, there will be no point to spamming. This suggests that rather than trying to secure all of the honest email, we should focus on removing spam from the network. I don't think blackholes work, however it is quite possible that a finer granularity and more intelligence might work. (See below)
So I welcome technical fixes against spam but think they should more involve information sharing than an attempt to cryptographically secure the email network, since the power of email is fundamentally that it is so easy to use.
I would propose that a group of people are selected around the world to manually go through their incoming email and note which emails are spam, preferably qualifying what type it is and using some simple tools to also note whether this is the work of nefarious arch-spammer types that play tricks on you, as opposed to honest mailing lists. It should be an open architecture which allows more than one organization to do the grading. Perhaps one will only filter porn, etc. I believe some large antivirus companies do something a little bit like this on an automated level to learn about thre
I don't believe this is proprietary. Yahoo is releasing a patch for Sendmail. AFAI can tell, while they're funding the dev work (because the spam rate is killing them), they aren't trying to milk this for more money.
One major problem with standards groups is that people like Verisign are on most security standards groups. Verisign has extremely strong motivations to ensure that email uses a Web-like interface, where one purchases an (expiring) Verisign cert for each email server one runs. They have strong incentive to block competing solutions. If you want to come out with a good system that prevents existing folks from milking a market, both industry consortiums and standards groups are pretty much useless. You need to do what happened with PNG -- have a bunch of talented, aggravated engineers sit down, write up a technically good spec, and put out reference code. Later on, let standards committees follow what's in place.
I can't figure out why replay attacks are an issue. I, personally, would suggest, off the cuff, including any To: or CC: lines in the message body (just for signing purposes, not actually sending either header in the body). This way, a replay attack would only allow resending the same email to the same destination from the same source. It's also pretty easy to include a timestamp, if folks are *really* concerned about replays.
Yahoo is pretty much doing what ESR and RMS have been hoping for for years -- contributing to open source systems because there's an itch that needs scratching.
Paul Vixie (disclaimer -- I don't move in his circles, and what I know about him is entirely secondhand) seems to be involved a great deal in politics, rather than technology. He leaves a bit of the same bitter tang in the mouth that Verisign does. He is, apparently, the source of at least some of the IETF objections. Vixie has also made a number of antispam statements that I tend to disagree with, including advocating mass blocking of mail servers on home email connections by netblock.
May we never see th
First let me say I agree with your premise. I have never received an anonymous delivery, email or otherwise, that I desired.
But let me show the fallicy of yahoo's actions.
Yahoos step 1 is to reject forged headers. Forged headers was just made illegal by the Bush administration IIRC. I completely approve.
Yahoos step 2 is to force a signature on every email by the server. Interestingly, Step 2 removes the need for step 1 and makes you wonder if step 2 is their real desire. Note that a solid step 1 also removes the need for step 2, given that open relays are shut down.
This is where I disapprove.
This proposes the same problem as DRM. Who controls which signatures are accepted? Once again we are right back with Verisign, et al. So unless your server has a PURCHASED KEY from verisign, or the like, your server won't be sending email to yahoo or any of the ISPs that adopt this.
I promise they won't be suggesting PGP either And so the spiral begins. Yahoo sells the rights to the certificates it will accept on a yearly basis. Verisign subsells this right in the form of the infamous certificate chain.
So what if the code is free, the certificates are not!