Slashdot Mirror
IETF to Look at Spam
m00nun1t writes "CNET has an article about the Internet Engineering Task Force (IETF) looking at what they can do about spam. According to the article, many of the proposals seems to "require changes in basic e-mail technology", which presumably means SMTP (and about time!). Maybe they are looking beyond just SMTP - anyone have any insights here?"
If an alternative to SMTP were developed, the protocol would not be likely to disappear immediately subsequent to the creation of its successor. The transition would be gradual, as reverse-compatibility could remain necessary for several years afterward. As suggested by the release of Apache 2.0, for instance, not every server administrator adopts a "technological improvement" until it becomes an adequately proven and stable product.
Do you like German cars?
Maybe their address got used as a reply-to in the latest "$1000 per week from home" letter...
Have you seen my stapler?
Although I'd find it hard for the IETF to swallow djb's personality, there's always IM2000.
http://cr.yp.to/im2000.html
Replicants are like any other machine, they're either a benefit or a hazard. If they're a benefit, it's not my problem
First Dupe!
Among many, many others, I saw Vernon Schryver, the guy behind Distributed Checksum Clearinghouse, on the list. It's been pretty high volume, though, and I haven't had a chance to really spend some time reading it yet.
Carousel is a lie!
The really interesting thing about dupes is that they tend to suggest that there are large numbers of readers who pay more attention to the site than the guys running it.
If I was running slashdot, I'd probably push the people who had the power to approve stories to read each and every story that gets approved. It seems like a reasonable minimal committment to the community even for volunteers, and presumably some of these guys are drawing actual paychecks for the work they do here.
The dupes show that the guys approving the stories don't really care enough to take the time to do that.
Someone posted a response to another spam story a few weeks ago, sadly I can't find it, but they described an interesting mail delivery system they'd created.. and it sounded, to me, as if it could certainly be the future of mail delivery.
They said that when someone sent a mail, it simply went to the local server, and no further.
It sounded like a 'reverse IMAP' style system to me. That is, your outgoing mail simply went to a folder on your server, which allowed you to edit and even delete mails BEFORE they were picked up by the recipient. The recipient's e-mail server would only receive a 'notice' that someone had mail for them.
When the recipient went to collect their mail, their own mail server would then have a basic list of where the e-mails for the recipient are, and then it'd go ask for them from the remote servers and feed them through.
So, how does this help spam?
It allows spam to be truly filtered on the OUTGOING rather than the incoming!
Why's that a great thing? Well, it means that if you're an AOL or MSN user, you're not going to lose 80% of your mail simply because of over-zealous filtering by your ISP. Instead, spam mail will not even be sent, let alone received!
Of course, bad eggs could always set up servers with no filtering systems on them and send their spam that way.. but BECAUSE e-mail will be picked up FROM the senders server with this system, it means blacklisting is a whole lot easier! You just ban a server and you know you've got rid of the bad eggs.. whereas the current SMTP system allows open relays and all sorts of 'trickery' to get around filtering systems.
So.. the conclusion is.. make e-mail stay on the sender's server until it's time for it to be collected. It allows you to edit or delete mail before the recipient collects it, it stops spam, and it reduces bandwidth(!) -- if someone never collects their mail, then the mail has never gone across the net.. it's still on the sender's server.
I hope the original poster of this idea will pop up here again and correct me if I got his ideas wrong, but he was certainly on to something.
mogorific carpentry experiments
The fundemental diference that protects most communication systems from Spam-like abuse is that the sender is responsible for a majority of the costs of the message. Yeah, there are telemarketers and junk postal mail, but the are seriously limited by the fact that there is a noticiable cost assosciated with each additional message they send. The fact that it costs money to send such communications makes it impractical to bother people with offers with an extremely low reponse rate.
SMTP/POP3 e-mail presently leaves the cost of holding the message during the wait for the intended reader to be available on the receive-side server. The spammer doesn't even have to maintain a constant and consistant Internet connection.
Under the current system, a sender can send 100 MB of messages in an hour without penalty. However, a receiver who gets 100 MB of messages in an hour usually will find any other messages sent to them bouncing.
Requiring the message be held on a sender-side server instead would transfer the costs of sending a large volume of e-mail onto the sender, and therefore discurages the practice better than any law ever could.
at last I will know to where in Nigeria I should go!
/solicited/ spam as being the only way you will be able to read salon. if it's still going by then.
seriously tho. even if it is all legal-ified and I'D correctly, there will still be such things as ticking the little box to say you dont want any spam from service X,Y, and Z.
In fact, the way online revenues are going i can see recieving
it would be nice(?) to have a better system but I never forget the age old adage of no system being tamperproof. Lots of enterprising folks enjoy anonymity for non-spam purposes, so naturally some form of workaround should emerge fairly quickly.
oh lord i'm sounding like Toffler.
<B>note to self:</B> <I>post as html</I>
Dan Bernstein has a project called Internet Mail 2000, ment to design a new Internet mail infrastructure which makes mail storage the sender's responsibility.
The technology exists, off the shelf, today.
There is a SMTP command called STARTTLS which will enable SMTP over SSL. It's defined in RFC 2487. Sendmail supports it with a compile-time option, and so do most other MTAs. It's backwards compatible with normal SMTP.
You will need a certificate, of course.
This has 2 big effects:
- encryption of email in transit between SMTP servers (a nice bonus)
- authentication of SMTP servers
Since sending spam isn't illegal in most jurisdictions, knowing WHO sent the spam (or relayed it) allows you to contact them and complain, threaten and retaliate (mailbomb, portscans, DDOS, etc.)
If you receive email from a host authenticated by versign (or whoever), you apply little filtering.
If you receive email from a host not using ssl, it goes into a queue for maximum filtering.
Much of the spam I receive today is from DSL customers who spew directly.
Downside:
- there will be additional CPU load for all the email servers
- cost of certificates
If any mail infrastructure reorganization were done before the finding this mount of this sendmail hole , that would have been be a good way to have a mostly forced deploy of compliant mail servers around the world.
Laws might not be able to stop spammers, but protocols have a better shot.
Simply put, it's too easy to create spam over the SMTP protocol. The from, and reply to fields are completely free text, and have no requirement that they must be a reflection of the actual sender of the message.
However, if SMTP were to fall out of favor for a new protocol, that new protocol could start the rules over and require that the server that is named in the from field must confirm that the username provided actually sent the message. Spoofing for the use of spam would then become practically impossible.
Once we have a confirmed from address, it puts a responsiblity to stand behind each e-mail sent through a server. Moreover, once spam use has been detected a really reputable server operator could simply stop authenticating that sender's e-mail.. causing auto deletion before its presented to a user.
If you can't make it illegal, try to make it impossible...
The article notes that one of the major problems is the filtering of genuine mail due to agressive spam filters necessitated by cleverer spammers. Consider this analogous to dropping some packets at the network layer. Just as the transport layer handles this problem, we can build a higher level protocol to handle filtered mail.
Note that having a mechanism to handle dropped mail allows us to employ agressive filtering: one that is sure to stop 100% of spam.
What I have in mind is as follows: when Bob receives a mail from Alice (i.e, it has passed through Bob's filter) the client software sends a confirmation mail back to the Alice. This is not a regular mail that the Alice will see in her inbox; it has a special header flag that marks it as a confirmation. Alice's client software keeps track of the confirmation messages; by looking at her "sent-mail" folder she can see which of her messages have not been confirmed (and are hence likely to have been mistaken for spam).
Finding that Bob has filtered her mail, Alice can either re-word it and send it again or do something like (assuming that Bob knows Alice): "Hi Bob, this is me, Alice. Your filter blocked this so I've rot13'd it to get past the filter. rot13 what follows to read my mail." Another option is to encrypt the mail with Bob's public key (assuming that spammers' scripts won't be clever enough to get your public key from your web page). Note that 99% of the time the mail is going to get through. You have to make that little effort to prove you are a human only once in a long while.
There is minor problem with requiring the receiver to send a confirmation message: Bob might check his mail only after a couple of days, during which time Alice may assume that her mail was blocked. There are 2 solutions: either Bob runs a script to filter his mail regularly, or else has his ISP implement his filter for him.
Note that this won't work if you have the receiver send a reply whenever the message did get blocked: the reply could itself get blocked etc. (This is called the red army - blue army problem in networking).
World of Ends, recently discussed on Slashdot, discusses why the simplicity (or stupidity) of the Internet is so useful. "The Internet isn't a thing. It's an agreement," they say.
That same argument applies to e-mail. Following their logic, it is best to leave SMTP alone. Simpler protocols are better. Leave the "value-added" pieces to the edge, and let the simple message transfer protocol alone.
J'aime mieux les méchants que les imbéciles, parce qu'ils se reposent. -- Alexandre Dumas
I know - I don't write code - and this probably sounds stupid. But I don't really see any way of forcing someone to quit sending you email. SMTP is short and sweet - but it can't continue to just hop mail. It has to be checked somehow. And it would slow down the mass emailers a lot. Hopefully someone a lot smarter than I in this area can come up with something.
Duke
FreeBSD: Nothing runs like a daemon with a pitch fork.
Watch for any attempt to impose digital certificates or other revenue generating schemes--wouldn't Verisign love it if now not only those who wanted SSL to work without presenting dire warnings to customers but everyone who wanted to be able to send email at all had to pay Verisign's extortion money for a certificate recognized by MSIE.
CEE5210S The signal SIGHUP was received.
email is by far the most widely used of all Internet services. I belong to an organization many of whose members are retirees are on fixed incomes, and it is only within the last two years that the number of people with email has grown to a critical mass (about 2/3 of the membership).
Of members of the lay public who regularly use email as a means of communication do not have the level of technical comfort that most Slashdot readers take for granted.
Of people who use email, the percentage who know how to use a web browser is much less than 100%. The percentage who can google for information is much less than 100%. The percentage who can successful extract and decode an email attachment is much less than 100%. The percentage who can view a government form or a corporate brochure in PDF format and read it with Acrobat is much less than 100%.
And the average age of their computers and operating systems is much more than three years--and they're not likely to update their email programs.
Whatever is done needs to be 100% backward compatible with existing email clients, not requiring even simple upgrades, or an astonishing proportion of real-world Net users will be disenfranchised.
(And please, let's not have any facile expressions of contempt for AOL users or webtv clients or people who bought email appliances (that includes one of the retirees I mentioned).
"How to Do Nothing," kids activities, back in print!
Hi, arvindn, it's me, 1nv4d3r. Your spam filter blocked my mail, so rot-13 this to see what I sent you.
TEBJ LBHE CRAVF OL FVK VAPURF, VS LBH URYC ZR FZHTTYR ZBARL SEBZ AVTREVN.
(not to mention that return token let's them know that you at least look at your mail.)
He uses the idea of emergent structure. To quote, " if all (or even most) players expect other players to act in a certain way, a predictable pattern of behavior emerges which becomes compelling for all players. This is the way all organizations work."
Replace "IETF" with "Microsoft" and you have thisslashdot story a whopping two weeks ago. Of course the slant then was how evil Microsoft was for daring to make people pay for email (which of course was not true... the article was about email accountability to reduce spam).
"It takes considerable knowledge just to realize the extent of your own ignorance." - Thomas Sowell
When the protocols we all use now were developed, everybody trusted each other. There wasn't a real need for advanced security options. Nowadays, with the current commercialization of the net (which also provides me with my income) it looks as if the commercials are winning. By commercials I mean those who have absolutely no respect for other peoples right or bandwith. Let's not forget that spam isn't the only problem: dos attacks are a real threat too.
Due to the original designs being not real secure, I'm quite sure that the spam problem can not be solved without fundamental changes in the way we use email nowadays. Perhaps the policy regarding blacklisting can be changed: at this moment most people accept mail from everybody, but not from a few blacklisted sites. It's likely that this will be changed: we don't accept your mail unless we know who you are. Unfortunately, even then there will always be people who will abuse it. Hopping from one account to another, or sue-ing every single ISP that has the guts to disconnect their connection after spamming. In short: it's not simply a technical matter, their will be a need of *globally equal* legislation too. Legislation alone won't do the trick either. No, it's time for Mr Geek to marry Miss LawAndOrder.
Don't forget that the IETF is not the first to attempt to find a solution. RIPE has its anti-spam workgroup for example.
I'm not a complete idiot... Some parts are missing.
The only problem with this is scalability. Sure SMTP has had its problems, but the nice thing about SMTP is you control the server. You control how fast mail comes in, from who it comes in, how fast people can give you e-mail, and how fast you give it out to subscribers/recipients. All of these schemes seem to remove that control from your machine.
Instead of adding a band-aid solution to spam, let's sit down and list what we need for an a-mail server. Scalability, reliability, fault tolerance, expandability and distributed servers top my list. I'm sure that there are other better ones out there too. If you're going to revamp the protocol, try to get everything in the first time, and let's try to get it right.
You think that I'm crazy, you should see this guy!
You are asking a little too much in a few ways.
Computationally, that's a lot of public key encryption in action. For sites that process large amounts of email, this is going to hurt. But let's say we can throw money and CPU at that problem. And I suppose we can do the same for the problem of the tens of millions of key/address pairs we'll need to store centrally. Not so bad, then.
Socially, the existence of anonymous email may be important and valuable. But I suppose anonymous remailers could appear and use their own corporate keys to signed messages in your scheme.
Practically, you'd need a way to prevent denial of service attacks against someone's email by generating sufficient fradulent 'bad reports' to cause their key to be centrally revoked. This seems bad.
Nothing totally insurmountable, but still pretty annoying to deal with.
(P.S. No packet on the internet is truly identified by its source in most cases. IP addresses can be forged fairly trivially. This doesn't really bear on your proposal, but thought you should know.)
Actually, it's the IRTF -- not the IETF -- that is undertaking this work. To quote from the IRTF home page - "[Mission] To promote research of importance to the evolution of the future Internet by creating focused, long-term and small Research Groups working on topics related to Internet protocols, applications, architecture and technology."
Don't expect a quick fix from this initiative.
No. Not all redundant mail is spam. There are plenty of legitimate distribution lists out there.
However highly localized approaches like personal mail filters will always fail due to the high variety of spam.
Yes. However, if a high enough % of people point at an email message and say 'spam' then it's spam. People are very good at spotting spam- so a mechanism that records what people think about the mail and not deliver it to anyone that hasn't already read it would be very successful. If 90% of the first few thousand people to read an email say it's spam, the other million need never see it at all.
-WolfWithoutAClause
"Gravity is only a theory, not a fact!"From what I take from all this discussion is that the only "solution" to spam is to do the types of things that we have been doing for years, but to do more of it and quicker. Use well run DNS blacklists (Spamhaus SBL, ordb, dsbl, etc.), use good content filters (bayesian filters, etc.), use bulk mail detectors such as DCC or vipul's razor, etc.) and per-user whitelists and blacklists.
Or, combine all of the above techniques by using SpamAssassin
--
I've been subscribed to the list since near the beginning and have been following it fairly closely. Much of the discussion has been rehashes of old topics such as "what exactly is spam?", "make the sender pay something, either money or CPU", etc.
The most interesting discussions that I've seen so far are:
Most spam specific programs will not queue and retry, and thus the spam will be dropped.
Spammers that use real mail transfer programs or open relays will need to be able to hold all their outgoing spam for a while, increasing the spammer's costs and slowing down the delivery of spam. Legitimate email will not be thrown out, it will only be delayed and only for the first time.
Of course, you don't really want the databases to remember every sender-recipient pair forever, nor do you want to remember pairs that were added by spam so this really isn't a "first time" database, but it is close.
Apparently the "canit" program already does this, but I had not heard of this technique before.
If you filter during the email receive process, you can make the sending MTA do the bounce. This means that you will not have to deal with spammers forging "from" and "reply-to" headers. You won't have to clean up bounces that never succeed, nor will you be responsible for bouncing spam to another victim that the spammer selected for the "from" or "reply-to" headers.
Also, false positives will recieve a bounce message instead of just disappearing. This reduces the danger of important email being lost.
Right now, there are DNS records that tell you which IP addresses are valid to try and send email to for a given domain (the MX records), but many ISPs have different machines for sending and recieving email. There are currently no DNS records to tell you which tell you which IP addresses a domain will send email from.
The problem with this kind of proposal is that there are many people who think they have legitimate reasons to forge "from" or "reply-to" addresses. It also forces ISPs to make sure that every time they add a new outgoing mail server, they need to update the list of valid IP addresses. If they forget to do this, then only bleeding edge spam filters will detect a problem.
SPF support for most open source mail servers can be found at libspf2.
...allow binary transfers. I can't tell you how much CPU time has been wasted by base64 encoding binaries, sending them over an inefficient protocol, and decoding them on the other end. yEnc does a good job but the whole encoding shenanigan is a major pain for anyone trying to send family photos or the latest AFI album. Please, IETF, make a better 8-bit clean push protocol, because SMTP is the only one we have.
"The lesson to be learned is not to take the comments on slashdot too literally." --Vinnie Falco, BearShare
I was the original poster. You got it basically right in your synopsis too.
To recount, my idea is for when someone sends a mail message, the message itself goes onto the local mail server and the header for the message would go to the recipient. The recipient's mail client would then download the message. However, it would be possible for the mail server to delete the message _before_ the mail client ever sees it in which case, the mail server tells the client this and the client would then throw away the header and the end user _would_never_even_know_ that the mail (spam) had been sent. This would be totally transparent. It would also allow of course, for sender of the mail to be able to tell if / when a mail message had been picked up. (Not read but simply picked up.)
One of the big advantages of a system such as this is that you know for 100% certainty where the spam (or other e-mail) is coming from. You don't have to spend time looking through forged headers etc. in order to send a complaint to the ISP.
ISPs on the other hand would be capable of canceling spam after it had been sent and before it was picked up by the end user. For example, someone send's 100,000 spams from an AOL account. Somebody notifies AOL that they received spam from the offending person, AOL looks and AOL then is capable of cancelling all unpicked up spams -- before they are ever delivered to the end user. Alternatively, AOL could also simply look on their servers and say: "hey, we have 100,000 messages that are waiting to be picked up, we had better look into this" and then make a determination from that point as to whether the mail should be canceled -- again before anybody (or very few) sees the spam.
Blacklists could easily be created too where the site is blacklisted for only a certain period of time. So after three days (for example) the blacklisting would go away automatically. This avoids the problem that many ISPs have where they get blacklisted due to a single user and then they can't figure out how to get off the blacklist. Using this approach, the blacklisting would only last for as long as the spambarrage continued.
Blacklists would easily be able to be created within certain organizations or groups of people who have similar "moderating" views rather than trying to make one (or very few) blacklist(s) meet everybody's needs as is now the case -- and often hurting people's ability to send and receive legitimate mail.
The protocol could not only specify what server the mail came from but also the user. So, for example, if someone were spamming from AOL, it might not be a great idea to blacklist AOL but only that user from AOL. This would work for mail systems where you know it is a legitimate business but with a few unruly users.
So, using this technique, it would be possible for a spammer to get a few spams out but it would be nearly impossible for them to spam very many people before it was caught by their ISP and canceled or the user/ISP was blacklisted for a period of time.
-Art
Pollarda@Lextek.com
Lextek: Suppliers of High Performance Text Retrieval Engines
Convincing "larger ISPs" to implement an alternative standard would also require prodigious effort.
Actually, the more mail your site receives, the more interested you tend to be in stopping the flow of spam. If you consider how much in resources they spend dealing with spam in terms of capacity (for storage, bandwidth, processing volume, and filtering) and user complaints, it isn't that surprising. If a workable implementation ever comes out of this, you can expect the larger ISPs to have test servers up pretty quickly.
I've been using email on the net for 20+ years and have been as grateful as anyone that it hasn't cost.
However , of late, with other people making such gross abuse of the world's mail systems that I feel I am (and all of us are) paying anyway, it might be worth revisiting this question. I'm not 100% convinced that paying per piece of mail sent would be more expensive than, effectively, paying (in both time and in dollars spent on ISP infrastructure) per piece of mail received. I send a lot less mail than I receive, and I bet most people do.
Kent M Pitman
Philosopher, Technologist, Writer