Slashdot Mirror
Replacing SMTP?
dousette asks: "In reading over one of the RFC's governing the SMTP protocol, and other RFC's as well, it's interesting to note that you see some big names and big companies from time to time. With all the loopholes in the current SMTP specification, is it possible for the Slashdot collective to come up with another one? Would it stand a chance in making it into a standard, or do they just listen to Cisco, AT&T, etc? I realize that a lot of people have a lot of ideas how things should be done (and they haven't been shy about posting them to Slashdot), but has anyone tried to write the RFC for a replacement protocol? As a side note (where I won't be shy about posting how things should be done), if there were a replacement trusted protocol, one could have mail received via that protocol bypass spam filtering, id checking, or whatever checks might be in place (saving processor cycles, etc). The regular checks could still be done on other mail received via the 'older' SMTP protocol. If more and more ISP's make use of this, SMTP could be gradually phased out... or if you are one for a sudden cut-over, just cut to the new one at the same time as the IPv6 upgrade!"
SMTP is so deep-rooted and pervasive already it will be a long, hard change to implement. Every little cellphone that comes with a mail-client. Every router that has smtp alerting. Every application that uses it for various tasks... they would all have to be updated!
Doesn't mean it shouldn't be done, but don't be fooled into thinking it's just a "propose a new spec, step 2?, profit" type of deal....
--D
I agree, any solution to spam that relies on replacing the SMTP protocol is bound to fail for this reason. The current issues with IPv6 migration should prove to everyone that the strategy of rip-out and replace does not work. I think what needs to be explored are added backwards-compatitable extensions to the protocol. Perhaps adding a few commands for whereby some sort of public key exchange is involved.
Hrm, never seen that before, im2000 has some good idea for simplifiying things, but it seems like it would just be unreliable and unfeasible.
With the current system, an smtp server can go down, and no one would notice because no one was received their email yet, but with im2000, if the sending machine goes down, then no one can read their mail from there. This would create a lot of unknowns, "why can't I read my email?". Also what about people that don't have a full on connection, you don't want to require those people to be connected just to read their mail. Sure you can queue it for downloading offline somehow, but that's going to be much slower than normal because you have to connect to say 30 different servers where your email is hosted.
Also there's the case of somesmallcompany.com sending out a mailer/advertisement to millions of people, because the email is hosted on their machine, their connection/server might become overwhelmed, causing heaches for everyone wanting to read their mail. "Why does my mail load so slow?"
It's a nice try, but it'll never work.
Another thing, what happens when the message is done being read? Is it deleted on the sender's machine? If so, then how will the user remember that they sent the email to check if it's been read. If not, when will the message get deleted? Obviously it can't stay there forever.
The great thing about the current system, is that you just send and forget. If it bounces, you get a new email message saying hey, something went wrong. But with im2000, if the message hasn't been read yet, WHY? Did the user just not check their mail yet? Is there connection/routing problem where they suddenly occurred after the hosting server sent the notification, etc.
Working out the details of an appropriate certificate policy is not trivial, though.
Actually, I've been working on a broader based piece of infrastructure than a new mail protocol, but the first problem I intend to attack is mail.
RFC 822 is fine for messages, but the transport needs a big upgrade. Also, envelope senders and receivers are non-verifiable, and therefor broken. One day, spammers are going to start using mailing lists and message boards to construct a profile of people you talk to, and send you mail that appears to come from them, thereby making whitelists useless.
The basic premise of my general transport is that all messages are addressed to a public key and come from a public key. All messages are signed by their supposed source ID, and most messages are encrypted to the destination ID.
A public key ID plays a similar role to an IP address in an IP packet. There will be distributed databases that hold (signed) mappings between public key IDs and their locations using other networking mechanisms.
I'm trying to design this protocol and its implementation so its easy to encapsulate it in almost anything. My first connection to an outside protocol will be IMAP/SMTP.
It's far from being ready for even a public alpha yet, but I do have preliminary code for creating certain kinds of messages at https://svn.generalpresence.com:5131/repos/trunk/C ++/pract_crypto/. I'm borrowing heavily from Bruce Shcneier and Niels Ferguson's latest book, Practical Cryptography. The initial implementation is in a mix of Python and C++. It requires Swig and the GMP library. I haven't designed the implementation itself to be in the least robust against attacks by someone who has root on your machine.
I am calling the protocol 'CAKE' for now. CAKE stands for Key Addressed Crypto Encapsulation. It is a layered protocol, since I intend it to be layered on top of any other protocol you can think of. :-)
One intention of mine is to publish a hash collision problem along with information mapping a public key to a mailbox. First time senders will have to solve the hash collision problem to avoid having the mail thrown away. I'm planning on simply wrapping an RFC 822 message in a CAKE shell.
Need a Python, C++, Unix, Linux develop
Well, I think those "loopholes" are all the zillions of features that people want, but they aren't good enough software designers to realize that the features belong in the layer above SMTP.
In particular, lack of authentication is a strength of SMTP, just as it is with IP. It means, for example, that I can implement my own authentication (or plug in PGP or whatever), and don't have to use the mail-transfer layer's after it turns out to have a serious hole that lets the spammers and con-men through.
Protocols that try to do everything for you have the inherent problem that, when a serious problem arises, you have to put up with it until the idiots at the vendor decide to solve it.
SMTP is simple enough that even a relatively incompetent programmer can do it correctly. You can type it yourself via a telnet connection.
And adding features in the higher layers is easy.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
That's not the answer either. Microsoft, Yahoo, et al have been lobbying for this approach, and for good reason. They want to function as the certificate authority (CA). They want to determine who can or cannot send email. They can use that power to literally sell the ability to send spam. They can also use that ability to censor their opponents.
Microsoft also wants a new patented standard that can't be legally implement with open source software.
There is a key difference, in that IPv6 affects the hardware of every net-connected computer (not to mention router, hub, switch, etc) in the world (I believe), whereas changing the mail protocol should involve minimal hardware changes, but admittedly extensive software changes. It's easier and cheaper to change software than hardware.
> In that case, who would define "correct" addresses, the ISP? And how would they be defined? I have at least 1-2 email accounts that I retrieve mail from with POP3, but send outgoing mail with the same domain through my ISPs mail server because there is currently no other way. I own (or, more correctly, lease) the domains myself, so no one can legally tell me tell me that I can't send email using those domains. The fact that I send outgoing mail through my ISPs mail server happens to be a necessary evil.
---
Aye, there's the rub. I do the same thing, what with having about 5 domains and various e-mail addresses for each.
There needs to be:
1) A way of verifying if you're allowed to use said mail server. Easy. Simple login/password over encrypted connection - technology already in place.
2) A way of verifying what e-mail addresses & domains are allowed on outgoing e-mails from said mail sever. That would be new, but should be easy to develop.
3) A way of destination server contacting the originating server and verifying the e-mail it received is from an authorized and stated e-mail account. That would be new, too, and would be a bit more complicated, but still fairly simple.
4) While you're at it, you might as well encrypt the whole frigging process. The saying, "E-mail is not like a letter; it's like a postcard." should be obsolete. It should be like a letter written in a language only the sender and receiver can understand. By default. Every time. The technology is around, but needs to be standardized and integrated and something the user never has to set up or think about. I loved a recent commercial that said that something was really private, "as secret as your e-mail password." Yikes. People _really_ don't understand e-mail technology.
Also, a way to have a mail server respond to a confirmation request only by servers it's sent mail to recently would be a good thing - that would cut down on trying to scan a server with a dictionary attack to get valid e-mail addresses to spam to.
The problem is not that these are difficult technical challenges - they're not, and the technologies exist in fairly decent form already. The problem is getting this done in a standard and accepted way and out into the field for everyone to use. _That's_ gonna be a real bitch.
As one of the posters in this thread pointed out, it would go a long way towards helping establish accountability on the part of spammers. If you have to be accessible in order for "recipients" of your mail to be able to read the mail, then I suppose you'd have to be pretty easy to track down and also sue under the current anti-spam laws.
... I've been using it on my server for years and really love it. I'd be particularly interested in hearing what your ideas are about the ways that we can solve the spam problem, given that you are the author of a complete modern mail system.
Anyway, it's just an idea. I think that the whole question of "should we come up with a new mail protocol" is kind of misguided because obviously the hard problems here are not technical, they're in dealing with the huge momentum built up around the current mail technologies. It seems to me that we already have all the technology that we need to solve this problem, it's more a matter of enforcement by ISPs. I was just posting the link because it was relevent.
If you're the same Mr. Sam that made Courier, then thanks
Sure, but for most of us the *time* involved in dealing with spam is a greater cost than the bandwidth involved in receiving it. Bayesian does a great job at solving the "waste of time" problem. And, as I said, if everyone used it I believe spam would disappear quite quickly because the response rate would fall too low for even spammers to have an interest.
Blacklisting servers and not accepting connections from them (and accepting the collateral damage) is the only practical option I have.
In the case of a few a spamhauses, sure. But as an effective spam-fighting measure that's a useless approach. You (or someone) has to keep up to date with the latest servers to blacklist (and then whitelist them when they become clean), or you have to deal with an annoying level of false positives that you don't even see. Sure, you can say that that's the price of users dealing with an ISP that is spam-tolerant. But some of us want to do business with those users even if they chose their ISP poorly--or if they don't have any local choice of ISP.
A combination of white lists/black lists, and Baysian filtering...
Stop yer bitchin', people, and implement the technologies that are already out there and work great.
This doesn't sound like a general solution for J. Random Homeowner.
It sounds alot like "Well, shoot! Quit yer bitchin' an just put in some tuned ports and performance cam! Hell, my grandmother could do that!"
My other car is a 1984 Nark Avenger.
I don't risk job loss (since I'm self-employed), but I do risk missed contracts. I use Bayesian. I've had an occasional false positive, mostly during the first couple months when I was building my corpus.
BUT: I'm currently receiving about 140 spams per day. Do you really think I'm going to do any better than Bayesian regarding false positives when I'm mass-deleting 140 messages? I think not. I'm almost positive I'm going to incorrectly delete more good messages than Bayesian's false positive rate when I'm dealing with such a huge number of emails manually.
Plus if your Bayesian system is half-decent you should be able to adjust its sensitivity. On mine 0-49% is almost always good email and 50%-99% is almost always spam. When I look for false positives I look for anything unusually low, such as 65% or so. If you want, just adjust your threshold to 65%. I've never had a valid email with a Bayesian score of, say, 90%.
A combination of white lists/black lists, and Baysian filtering stops so close to 100% of spam that it's really silly for anyone to be bitching about spam these days. I don't GET any spam anymore - 0. Not 0.001%, 0 - the integer 0, as in none.
Have you done the power-curve analysis on that? My mother works at a law firm, and they once tried to install a spam filter. It was state-of-the-art, with Bayesian filtering, and white/black lists, and additional whitefilters on top. It blocked most (not all) of their spam. But it also blocked some tiny fraction of legitimate messages.
Even if you have the (extremely impressive) power curves of Paul Graham's Plan for Spam -- and that was on a very well-trained Bayesian filter written by a coding genious -- it is Not Good Enough when missing a legit email could get you sued for millions. Either you risk blocking legit email, or else you have to wade through a pile of spam bigger than that legit email... either way, another protocol would be nice.
I hereby place the above post in the public domain.