Slashdot Mirror
Replacing SMTP?
dousette asks: "In reading over one of the RFC's governing the SMTP protocol, and other RFC's as well, it's interesting to note that you see some big names and big companies from time to time. With all the loopholes in the current SMTP specification, is it possible for the Slashdot collective to come up with another one? Would it stand a chance in making it into a standard, or do they just listen to Cisco, AT&T, etc? I realize that a lot of people have a lot of ideas how things should be done (and they haven't been shy about posting them to Slashdot), but has anyone tried to write the RFC for a replacement protocol? As a side note (where I won't be shy about posting how things should be done), if there were a replacement trusted protocol, one could have mail received via that protocol bypass spam filtering, id checking, or whatever checks might be in place (saving processor cycles, etc). The regular checks could still be done on other mail received via the 'older' SMTP protocol. If more and more ISP's make use of this, SMTP could be gradually phased out... or if you are one for a sudden cut-over, just cut to the new one at the same time as the IPv6 upgrade!"
Why doesnt the new implementation use the evil bit. It the server is written by m$, or running on an m$ platform it sets the evil bit. If its running under linux it doesnt set it and ignores all mail comming in using evil bit! :P
Simple really :P
Can't Jabber do a lot of what you're asking for?
"is it possible for the Slashdot collective to come up with another one?"
SlashDot Transfer Protocol - Essentially, the way it works, is the information is posted on one single, easily crashed server. Then, this information is linked to by Slashdot. Then, said server is taken down. However, 1,000 other posters will have mirrored it by then, therby helping in the "transfer" of the information.
Jason Lotito
I suppose you've never heard of mod_gzip before, then?
You're probably typing on a QWERTY keyboard, right? Why? Its function is to slow you down so that you don't jam the typewriter.
Moral: Just because one design is better than an already widespread yet inferior design does not mean that it can and will replace the current one. Change is not easy in the least.
I don't think the name needs more work. We can just call the new one SMTP Hi-Speed and the old one SMTP Full-Speed. If the USB people can do it, so can we.
SMTP is so deep-rooted and pervasive already it will be a long, hard change to implement. Every little cellphone that comes with a mail-client. Every router that has smtp alerting. Every application that uses it for various tasks... they would all have to be updated!
Doesn't mean it shouldn't be done, but don't be fooled into thinking it's just a "propose a new spec, step 2?, profit" type of deal....
--D
YOU CAN'T RUN A PAY BY THE EMAIL SYSTEM UNLESS THERE IS *1* EMAIL SERVER (or network of servers run by the same entity) FOR THE WORLD.
you don't realize that IM is a form of email? you are just sending packets of text... the second someone charges for SMTP, i'll just run my own. you could just charge the end users for data transfered instead of flat monthly fee, but most wouldn't go for that.
MARIJUANA, SHROOMS, X: ONLINE?! - E
I'd like to simply see SMTP updated to require work. On establishing a connection, a recipient should be able to give the sender a task to complete that takes a second or two. The recipient will only accept the mail once the work unit has been done.
This would make it too slow to send spam, by making it simply too processor intensive. Legitimate users would be unaffected.
I agree that something ought to be done to cut down on the huge volume of spam that clogs most SMTP traffic.
On the surface of it, a white-listing system, perhaps based on public-key cryptography and endorsements might work.
But, as someone who values freedom and anonymity, I'd hate to have a system that closes off completely the opportunity for more anonymous communication via email.
Whistleblowers in the government and in the corporate sector, dissidents under a repressive political regime are some of the use cases for email that I'm not really inclined to sacrifice merely to eliminate spam.
"Provided by the management for your protection."
The "pay for email" approach would only work if it was possible to whitelist addresses who would then not have to pay. The mailing list problem then would not exist -- you simply require that anyone who signs up whitelists the mailing list address.
How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
I agree, any solution to spam that relies on replacing the SMTP protocol is bound to fail for this reason. The current issues with IPv6 migration should prove to everyone that the strategy of rip-out and replace does not work. I think what needs to be explored are added backwards-compatitable extensions to the protocol. Perhaps adding a few commands for whereby some sort of public key exchange is involved.
The QWERTY-slow typewriter story has been debunked. QWERTY forever!
To me a big problem with SMTP is that is never authenticated. There's no way you can verify anyone actually sent you an email, short of PGP keys.
At least if some one had to authenticate to send as joe@bar.com, some spammer would have to hack your password before they used your email address as the "From:" in a mailing...which just happened to me.
Like IPv6? You mean most things will already be there but no one will support it, no one will care apart from a few and no one will implement regardless of how hopeless and disastrous the current implementation is?
Ah yes, like IPv6 indeed. You know, I'll send a shiny mail delivered by SMTP2* over an IPv6* internet about the release of Duke Nukem Forever* to my gaming-addicted girlfriend* on the day SCO coughs up some evidence*
Note:
* = May or may not require divine intervention.
Hate me!
Mod the parent up. Another link about the qwerty myth is here.
How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
Hrm, never seen that before, im2000 has some good idea for simplifiying things, but it seems like it would just be unreliable and unfeasible.
With the current system, an smtp server can go down, and no one would notice because no one was received their email yet, but with im2000, if the sending machine goes down, then no one can read their mail from there. This would create a lot of unknowns, "why can't I read my email?". Also what about people that don't have a full on connection, you don't want to require those people to be connected just to read their mail. Sure you can queue it for downloading offline somehow, but that's going to be much slower than normal because you have to connect to say 30 different servers where your email is hosted.
Also there's the case of somesmallcompany.com sending out a mailer/advertisement to millions of people, because the email is hosted on their machine, their connection/server might become overwhelmed, causing heaches for everyone wanting to read their mail. "Why does my mail load so slow?"
It's a nice try, but it'll never work.
Another thing, what happens when the message is done being read? Is it deleted on the sender's machine? If so, then how will the user remember that they sent the email to check if it's been read. If not, when will the message get deleted? Obviously it can't stay there forever.
The great thing about the current system, is that you just send and forget. If it bounces, you get a new email message saying hey, something went wrong. But with im2000, if the message hasn't been read yet, WHY? Did the user just not check their mail yet? Is there connection/routing problem where they suddenly occurred after the hosting server sent the notification, etc.
> I wish people would stop inviting rate increases or new charges as an answer to spam. It's not the answer.
:)
:)
And the perfect example is regular junk snail mail. It costs them to send it, yet even in the Internet Age(tm), I still get a ton of it. Obviously that's NOT the answer, so "Don't Go There"(tm).
I think locking down SMTP servers and requiring verified & correct return addresses would go a long way toward curbing spam. Then when you disallow someone to send you mail, it could really work.
A combination of white lists/black lists, and Baysian filtering stops so close to 100% of spam that it's really silly for anyone to be bitching about spam these days. I don't GET any spam anymore - 0. Not 0.001%, 0 - the integer 0, as in none. If I ever get another piece of spam, then I'll change my email address (I can do that more easily than most as I have my own domain.), though this isn't the answer for everyone - lots of people have e-mail addresses printed up on lots of expensive cards & letterhead, etc. For them, the white list / black list / Baysian filtering solution should suffice way more than anyone should practically need.
Stop yer bitchin', people, and implement the technologies that are already out there and work great. Plus use yer freakin' brains for a change, and don't spew out your real e-mail address to everybody who asks for it. Use your friend's!
Working out the details of an appropriate certificate policy is not trivial, though.
C'mon now; the IPv6 upgrade will be spread out over at least several decades. And both Microsoft systems and many US Government installations will still be using it a century from now, because it's "standard".
After all, it's now past the death of typewriters, and we're still using the typewriter keyboard from nearly two centuries ago. And we use a ridiculous rail gauge, because the standard was set centuries ago.
And here in the US, we're still using inches and feet, measurements based on the lengths of the thumb and foot of a long-dead king. And we call them "standard".
We will be stuck with IPv4 for long past the final download of anyone reading this.
SMTP will probably be around even longer. But that's OK; it's fun to impress friends by a "telnet 25", followed by typing in a message directly to the server. I like to use "MAIL From: dubya@whitehouse.gov", and ask them if they'd be interested in a nice job in the TIA program. Then I challenge them to prove from the message they get who actually sent it.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
Technological fix to a social problem.
It's simple. Don't bother.
The problem will remain, it will just shift tactics. By 'fixing' SMTP you're not addressing the problem, you're addressing a symptom of the problem.
Anything we do on the technology side to fix this problem will ultimately do nothing.
That's not to say that SMTP can't be improved on... but improving on it purely to 'stop spam' is a waste.
Chicks suck.
Guys are ugly.
Pass the kleenex.
Actually, I've been working on a broader based piece of infrastructure than a new mail protocol, but the first problem I intend to attack is mail.
RFC 822 is fine for messages, but the transport needs a big upgrade. Also, envelope senders and receivers are non-verifiable, and therefor broken. One day, spammers are going to start using mailing lists and message boards to construct a profile of people you talk to, and send you mail that appears to come from them, thereby making whitelists useless.
The basic premise of my general transport is that all messages are addressed to a public key and come from a public key. All messages are signed by their supposed source ID, and most messages are encrypted to the destination ID.
A public key ID plays a similar role to an IP address in an IP packet. There will be distributed databases that hold (signed) mappings between public key IDs and their locations using other networking mechanisms.
I'm trying to design this protocol and its implementation so its easy to encapsulate it in almost anything. My first connection to an outside protocol will be IMAP/SMTP.
It's far from being ready for even a public alpha yet, but I do have preliminary code for creating certain kinds of messages at https://svn.generalpresence.com:5131/repos/trunk/C ++/pract_crypto/. I'm borrowing heavily from Bruce Shcneier and Niels Ferguson's latest book, Practical Cryptography. The initial implementation is in a mix of Python and C++. It requires Swig and the GMP library. I haven't designed the implementation itself to be in the least robust against attacks by someone who has root on your machine.
I am calling the protocol 'CAKE' for now. CAKE stands for Key Addressed Crypto Encapsulation. It is a layered protocol, since I intend it to be layered on top of any other protocol you can think of. :-)
One intention of mine is to publish a hash collision problem along with information mapping a public key to a mailbox. First time senders will have to solve the hash collision problem to avoid having the mail thrown away. I'm planning on simply wrapping an RFC 822 message in a CAKE shell.
Need a Python, C++, Unix, Linux develop
This scheme is an important part of the old UUCP package. Part of its handshake protocol is a message that lists all the protocols that the caller understands, in the order the caller would prefer to use them. The recipient goes through the list, picks its favorite, and sends back a message saying "Let's use X."
The advantage to this is that you can introduce new protocols completely painlessly. You pick a new name (after asking around on the newsgroup if anyone is using it), link your new protocol module into the protocol tables on the systems where you want to use it, and start using it. If you connect to a machine that doesn't have your protocol, it will simply tell you to use one of the others on your list. If your protocol is good, it will spread and will be early in the table for a lot of software. It can then slowly supplant the older protocols.
And you stay compatible with older systems by merely keeping the old protocol modules in your tables.
This is 1970's technology. So I suppose we'll soon read that Microsoft has just patented it.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
Because the cost becomes built in to their business model. it won't stop, it will only hurt regular users to charge for email/services. Sure, their profits may be cut a little bit, but that's not going to stop them. if anything, they'll do it more, because if their profit margin is smaller, they'll have to spam harder... right?
Non-email messaging systems have been thinking about virtually the same problem quite a bit, and have come up with a set of solutions that try to solve what are fundamentally the same issues: message integrity, message non-repudiation, and message authentication. And the surprising part of this is that nobody really focused on the protocol, because it doesn't provide the path to a meaningful solution to the problem.
Case in point: web services. While initially the people who were playing iwth web services started out doing security at the transport level (i.e. with SSL and various derivatives thereof), but realized that something like WS-Security (where the security of a message is a part of the message itself) is the more optimal approach.
Why not just force the issue into the realm of S/MIME (and similar extensions to rfc822) and handle it at MUA space? You can cover virtually all the problems with SPAM by following the example of the reliable messaging systems and doing more with the contents of the message itself, rather than trying to say that messages have to transmit over a particular protocol. For example, depending on your trust environment, S/MIME signatures solve the authentication, non-repudiation, and integrity problems perfectly. What more do you need/want?
Maybe ISP's should charge users for each outbound SMTP connection they make? I'd happily pay 10 cents per email I sent if it would reduce the amount of SPAM I received. It would only cost me a couple of bucks a month too at the rate that I send email ...
John Dvorak suggested a scheme along these lines, and in theory, it's a good one, though I'd suggest a tenth of a cent, which would still make sending a million emails prohibitively expensive.
In practice, though, it's not workable. Spammers aren't using the SMTP server their ISP provides; they're using their own, just like most desktop Linux users are. As far as the ISP is concerned, Spammer X is making a bunch of outbound connections, but they're streaming out through the ISP's switches and routers, not through their SMTP server.
To impose a tax on certain kinds of TCP connections would require detailed inspection of outbound packets. This is because a single SMTP connection can involve the transfer of many messages. To be reliable, the ISP would have to parse every outbound packet bound for port 25 on a remote system in order to count the number of emails sent. I don't think most people want that level of attention paid to their private emails.
Moreover, this presumes that all ISPs participate honestly and thoroughly in such a system. All it would take is a few spam-friendly ISPs (and they exist, are legion, and jump around IP ranges like ferrets on a hot skillet) to render such a system useless.
The alternative would be to implement email billing at the recipient side. Maybe AOL and Earthlink can pull that kind of blockade off, but small companies and J. Random Luser cannot.
Bernstein's IM2000 proposal at least keeps the bandwidth consumption down, but that's primarily a cost issue for ISPs. (Don't try to convince me that if the amount of spam declined, ISPs would lower their prices.) The main hassle of spam for the user is that it takes time and energy to delete spam, and having to inspect the stuff with ambiguous could-be-from-someone-I-know subject lines would not be alleviated by IM2000; you'd still have to pick and choose what pending inbound email to read or delete.
The fundamental problem with email as a mail system is that it's open to anyone who wants to send mail -- which is part of the point of mail in the first place -- but there is no economic limiting factor for the sender as there is with paper mail. Since we can't eliminate the openness without destroying the utility of the system, the only possible strategy is to artificially impose a cost on the sender. Unfortunately, owing to the nature of public networking, the only remotely reliable way to do that would be to route all mail through a centralized clearing house. No one company will be able to establish such a monopoly, and I don't think anyone wants the alternative -- which is to have the government do it.
This may or may not be a soluble problem, but it is, as of today, still an unsolved problem. Personally, I think it's going to take national legislation and international agreements to stop it, and that will no doubt take a long time. Paper (actually clay tablet) mail existed for several millennia before the International Postal Union was finally established. Let's hope email is brought into line a little faster than that.
Proud member of the Weirdo-American community.
If the sender has to store the message, then suddenly they have to be accessible.
That's important. A reachable IP makes identification easier. Besides, the sender is also responsible for storage and bandwidth, and those are costs I can bear for my personal correspondents, but that would move spam a little further from free.
The question is a better one than I thought when I first read it. Although it doesn't sound like IM2000 can guarantee a low enough level of spam by itself, I would not only install a new client, but actually serve a new protocol if it was a way of delivering email without spam.
Assembly is the reverse of disassembly.
D. J. Bernstein, the author of the supremely reliable and secure qmail mail server, wrote a proposal for a new Internet mail system a couple of years ago.
The key words there are "a couple of years ago." Yes, a few years ago he published that web page. And you know what happened as a result of that?
Absolutely nothing.
Why?
Because it makes no sense whatsoever.
Bernstein came up with a lot of stuff over the years which did make sense, and which did take off, and became accepted. This isn't one of them.
So what if the mail's contents are now stored on the sender's system? How's that going to prevent a spammer from keeping a copy of the spam stashed somewhere, and then spamming a few hundred million mailboxes with a link to the spam. So, what have we accomplished here?
The problem with spam is not where the bits and bytes that make up the spam are physically stored. The traditional definition of spam is "unsolicited junk E-mail". Jiggering the bits around, and saving them in a different directory, or on a different server, will not magically turn unsolicited junk into solicited E-mail.
Well, I think those "loopholes" are all the zillions of features that people want, but they aren't good enough software designers to realize that the features belong in the layer above SMTP.
In particular, lack of authentication is a strength of SMTP, just as it is with IP. It means, for example, that I can implement my own authentication (or plug in PGP or whatever), and don't have to use the mail-transfer layer's after it turns out to have a serious hole that lets the spammers and con-men through.
Protocols that try to do everything for you have the inherent problem that, when a serious problem arises, you have to put up with it until the idiots at the vendor decide to solve it.
SMTP is simple enough that even a relatively incompetent programmer can do it correctly. You can type it yourself via a telnet connection.
And adding features in the higher layers is easy.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
SMTP works for it's intended purpose lets extend it to SMTPv3 (I think it's been extended once allready) So here is a general brainstorming.
If you exent it two servers with the extensions will send mail in the new more secured format.
Frst things first keep it friendly you should be able to do everything via telnet because it jsut makes testing easy.
So what do you need a little bit on how to get into this new mode once your connected to port 25.
The server must offer what encryption methods it allows as a list more perfered methods first. Unencrypted should be an option all senders and receivers MUST allow it as encryption is nice but CPU heavy and you can allays depreciate enencrypted senders.
There should be a DNS entry for the sending mail servier in the domain that the from address and the reply to address originate (some new DNS field well it's a nice big distributed DB with cache so why not?) This needs more work and it sorta outside of scope.
If the sender domain is part of the servers domain of responcibility the server must use the from and replyto addresses to authenticate the user(s) passwords via CHAP, Kerebros etc this MUST be done after the encrypted state is up and can NOT allow unencrypted passwords and perferable uses a CHAP like system where the password is never on the wire.
The receiving server must include all options specified by the sending server as message headers.
The server MUST only accept mail that is destined to it's domains or source from one of it's domains if accompinied with valid credentials.
A server to server intermediary authentication may be implemented.
OK thats no where near completed but it's a start.
No sir I dont like it.
SMTP already has authentication, and anyone who operates an SMTP server is free to accept or not accept mail from whomever he wants. You don't need a new protocol to require mail to be authenticated. If you can solve the trust problem, you can implement a trusted mail solution more quickly and easily with SMTP than by requiring deployment of an entirely new protocol.
Ever heard of it?
That is a very good point. Making spamming costly will not stop all spammers. Those with a legitimate business will still spam because there will still be enough legitimate customers out there who buy based on their spamming.
But making spam costly will indeed stop the majority of spam that is sent today, which is useless, annoying stuff that far less than 1/100 of 1 percent of people actually make a purchase based on.
That is the kind of spam that I really want to stop, and I think that making spamming cost money will have the desired effect.
In my mind I see a smooth graph of how much it costs to send junk email versus how many companies will send junk email. Right now we are at the end of the scale where the cost to send spam is almost nothing, and so the number of "companies" that will send out spam is HUGE, and not-concidentally the content of that spam is very poor.
We need to work our way to the end of the scale where spam costs real money to send, and only a moderate number of legitimate businesses then find sending spam worthwhile. And the kind of spam which is sent is then would be less annoying (to me anyway) as well. Spam would be more like advertising of actual products than make-money-rich schemes based on trying to sell snake oil.
That's not the answer either. Microsoft, Yahoo, et al have been lobbying for this approach, and for good reason. They want to function as the certificate authority (CA). They want to determine who can or cannot send email. They can use that power to literally sell the ability to send spam. They can also use that ability to censor their opponents.
Microsoft also wants a new patented standard that can't be legally implement with open source software.
While we're at it, why not PGP sign with the server as well? Signed mail from allowed server or recipient could pass through the spam filters while everything else could be sent through much more stringent filters. This way all of your family using hotmail could still reach you, which mail with spoofed hotmail source addresses could not. If nobody is willing to host the key servers, each mail server could provide its public key with an incoming connection. Also, clients sending mail should do so via an extension to POP3 or IMAP to end the nightmare of auth/relay configuration and clean things up further.
Whatever is implemented must be allowed to work with the current SMTP during a transition phase and must prioritize itself in some way (otherwise the spammers will just keep using SMTP).
There is a key difference, in that IPv6 affects the hardware of every net-connected computer (not to mention router, hub, switch, etc) in the world (I believe), whereas changing the mail protocol should involve minimal hardware changes, but admittedly extensive software changes. It's easier and cheaper to change software than hardware.
So... instead of a friendly hello, the mailserver should instead answer for an incoming connection request with a gruff "nuqneH!" (Klingon for "What do you want?")
or SMTP Mail Transport Protocol in the GNU tradition..
I don't think the /. community could do this. Why? Too many idealists. Look at all the "successful" protocols (HTTP, POP3, etc) - they all are loaded with problems, but regardless, they get the jobs done and where appropriate, get fixed over time. A pragmatic approach is required IMHO - something that does the job and that a large group of people could agree on. Pragmatism & consensus are not things the /. community are renowned for.
Read reviews of shopping cart software
As one of the posters in this thread pointed out, it would go a long way towards helping establish accountability on the part of spammers. If you have to be accessible in order for "recipients" of your mail to be able to read the mail, then I suppose you'd have to be pretty easy to track down and also sue under the current anti-spam laws.
... I've been using it on my server for years and really love it. I'd be particularly interested in hearing what your ideas are about the ways that we can solve the spam problem, given that you are the author of a complete modern mail system.
Anyway, it's just an idea. I think that the whole question of "should we come up with a new mail protocol" is kind of misguided because obviously the hard problems here are not technical, they're in dealing with the huge momentum built up around the current mail technologies. It seems to me that we already have all the technology that we need to solve this problem, it's more a matter of enforcement by ISPs. I was just posting the link because it was relevent.
If you're the same Mr. Sam that made Courier, then thanks
Keep using SMTP... But after the message is recieved, and the connection dropped, have another mechanism connect to the senders server (from the MX in DNS), asking if it sent this MD5SUM(message), on this date/time? If so, let it through. In not, check the user's preferences to see if they allow non-checked emails, and processs accordingly (place it in a users subfolder, forward to admin, whatever...)..
This doesn't break anything as it stands now, users/admin can choose how its handled, and should be fairly simple to implement.. There would be an overhead cost of keeping track of the MD5's... But it could be done...
Just an idea... Waiting to be shot down...
Slashdot is like Playboy: I read it for the articles
The simplest such policy is a whitelist -- but this means you can't just give your email address to a friend, and expect her to be able to send you mail. It means that if your friends change email addresses, they can't just send you email saying "This is my new address -- my old ISP stinks!"
A more complex policy might include some public key infrastrucure, where a user needs to have a valid key to sign their messages, in order to send mail. This brings up "who do you trust" in terms of who can sign message-signing keys, but more importaintly, who is going to have more trouble with this, a spammer that wants 100 throw-away accounts with keys per week, or someone's Mom who just wants to type a message and click "send?"
I think that it will be little use in trying to come up with a protocol solution to spam, without first defining what security policy you need to enforce. Once you do that, you can design exactly what you need to make your new policy work. (You might even find that SMTP is not at fault -- you might just need a layer above that transport to secure your mailbox.)
Can't we use Karma?
Simple premise - everyone in the world signs onto the 'KarmaMail' service, and get to send mail at "1". Once enough KarmaMail users validate the user's email as being legitimate, their Karma goes up. Registered users can also complain about a spammer, thus making their Karma go down. Marking email messages as 'urgent' requires a higher Karma. Users with a negative Karma (>= -5?) can only send at '0'. Users with a very negative Karma get booted off the system.
Then individual users can use Karma plus Whitelists to decide who to read mail from. Whenever a server receives mail, it checks with the central KarmaMail repository and inserts the user's Karma into the mail headers (optionally, Karma can be assigned to the *server* as well, eliminating the open relay problem). The header can then be processed by the mail reader.
Maybe someone would care to expand this idea further to clear up the many loopholes I've left?
You can accomplish anything you set your mind to. The impossible just takes a little longer.
SMTP is not the problem. Open Mail Relays are the problem. As long as you can drop an SMTP box on the net and have it spew Spam to other mail servers you will have Spam. Any new Mail Transport Protocol will have to be backwards compatible. i.e., receive mail from old SMTP servers. You can't switch all machines over at one time, you need to roll out one box at a time and keep it all working all the time.
:-)
In the country where I live there is a general rule for farm animals, the farmer is not responsible for fencing them in, it is your job to fence them out. Mail is the same, its not my job to stop spam being sent, but to stop it being delivered (to my users) There are many ways to do this, a combination of a few can be very effective.
As for the home user, well stop buying into the "submit your e-mail and we will send you porn" forms on the sites you wife does not want you to look at.
There's not a lot wrong with SMTP. The trouble is that SMTP is always implemented so it delivers mail as fast as possible. And that's the problem.
Judicious teergrubing (intentional slowing of responses; teergrube is German for tarpit) can alleviate many problems.
For example, let's examine the Rumplestiltskin attack (a form of dictionary attack to guess e-mail addresses). The trouble here is that most mail servers send back their "No such account" response immediately, so an attacker can try about 5-15 addresses a second. If the mail server was programmed to wait 5 seconds before sending back the response, then the Rumplestiltskin attack would be slowed down by about 50 times. Even better would be to make the delay longer and longer for repeated attempts from the same IP. This way, a normal user with a couple of dud e-mail addresses is not harmed much, but the Rumplestiltskin attack eventually gets bogged down in the tarpit. We have a 3 second delay at the login prompt if we enter the wrong password, so why not a delay at the mail server for incorrect e-mail addresses?
Another way to slow the spam is to teergrube *all* e-mail connections so all email takes a few minutes to send. Legitimate users aren't harmed much by this, but spammers are hurt a lot. Spammers rely on speed to send all their e-mail, and if we slow them down we can hurt them.
Then there's the question of what happens if a spammer sends another RCPT or other similar packet before receiving the response from the first? SMTP can legally drop the connection because such command buffering may be "unsupported". So the spammer must be teergrubed or must experience a *lot* of dropped connections.
There's no need to replace SMTP yet. Instead, we use the tools we have in a slightly different way, and the spammer can be inconvenienced a lot.
For more information on teergrubing, go here.
The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
I come to this discussion as an expert, albeit a bit dated, as I spent a number of years as the lone software developer supporting ALL email software at Apollo Computer (before it was bought by HP).
There once was a very interesting competing standard from OSI, the X.400 standard. Most people now think of X.400 as an interconnect standard for bridging the various email systems out there. Yet, it actually is a specification for a very robust email system in and of itself. It is based on a self-describing data representation... no, not XML since XML wasn't even a twinkle in someone's eye at that time, but ASN.1. That standard has been somewhat successful as used in X.500, which has become somewhat popular through its exposure via LDAP.
SMTP has never been a particularly strong standard. First, it is not the specification for a complete email system. It mearly describes a protocol for exchanging messages between two processes via the network. This is not sufficient to build an email system. Thus we also get POP and IMAP, and any number of supplimental bits that are not necessarily standards. Even sticking to exchanging email between two processes, SMTP has always been rather loosely specified. Sendmail has served as the reference implementation. Supporting sendmail was more a matter of figuring out what it was doing than reading the SMTP specification since sendmail used a far richer protocol for exchanging email than described in the specification. Thus, the question of what comprised a compliant implementation was more like (does it interoperate fully with sendmail) than going through a specification and checking off each element it described.
Apollo started a project to produce a native X.400 email system. It had a very rich set of features that go far beyond what we see today in Unix and Windows email systems. The project was put on hold when I was reassigned to a higher priority task, I was a member of a strategic technology team given the task of determining what "everyone" meant by the term "CASE Integration" with the goal of producing a corporate strategy and piloting and/or prototyping some initial products. Given the state of the CASE community, it sure seems like pursuing the email strategy would have had better long term success. Of course the CASE Integration project died a painful and horrible death when HP bought the company. Surely "SoftBench" did everything and more...
Check out RF2821.
This from a guy whose .sig is only one step above spam.
-a
The key to solving the spam problem is to attack the definition of spam: "unsolicited junk E-mail." All that's needed -- really -- is a simple convention that may be used to designate whether a given E-mail message was solicited by its recipient, or not. That's it.
For example, you see one of your own message IDs in the References: header of an incoming message. That tells you that it's a solicited response to one of your own messages. Unless you're a troll, presumably each one of your messages merits a response from an interested party. Additionally, by the virtue of sending the original message you've implicitly solicited appropriate replies.
Now, I don't think anyone needs to keep track of every message they send. This is just a conceptual example of identifying solicited versus unsolicited E-mail. Once a method for doing so can be worked out, and put in widespread use, the spam problem will be gone.
This is currently being discussed on NANOG (where it's an offtopic favorite). I highly recommend this list for peeks and views into the people who keep this Internet thing working.
In the discussions yesterday and today, there's been a lot of talk about how to "bootstrap" this new protocol. There are interesting discussions of the business ramifications of being an early adopter of something like this -- very sililar to those for IPv6.
It's been said by far wiser people than me: spam is a social problem, and it must have a social cure. Any solution which does not respect these two facts is doomed to failure.
There is a solution on the table and US law that requires the US government to use it. Its called X.400 and it is a mess. For a start you have to register your server and that used to cost something like $25,000 or maybe $40,000 for businesses. The Gossip program for gov email requires all email systems to migrate to this x.400 nonsense but I manged to get them to allow a migration path through SMTP (the others were worse and the only two that were even consididered that worked were SMTP and UUCP). The only encrytion addon for sendmail happens to be a result of work that started from encrypting x.400 stuff.
If you want to fire up your own X400 server to play with, grab isode and try to get it to compile on your machine without gagging if you can. Its one nasty bit of bad code.
SMTP isn't that broken. It works for about a billion people. Any attempt to "fix" it will break it for way too many of them.
After looking through the posts here (most of the +5 should be -5 Stupid), its clear that most of the experts don't understand email in the real world.
Encryption:
The 1st tings is email must be interceptable. Many governments won't allow high level encryption that isn't full of holes that allow them to play pack recorded streams. Most large email servers can't deal with the CPU load of full encryption anyway so 100% solid encryption is out.
Authentication:
Authenticating the server is very importaint to many sites. Once you start doing some level of encryption, you need to make sure you know who your connecting to.
Authenticating the client is the where spam issue comes from. There are many ways to do this but none of them are being done and none of them work 100% (which is why none of them work)
There is no way of knowing of a new business is a spamer or not. Therefore there is no way to filter out spamers that have enough cash to hook up to new ISPs all the time. (there are some stupid ideas like charging--my isp is rich enough, forcing all email out--my isp's mail server is up 100% doesn't understand MX,I can run my own server and it works so why chnage?)
reverse MX record checks only work if you can trust the ISP to get reverse dns working correctly and they won't deligate it to a spam house. The other choice is a verisgn like company to whitelist everyone or some sort of distributed whitelist (which the spamers will try to hack into)
As far as fixes:
The solution is patch sendmail, qmail, postfix, exim to understand email on port 26 (pointed to by a srv record) and if mail comes in on the new port, then it must be checked with a reverse MX record or its dropped. Get the clients to stop handing off email on port 25 (sendmail allows port 587 for that) Use something like the SSH transport layer to encrypt (i.e. set up the encrypted channel 1st and then figure out whos talking). Add a new smtp verify_message command so I can ask another server "did you send me messages Xcxczxczqweczx?". Patches for all 4 systems must come out at the same time but be tested aginst each other. The when an ISP figures enough of its mail comes in on something other than 25, kill port 25 forever. That will kill all the proxies and all the old email gateways that haven't been updated in years.
Or save up your money and buy your self an X.400 gateway license adn tell all your friends about your cool new email address with all thouse nice slashes and no @.
A combination of white lists/black lists, and Baysian filtering...
Stop yer bitchin', people, and implement the technologies that are already out there and work great.
This doesn't sound like a general solution for J. Random Homeowner.
It sounds alot like "Well, shoot! Quit yer bitchin' an just put in some tuned ports and performance cam! Hell, my grandmother could do that!"
My other car is a 1984 Nark Avenger.
I don't risk job loss (since I'm self-employed), but I do risk missed contracts. I use Bayesian. I've had an occasional false positive, mostly during the first couple months when I was building my corpus.
BUT: I'm currently receiving about 140 spams per day. Do you really think I'm going to do any better than Bayesian regarding false positives when I'm mass-deleting 140 messages? I think not. I'm almost positive I'm going to incorrectly delete more good messages than Bayesian's false positive rate when I'm dealing with such a huge number of emails manually.
Plus if your Bayesian system is half-decent you should be able to adjust its sensitivity. On mine 0-49% is almost always good email and 50%-99% is almost always spam. When I look for false positives I look for anything unusually low, such as 65% or so. If you want, just adjust your threshold to 65%. I've never had a valid email with a Bayesian score of, say, 90%.
> They either can't figure out the tools or don't think they should have to.
:-)
And this is the thing - they really _shouldn't_ have to. Bad UI really ticks me off.
> I agree, however, that people are generally naive/dumb when it comes to common sense issues like sending out email addresses at will or even worse...
The thing is - I intended that for this particular audience. _SLASHDOT_ users, of all people, should know by now how to avoid getting spam. I mean _really_.
> clicking on the "Remove" links from spam! VERY DUMB!
Actually, I've proven to myself this is a myth.
Here's my story:
Last year, around, say, September or October, I was getting, on average, about 200-250 pieces of spam PER DAY. This, I realized, just Would Not Do(tm).
So, since it was obvious I was going to have to shut down all my existing e-mail addresses, generate new ones, and be ultra-selective about giving them out in the future, I realized it was time to test that "Don't click on the remove me links" piece of advice. I'd given it out myself many times, even in an article I once wrote. Time to put it to the test! So, for the period of one month, I followed all the instructions on each piece of spam, every day, to see what would happen to my flow of spam. I kept track of who was sending me spam (the company/product/service, not the 'return address'). I found out that you WILL get LESS spam if you actually follow the advice in the spam, in general. My spam reception went from the 200-250 per day to around 20 per day, in the span of about a month. Obviously, this was still way too much frigging spam, but let me say this: the spam I kept getting was almost entirely from the sources that didn't have a (working) removal method, not from the ones that did. Many of the ones that did have an (apparently) working method DO indeed take a few weeks to start working. But, surprise of surprises, it CAN indeed lessen your spam, when it's offered and is working. Bizarre, I know, but I swear it's true.
I'd still rather make spam technically impossible than rely on that, though.
I propose TMTP - the Trusted Mail Transfer Protocol.
I have a technical website where users can subscribe to a nightly mailing list that sends them a single email containing all the messages posted to the forum in the last 24 hours. The users subscribe by signing-up with their email address. A single email is then sent to that user who is then asked to click a link to confirm they want to receive the nightly mailing list.
Now, I've heard people propose that there should be a system whereby an "unknown" email is charged 10 cents (or whatever) unless the receiver subsequently tells the system "Yeah, I wanted that" at which point the 10 cents is effectively refunded and, presumably, all future emails from that source have the charge waived.
The problem is if someone starts signing up random email addresses on my mailing list. Each time the system sends out the single confirmation email I will be dinged 10 cents with the expectation that everyone will "refund" that back to me since they asked for it. But what if someone with a grudge and a lot of extra time goes to my site and starts signing up lots of people--people that have never heard of my site. The "confirmation email" is precisely to protect users from getting on my mailing list without asking for it, but I'm not at risk of having to pay to have that confirmation email delivered.
If such a system were imposed then how would a mailing list be able to send out confirmation emails without the risk of someone maliciously signing up random users just so that I get hit with a bunch of email charges?
If you do trust the applet (man, I should really verify the results to be sure
1. You stay on the home row more.
2. You alternate hands more.
3. You change fingers more.
4. Your fingers don't travel as far.
Well, whatever the reason, I feel more comfortable typing on a Dvorak layout
It's not so much the storage, it's the bandwidth that costs, I think.
Sure, they can store one email and send out millions of notifications from "different" addresses.
But as soon as people start reading it, those networks are going to start getting really busy.
A self-slashdotting type effect.
- Muggins the Mad
Ok, so that requires additional steps. Before the user signs up the user must be made aware of the address from which emails will be sent so that it can be whitelisted BEFORE the verification email is sent. A new business plan for spammers wold be to cruise or spider the net and gather the email addresses of common mailing lists and use that as the "From" when sending spam to increase the probability of using a whitelisted, toll-free address.
If you're talking about whitelisting an email address AND an email server that's fine for tech people but would probably a bit complicated for normal users... and for lazy technies.
Implementing it on a recipient-controlled basis is not only politically correct anarcho-capitalism (as opposed to central-planner-based and doomed to failure), it matches the main problem with spam, which is that it wastes the recipient's time, and only the recipient knows what that time is worth., and the recipient of the mail is the one who gets the money, not some middleman whose time isn't being wasted. If I dislike spam more than you do and am more willing to reject mail from people who might have interesting things to say, I can charge more money to read email than you do. Yes, email-provider ISPs also have higher workloads because ~50% of email is spam, but that's a much smaller amount of money per recipient than the wasted time, and they can charge the users accordingly.
Of course, if the recipient of the email gets paid for receiving the email, there's an obvious product for spammers to sell "Get paid receiving email" ;-)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
A combination of white lists/black lists, and Baysian filtering stops so close to 100% of spam that it's really silly for anyone to be bitching about spam these days. I don't GET any spam anymore - 0. Not 0.001%, 0 - the integer 0, as in none.
Have you done the power-curve analysis on that? My mother works at a law firm, and they once tried to install a spam filter. It was state-of-the-art, with Bayesian filtering, and white/black lists, and additional whitefilters on top. It blocked most (not all) of their spam. But it also blocked some tiny fraction of legitimate messages.
Even if you have the (extremely impressive) power curves of Paul Graham's Plan for Spam -- and that was on a very well-trained Bayesian filter written by a coding genious -- it is Not Good Enough when missing a legit email could get you sued for millions. Either you risk blocking legit email, or else you have to wade through a pile of spam bigger than that legit email... either way, another protocol would be nice.
I hereby place the above post in the public domain.
The people who design a new mail protocol should, at the very least, all know the current one backwards and forwards.
For most people, unsolicited email isn't the problem. Unsolicited bulk email is the problem. Generally, unsolicited, personalized, individually sent emails aren't sufficient to bother people. ... Any solution which depends on security through obscurity (hiding my email address) or refusing unsolicited personal email from strangers, is unacceptable to me.
Ahh, but if your E-mail address is out there on some web site -- together with some arbitrary content -- and someone chooses to send you E-mail after reading said contents, and wanting to respond to you for some reason; then, well, from where I'm standing it looks to me like you've solicited that response, didn't you? You certainly didn't solicit some spambot sucking it out of the HTML, and then feeding it to a Viagra spam engine, of course. But, by the virtue of signing your name+E-mail address to some article, I would argue that you've solicited individual readers to drop you a note after they've read the article, if they felt like it. If you didn't want, or solicit, people to reply to your Slashdot posts, then you simply leave out your address, that's it. What good would posting your E-mail address here if you don't want people replying to it?
I mean that's what an E-mail address is for, I think: for people to know how to get in touch with the author, regarding the written subject matter.
Similarly, by signing your name to your Slashdot posts, I would argue that you've solicited, or invited, E-mails from anyone who've read them and wanted to give you a piece of their mind, for some reason.
I frequent other web discussion forums, where I do NOT use my E-mail address. That's because I really don't care for any replies on the subject matter over there. Here, if someone wants to flame me away for something -- go right ahead. I'll bitbucket it, of course, but I wouldn't say that it was unsolicited.
So, I think, it all comes down to "solicited" vs. "unsolicited". It's rather impossible to give a precise definition of "bulk". What is "bulk"? If "bulk" means a certain amount of substantially similar messages, then there has to be some number X where X is bulk, but X-1 is not bulk.
So, what is X, then, and can you provide a valid, cogent argument why X is bulk, and X-1 is not bulk?
I don't think this is going to work. I don't think you can make a common-sense based argument for bulk vs non-bulk. But I think an argument on solicited vs. unsolicited can be made, based on a common-sense definition of "solicited."
It appears that your definition of "solicited" means something that's exclusively a response to some previous written screed of yours. I think that a more liberal definition of "solicited" works better: meaning "did you reasonably expect to receive this kind of a message." Obviously, from the fact that you've posted a message of your own to a discussion group, with a valid E-mail address, it can be reasonably inferred that you've asked -- hence solicited -- replies. But, at the same time, if you put up an average home page, where you wrote that you're a graduate of East Side Nerd High School, and if you've provided your E-mail address, then it can be reasonably said that if an old buddy of your from East Side Nerd High accidentally stumbled across it, his E-mail wasn't exactly unsolicited. He didn't just sent something, addressed to your E-mail address, by random. It was in direct response to what he read -- hence it was solicited.
On the other hand, if some spambot lifted your address, and started sending you Viagra spam, then it can't be reasonably argued that you've solicited Viagra spam simply by the virtue of describing your high school follies.
Now, you might think that this approach is not going to work because "reasonable" is in the eye of the beholder. Something may be reasonable to one person, but not reasonable to another person. Spammers will argue that using your E-mail address on slashdot can be inferred to reasonbly means that you want
A combination of white lists/black lists, and Baysian filtering stops so close to 100% of spam that it's really silly for anyone to be bitching about spam these days. I don't GET any spam anymore - 0. Not 0.001%, 0 - the integer 0, as in none. If I ever get another piece of spam, then I'll change my email address
[...]
Stop yer bitchin', people, and implement the technologies that are already out there and work great. Plus use yer freakin' brains for a change, and don't spew out your real e-mail address to everybody who asks for it.
Don't make claims that the current technologies are "so close to 100%" if you're not willing to put your money where your mouth is. Post your email address far and wide, then see if you still don't get any spam. Otherwise, don't get up on your high horse and tell people they shouldn't worry about spam.
Security through obscurity sucks. We need a better solution for spam.
Deven
"Simple things should be simple, and complex things should be possible." - Alan Kay
For a measley $20 or so a year, you can register your own domain name, get all e-mail to that domain forwarded to your "real" address (which you make unguessable and never give out), using a service like Zone-Edit. I've done this and effectively cut off ALL spam, since I give out a different e-mail address to each entity (usually something like, "[company-name]@[mydomain].com").
When I discover a piece of spam, I check the sent-to field, and set a rule in my mail program to color it a certain color using the sent-to field as criteria (or transfer to trash), since I know that this single address has been compromised somehow. Since you can clearly know WHICH e-mail addresses are compromised, it makes it very easy to filter these out.
> I wish people would stop inviting rate increases or new charges as an answer to spam. It's not the answer.
But wait. What if ISPs simply charge each other for traffic depending not on the direction of traffic but depending on which side initiated the TCP connection. That way the person downloading from a web site will be the one paying (because he made the HTTP connection) and not the web site host. And the person sending the email will be the one paying (because he made the SMTP connection) and not the recipient.
If only a few big ISPs agree to work like this others will follow and soon even small ISPs will start charging their customers for traffic based on this method.
Could this help to put an end to spam?
It is probably the case that SPAM is a real "tragedy of the commons". However there are two separate issues. One is the end recipients problem, our problem, the pain of dealing with SPAM, and the cost in terms of the extent to which it costs us bandwidth money or degrading our "service". There are a number of different more or less successful methods for dealing with this and we can all do it on a case by case basis to scratch our respective itches.
But the more important problem is the impact that the problem has on the overall infrastructure of the net. For example LINX, the (one of?) the major link points into the UK, recently reported some metric about how much spam was coming down the lines. (Sorry can't find the reference). It is when the concentration of this rubbish gets so higha s to affect this level of infrastructure that we all have a problem.
Now the tricks required by the spammers to try and get around the filters may preclude some of these ideas, but the idea that I should receive 'a copy' of Jane, or Sarahy or Ken's Viagra offer, ie a copy of the same mail that was sent to all the people at my mailhost (this is particularly important for big providers like AOL or BT where they have millions of customers) rather than an individualised one would certainly reduce the strain on the underlying infrastrucutre. How would one achieve that? Good question, I haven't really thought about it enough, but I am not suggesting an economic answer, I don't want to try and coerce the spammers, let's try and make the solutions in their own interests before we use the price tool.
However having rasied the price question, it is critical to remember that the reason why the junk snail mail, SPAM comparison falls down is that the marginal cost of increasing the distribution list of a SPAM is zero and the marginal cost of increasing the distribution list of JUNK is (in addition to postage) the cost of the JUNK. This provides a natural price imperative to the mass mailer. Can we introduce such a cost to the spammers. Er, well um no. Not that it is hard, ot is just impossible. Think as hard as you like and anything you come up with will be flawed in some way. So don't try. Just accept it.
So, we must return to the politics of SPAM. Clearly the sender of the spam must be identifiable. Perhaps it must be identifiable from the "International register of spammers" what ever that might be. SPAM not sent by one of these servers is automatically dropped. Then one must decide how one determines what is SPAM and there is probably the rub. I don't really know maybe a global bayesian filter?
Anyway. The point is that the first problem is to contain SPAM, make it in the spammers interest to identify their mail as SPAM. From then we can manage the SPAM bit by bit. Of course the obvious solution is _don't buy from spamvertisers_ but then it only takes a few idiots to make that strategy infeasible.
"The first thing to do when you find yourself in a hole is stop digging."
SMTP is a rewrite of FTP. The original MTP was actually just a feature of FTP. You moved email in those days as if it was a file.
FTP is itself actually written as a feature set for telnet. You would log into the remote machine and then tell it to move files about.
FTP has been rewritten, it is called HTTP. The only reason HTTP exists is that FTP is horribly inefficient if you only want to move one file and provides no metadata to describe what the file might contain.
There is plenty of cruft that could be removed from SMTP, features that should never have been there. There are also features that could be usefully added, like ways to reliably authenticate senders at the transport layer. And no, STARTTLS and RMX do not meet that need, PGP and S/MIME are message layer and both require the full message to be read before any decision is made to accept or reject.
But none of this requires a complete ditching of SMTP and start again protocol. If the industry did think it should go that way it would choose a Web Service based protocol layered on the WS-Security stack.
Sure folk can write their own mail protocol, just like anyone can write their own programming language. Just don't expect anyone to use it unless you start off with a commitment to implement from the major mail server vendors. That would be Microsoft and Notes.
Making changes to the email protocols is very hard. The IETF has been completely unable to take any sort of lead in this respect. In twenty five years they have only revised the basic mail protocols once and that working group started with a mandate not to actually change anything. RFC2821 is an improvement but nobody is actually listening at this stage. The ASRG group is not even in the IETF it is in the IRTF, the R is research. At this stage the group has alienated every major vendor that has ever participated.
There is actually one change to the mail infrastructure that is emerging but it is not a rewrite of SMTP, more a sort of supplement. One of the things that SMTP does worst is management of mailing lists. Sure it is great if you want a mailing list, but what is really going on in a mailing list is push publishing. A dedicated push publishing protocol could eliminate a lot of the pain in mailing lists that comes from the fact that there is no real concept of a 'subscription' in SMTP. But again, that is not necessarily another protocol, probably just adding SMTP support for RSS.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Filters are -very- expensive (both for the computer, you and me), and a fee-per-email system is silly, and does nothing to actually control spam.
The only really effective way I can think of is another fscking registry. ISPs and companies large enough to really need external relays pay the fee to register their mail-server there, and the new implementation of the SMTP-protocol only accepts external mail from other listed servers.
The downside? A fee comparable to the price of a domain name for ISPs, companies and stubborn individuals. Don't give me the old crap about "having to run your own relay", because you still could, by in turn having it relay through your ISPs server. Your ISP doesn't provide you with a relay? After this, they would have to.
The upside? It would be a lot easier to blacklist spammers. No more hijacked boxes on broadband-connections flooding us with spam.
Oh, I know, it will be shot down because there's a fee involved, but keep in mind that I would be one of the people that would have to pay that fee, and it would be a very small price to pay to protect myself and my users from spam.
So why not just stick with the whitelist, and not worry about the 'pay for mail' part?
You've never wanted to e-mail somebody who doesn't know you personally?
How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?