The "packets of 576 bytes can't be fragmented" is a commonly stated reason, but it is wrong. It is a myth/misunderstanding. It is, in practice, true has has been true since probably the late 1980s, but DNS was around long before that. Indeed, if you read some of the earlier RFCs, it is quite clear that packets of any size could be fragmented, down to something like 16 bytes of payload per fragment.
No,the reason for the 512 byte payload size is much more basic than that. Back in the early 80s, memory was tight, you could have mainframes supporting dozens of users on a machine with maybe 1MB of memory, each of user could have more than one active network connection. IP supports packets sizes up to around 64k, but it would be unreasonable to expect every host to be able to accept such a large packet size. It would mean that they could get fragments from all those packets piecemeal and out of order, so reconstructing each packet would require holding lots of 64k buffers, each of those buffers would be 6% of all available memory.
It would be very unreasonable to expect every host on the internet to be able to accept any size packet, even if those packets came in fragment that wouldn't saturate your connection. Now, protocols like TCP have the ability to negotiate the packet size, but for UDP, it gets messy and slow. So, it is a *requirement* that each host on the internet can accept a packet with 512 bytes of payload. That packet can be fragmented, but it has to be accepted.
Intelligence as a requirement for voting has been fought for a long time see voting tests.
There is a certain amount of irony with you saying this, followed by your.signature of:
Knowledge = Power
P= W/t
t=Money
Money = Work/Knowledge so the less you know the more you make
This "joke" is clearly aimed at people who think they understand math/physics/science, so it won't be funny to most people. But, it also shows a complete lack of understanding about how equations should be interpreted. What the formula "money = work/knowledge" says to increase the amount of worked done, you need either more money or more knowledge. In other words, "the only substitute for knowledge is money", or "a fool and his money are soon parted".
You are a case of "a little knowledge is a dangerous thing". By your own statments, you shouldn't vote.
The big problem with DNSSEC, if widely used, is that it prevents forgery of DNS responses. ISPs and internet cafes will not like this, since that means they can no longer forget DNS replies to missing domains or to force people through registration pages. I can see a *LOT* of push-back from having end-users using DNSSEC.
Trust is the same for DNSSEc, it's just that instead of using the root servers as a trust chain, you use a 3rd party that every domain owners had to pay for.
DNSCurve does not require you to pay any third parties, it is like DNSSEC where you publish your own information. Both technologies are (or in the case of DNSCurve, will be) free.
DNSCurve is much easier to implement than DNSSEC and and also advantages in term of cryptography speed and increase of traffic.
DNSSEC has many years of actual deployment, not as wide spread as it needs to be, but it has been out there and tested.
Can you point me to a single implementation of DNSCurve? Can you even point me to a specification of what exactly it is? I've looked, and the best that I can tell, there aren't any. More over, it doesn't appear that DJB's website has been updated since he proposed DNSCurve last year.
DNSCurve is interesting technology, but it has many problems, not the least of which is that it is mostly hype right now. It does not really replace DNSSEC in functionality, but rather, it is closer to TSIG. That is, instead of securing the actual DNS records, it secures the communication between name servers and resolvers. With DNSSEC, you can get your DNS records for a totally untrustworthy server, and yet be able to prove if they are valid or not, but there isn't any form of encryption so there isn't any privacy. DNSCurve encrypts the transactions, but you can often figure out what is there anyway by watching which name servers you are contacting and monitoring other things to figure out what you were looking up.
I like DNSCurve, I hope it goes some where, but I also hope that DNSSEC takes off soon.
To the contrary, DNSSEC could possibly kill the goldmine that is the SSL cert racket. That is, unless having your DNS entry signed somehow becomes a "value added" service you need to pay for extra.
I'm a layman here, but glancing at how DNSSEC works, I see no obvious way selectively signing some but not the rest of entries could work. This means, DNSSEC would provide a more secure way to give the public key to a viewer.
You may be a layman, but you appear to have far more clue about this stuff than most.
Yes, once DNSSEC is deployed, anyone with a domain name can publish CERT records and have about the same security as a paid-for CERT. Granted the cert authorities right now require you to give your name and address and such, which publishing CERT records in the DNS won't require so they aren't exactly the same, but close enough considering how little checking the cert authorities do on such information
SPF policies apply only to the envelope sender address, not the message's From: header.
Most of the time, the email address in the "From:" header gets copied to the envelop "from". And, most importantly in this case, the envelop "from" is where bounces get sent to, so the bounces he receive could have been stopped if he had published an SPF record *and* everyone checked it.
If your an org that sends out emails from a 3rd party from time to time, (IE, surveys, or other crap), then you will have issues with SPF.
Both SPF and DKIM have problems with this kind of thing. With SPF, you need to include these third parties in your SPF record (possibly using the SPF "include:" mechanism). As you mentioned, with DKIM, you have to send them a key.
I've also heard about relaying problems in large companies, with decentralized email, ie, each divison or site has its own mail server.
Yes, tracking down all your mail servers is a huge problem for large organizations. SPF can help identify these sources by means of a "tracking exists: mechanism". This can then be used to help add IP addresses to SPF records or distribute keys for DKIM.
Just curious, why was DKIM chosen as an IETF standard and not SPF?
The IETF standard process runs by rough consensus. The IETF working group (MARID) that tried to standardize some of this stuff had everyone from Richard Stallman to Bill Gates trying to influence the outcome, along with a dozen proposals from "competing" technologies that would preferred no standard to theirs being excluded (this group included many folks who supported Domainkeys). SPF grabbed an early lead in deployment and mindshare by working outside the IETF, causing no great love with the IETF insiders.
It shouldn't be a big surprise that the MARID working group fail to reach consensus and was shut down without publishing any standards.
The folks involved with SPF (including me), and Microsoft both put forward their systems as "independent RFC submissions". These RFC submissions usually do not get put on the standard track and it could easily be argued that even being labeled "experimental" rather than "information" is a positive sign that the IETF sees at least some value in them.
By the time the DKIM working group was started, most of the hype had died away and a much smaller group of people were able to find consensus. DKIM was also backed by IETF insiders and most of the people from the SPF project that participated in the DKIM working group saw it as a complementary technology, rather than a competing one.
what does DKIM provide that SPF cannot provide?
For email sent directly from one party to another, SPF, Sender ID, DomainKeys and DKIM all work very well.
In general, SPF (and Microsoft's Sender ID) has many more problems with email forwarders. There are ways around the problem using things like SRS or what gmail does and munges through the Received: headers to see which ones can be trusted and are forwarders. DKIM (and DomainKeys), on the other hand, rarely fail on forwarders, although it does happen when the forwarder adds trailers, do mime-rewriting, virus/worm scanning, etc.
In general, SPF has few problems with things like mailing lists. DKIM (and DomainKeys and Sender ID), on the otherhand frequently have problems with mailing lists.
Those of use that see SPF and DKIM/Domainkeys as complementary see domainkeys as very useful in identifying trustworthy forwarders which can then be used to make SPF more reliable, and SPF as a way of identifying trustworthy mailing lists, which can then be used to mail DKIM/DomainKeys more reliable.
In the 1970's, I used a the CDC PLATO system, which looked more like the modern internet than the internet in the 1970's looked like the modern internet. It had instant messaging (term-talk), email (pnotes), forums (notes, later to evolve into lotus notes), and chatrooms (0chat). No one talked about one replacing the other because they were all good for different things.
In the early 1980's, I used IBM's CMS system. It had instant messaging (#cp msg) and email, but sadly, no forums nor chat rooms. People talked about needing the later two.
In the mid 1980s to the early 1990's, I used unix. It had IM, email, forums and chat rooms.
Since the early 1990s', I've used unix on the internet. It has IM, email, forums and chat rooms.
Now, in the 2000's, people claim that IM will kill email? Huh? I don't see it. Did these people never have IM before?
Before "correcting" Karl Auerbach, you might want to to see just how many google RFC's he has been involved with, not to mention being kicked off the ICANN board for trying to stand up for the individual.
... but data mining the root name servers would be next to useless. These are the Root name servers and as such all they know about are TLD (top level domains). You ask one of the roots "who is in charge of.com" or.edu or.uk, and they respond. The only data you could ever get from them is distribution among TLDs.
No, that isn't who DNS works. If a machine decides to send a query to the root name servers, they will send the complete domain name. The root name servers will then reply "I don't know the answer, try that name server over there". In theory, most machines should have the TLDs cached and not send the query to the root name server first, but there are a huge number of broken resolvers out there. The Measurement Factory has some published studies about just how much bogus crud gets sent to the root name servers, and there are a bunch of other studies that would require a little more work.
Seriously, yes, data mining the root name servers can be done. One of The Measurement Factory studies did just that. It could turn up a lot of interesting stuff.
Actually, wikipedia has a very good summary of when IPv4 address space exhaustion will likely happen. In particular, while the IPv4 allocation graphs made by Geoff Huston aren't as pretty, they are likely far more accurate than xkcd's. The only problem with Geoff's predictions is the exhaution date keeps getting moved forward so his dates are probably best-case predictions.
Basically, yes, the IPv4 space is running out. It is still 3-5 years out for IANA exhaustion and further for the RIRs and ISPs, but it is something that people need to start planning for. The predictions about IPv4 addresses running out back in the 90s was before the development of things like CIDR allocations, NAT, RFC1918 private network numbers, HTTP1.1's virtual hosts, DHCP, and the dot-com crash. There haven't been any new "gee, we can make the IPv4 space go a lot further if..." type ideas for years and it doesn't appear likely that any more large savings will happen before it is too late to deploy them.
Hey Wayne, multiply that $200/employee/year by, let's say, 50 million people in the US who use email in their workplace. Not such a small number anymore, is it?
It is still $200/employee/year, which is far far less than many other costs. The "spam problem" is no worse than the "heating/airconditioning problem", as far as costs go. Would any company seriously consider not heating/cooling their work place to the point of hurting productivity just because it costs a couple hundred per year per employee?
I guess I should have been clearer in my original post. The solutions to the spam problem have largely been known for many years now. What needs to happen is for wider implementation of those known solutions. Image-only spam has been around for years and it was predicted that once things like bayesian filtering became common, that this is the direction that spammers would move to. This is simply causing the anti-spam field to shift from "content-analysis" to the "sender's reputation" model. Only now, the industry has much better feed-back loops to let people know that a sender is sending unsolicited (or at least unwanted) email, and that they are sending it in bulk.
I never said that protecting your inbox from spam would be cheap, just the opposite, I said it would cost but that you shouldn't be penny wise and pound foolish. I also said that senders need to become much more careful about what they send, or they won't get their email delivered. However, as you say:
No, I think it's going to take some serious international strong-arming ("We're going to impose tariffs on your exports until you start arresting spammers") to deal with it this time. You can't use a technical fix for a social (or in this case, criminal) problem.
Agreed, this is another part of the spam solution. Law enforcement simply hasn't caught up with this new type of criminal activity.
My understanding is that botnets, mostly made up of weakly-secured home machines, are the source of the majority of spam.
Correct.
Thus the main problem is not network administrators not taking good care of their networks (which are usually quickly identified and isolated using blocklists), but rather the woefully insecure configuration of home desktop machines out-of-the-box.
No, it is the network administrators not taking care of their networks, in particular the network administrators that have home desktop machines hooked up to their networks. Just because they may call themselves an "ISP" doesn't mean they don't run a network.
These network administrators either need to make sure that their customers don't get infected and that they don't have spammers as customers, or they need to make sure that their infected machines/spamming customers don't hurt the rest of the internet. Chances are that these network administrators don't want to do either of these because it is cheaper to let everyone else on the internet block the abuse and then blame everyone else on the internet when their customers complain about being blocked.
I know people like to rant about the "spam problem" a lot, but for all practical purposes, the problem has been largely solved for several years now.
I work for a company that has about 700 employees. [... long story about how you solved your spam problem deleted...]
So my spam problem is solved, right? Yes and no. Spam is no longer crushing my meager inbound mail infrastructure, but I'm paying close to $14k per year to get out from under the crushing spam load.
That comes to $200 per employee per year. How much is your employee's time worth? How much would be lost on paying employees to filter their spam themselves or having lots of false positives?
This is exactly what I was talking about. You are being penny wise and pound foolish if you don't think that $200/person/year is cheap. As you admit, the spam problem is largely solved for you.
OK, but here is the important point. If everyone in the world took as much effort to solve the spam problem as you, and many others, already do, then spam would largely disappear and spam filtering wouldn't be so expensive.
For email senders having problems getting caught in spam filters, some of this is due to people running bogus spam filters and that is the receiver's problem more than yours. Most of the rest is due to either you not running a standard-compliant mail server on a static IP address that can have a reputation built up for you being a good server, or because you really do send out spam
No, false positives are a problem for everybody, and they are unavoidable. I'll give an example. I've signed up for the theyworkforyou.com alerts that email me when my MP speaks in parliament.
Let's see. The "theyworkforyou.com" domain doesn't have a valid rDNS pointer, something that the RFCs require for all hosts on the internet. They don't publish SPF records, nor use domainkeys.
SpamAssassin flags them all as spam, and I looked into why today. There are only two rules being triggered.
1. The "foryou" in the domain name is very spammy.
2. The Bayesian filter thinks what my MP says is 99% certain spam.
Which of these is the bogus rule? Should I stop using the Bayesian filter? Should I let in all the spammy "job4you" domains (there are a lot)? Should I tell theyworkforyou.com to change the name of their organisation?
If you are going to use things like Bayesian filters, you will need to train them. For spamassassin, you need to run an "sa-learn --ham" on a few of those emails. Many systems will whitelist senders that are in your address book, with spamassassin, you might need to add a "whitelist_from *@theyworkforyou.com" to your ~/.spamassassin/user_prefs. I also suspect that if theyworkforyou.com is getting that high of a spam-probability on the bayesian analysis, that they are sending email that looks spammy for other reasons.
Make sure your spamassassin is set up correctly. Enable the "network checks", some people turn them off because they are "too expensive". Also enable the auto-whitelist, even though that costs a little disk space and some people consider it to be "too expensive". Oh, and make sure you have a per-user dictionary for your bayesian analysis, not one for the entire domain. Again, it costs a little more, but it helps a great deal.
And, yes, picking a good domain name is important. If you are "Big Bank", don't go creating a brazillian different domain names, use subdomains.
I know people like to rant about the "spam problem" a lot, but for all practical purposes, the problem has been largely solved for several years now.
If you run reasonable spam filters, including many open source ones, you will not end up with much spam in your inbox. Yeah, there will be lots of spam still being sent, but the real, significant, cost of spam is really mostly people's time, not machines. Any ISP, company or person who gets "too much spam" is simply being penny wise and pound foolish. The same goes for systems that get too may "false positives", that is, legitimate emails being rejected. Almost all of that is due to trying to run "cheap" spam filters, or buying snake-oil systems. Upgrade your mail servers or switch to someone who runs reasonable spam filters.
The "spam problem" of today is really the "you can't do anything about spam" problem. Too many people are convinced that you can't stop spam, so you shouldn't try harder. The problem is low expectations. The problem is people cutting corners.
For email senders having problems getting caught in spam filters, some of this is due to people running bogus spam filters and that is the receiver's problem more than yours. Most of the rest is due to either you not running a standard-compliant mail server on a static IP address that can have a reputation built up for you being a good server, or because you really do send out spam, either due to "bad" customers or backscatter (bogus bounces, challenge/repsonse systems, autoresponders, etc.). Don't be cheap and think you can get away with not running spam filters on your outbound email and catching your "bad" customers. Don't be cheap and spew backscatter. Don't be cheap and say you can't afford to do port 25 blocking of dynamic IP addresses, or not allow customers to configure their reverse DNS.
The vast majority of knowledgable people in the area of spam do not munge their email addresses. The vast majority do not suffer either lots of spam in their inbox nor lots of false positives.
Because, of course, a spammer's objective is to be feared...
Sarcasm noted and you are right, spammers don't care at all if they are feared or not. That's not the point though.
The great thing about email is that people I don't really know, strangers, can easily contact me. Hiding your email address does kind of work, but it defeats one of the main advantages of email. Hiding your email address also only works as long as you never give it to anyone who gets their PC owned and turned into a zombie.
People who "need" to hide their email address are usually people who have not used any of the many good modern spam filters. I use spamassassin (sa-exim with graylisting, actually), but there are a lot of other good ones too. Use them. They work. Don't be cheap and say that they spam filters are "too expensive", because your time is worth a heck of a lot more than a few CPU cycles.
Indeed. I have noticed that almost everyone who is involved with stopping spam does not munge or hide their email addresses. Julian Haight is the only person that I can think off of-hand that does not publish his email address.
I've been publishing my email address since the late 80s, I'm not going to start hiding it now.
I suspect a lot of people here have never experienced the Internet without the web.
Let me tell you: it wasn't that bad!
Instead of forums and such, we had mailing lists and usenet. They both uses basically the same format for messages, so you could often use the same client to deal with both. They had some really nice advantages, such as almost all of the UI was done by client. You could easily change how stuff looked and worked anyway you wanted without changing the whole system and all the forums (newsgroups) worked the same. It isn't like web forums and blogs, where some have threading, and some don't, and each has a different navigation system, and most don't have nice ways of just reading new messages.
Before there were things like sourceforge and freshmeat, there were comp.sources.*. When the latest update of something you used was released, you just grabbed it off usenet as it rolled by. If you missed it, you could use ftp or even ftp-mail.
I'm not saying life was better before the web, just that it was quite livable. Almost all of the large multi-user systems I've used since the 1970s have had email, so I guess the web has only been around for less than half my "computer life".
It is no coincidence that spam pretty much killed usenet just went the internet was getting popular. I guess all the different blog/forum systems somewhat protect them from spam since you can't just write a simple program to cross-post your greencard spam to all the blogs/forums like you could with usenet. If spam does end up killing email like it did usenet, life is going to be rough for IM/blogs because spammers will be forced to put their full effort on them.
Whenever I receive a phishing email, I immediately capture what I can of it (including headers,) then head to the legitimate site and look for a place to report it.
So do I, and I have for years now. It is called spamcop. One of the things they do is send all spam/phishing reports to another company called "cyveillance" that scans the spam for phishing, trademark violations (viagra, rollex, etc.), etc.
a form of trespass, little different than intruding on their land or making unwanted use of their private property.
... but to suggest that emailing someone is equivalent to trespass??!? Just how out-of-touch and confused does the state have to get with technology before they're sat down in an electric chair in front of a monitor, with a sticky on its side saying "Learn"?
Yes, "making unwanted use of their property" is a form of trespassing, known as Trespass to chattels, which is a well defined legal concept that has been around for hundreds of years. "Chattel" is the archaic legal term for personal property, in contrast with land or real estate.
Having watched the talks given at the last several years of MIT Spam Conferences, I can safely say that the people involved with drafting Virginia's anti-spam laws and prosecuting this particular spammer have a very good understanding of technology in general, and email in particular. They probably have a better understanding than than the average slashdot user. As horrible as it may be for some geeks to imagine, yes, there are a lot of lawyers that are very smart and can learn very technical stuff.
Firstly it is the confluence of internet/SM protocols, not the spammer, that puts the email on your server - although in the vast majority of these cases, you can believe that the recipient doesn't own the server at all.
You seem to have a very fuzzy concept of the internet and protocols. When someone puts a packet out on the net, they are, indeed, knowingly creating a process that will result in the packet ending up on the receiving computer's network port. It may not be the same exact electrons, but that is irrelevant. And, I assure you that AOL owns their servers and they are the ones that received the spam. Yes, customers of AOL rent the mailboxes from them, but AOL still has legal rights to the servers. This is no different than a hotel or apartment owner that rents out rooms/apartments. They still have legal rights to their property.
Not everyone likes the idea of applying the age old concept of Trespass to Chattels to the internet, for example, the EFF sees problems with it. I agree with the EFF on most things, and have contributed money to them, but in the area of spam, they act too much like chicken-little. The Virginia anti-spam law was narrowly taylored and well thought out. It is a shame that it large parts of it have been overridden by the much worse federal CAN-SPAM act.
* Spurious criterions for getting listed. Like "unsolicited bounces" or "sent mail to secret spamtrap"
Neither sending bounces to forged email addresses (backscatter) nor sending email to spam traps is a "spurious criteria" for getting listed. If you are doing either of these, you are part of the spam problem and you deserver to have your email blocked.
Configure your mail server to reject *during* the SMTP session, or only send bounces after the fact if the email passes something like SPF. Using SPF's "best guess" default record of "v=spf1 a mx ptr" when there isn't an SPF record may by safe, but you are still risking sending backscatter to innocent people. Domainkeys and DKIM validate the From: header, which isn't where you should send bounces to, so it can only be used when the Envelope-From and the From: header match. Really, the easiest thing to do is just to always reject during the SMTP session and not accept-then-bounce.
Greylisting will always be useful, at least to a certain degree. When you get email from an unrecognized source, one that you can't judge as being a source of legitimate email or if it is a spam source, greylisting lets the reputation systems like DNSBLs and DCC/Razor/etc. catch up. Also, good ISPs and mail admins have tools to watch for likely spam runs. Again, greylisting gives those tools time to act and purge the spam.
Sure, greylisting also gets rid of spam from spambots that don't retry email, and that's a plus but, as you mention, making sure the email gets retried isn't that hard.
Greylisting is not a Final and Ultimate Solution to the Spam Problem, but it will always be a useful tool.
Slashdot Mirror
The "packets of 576 bytes can't be fragmented" is a commonly stated reason, but it is wrong. It is a myth/misunderstanding. It is, in practice, true has has been true since probably the late 1980s, but DNS was around long before that. Indeed, if you read some of the earlier RFCs, it is quite clear that packets of any size could be fragmented, down to something like 16 bytes of payload per fragment. No,the reason for the 512 byte payload size is much more basic than that. Back in the early 80s, memory was tight, you could have mainframes supporting dozens of users on a machine with maybe 1MB of memory, each of user could have more than one active network connection. IP supports packets sizes up to around 64k, but it would be unreasonable to expect every host to be able to accept such a large packet size. It would mean that they could get fragments from all those packets piecemeal and out of order, so reconstructing each packet would require holding lots of 64k buffers, each of those buffers would be 6% of all available memory. It would be very unreasonable to expect every host on the internet to be able to accept any size packet, even if those packets came in fragment that wouldn't saturate your connection. Now, protocols like TCP have the ability to negotiate the packet size, but for UDP, it gets messy and slow. So, it is a *requirement* that each host on the internet can accept a packet with 512 bytes of payload. That packet can be fragmented, but it has to be accepted.
There is a certain amount of irony with you saying this, followed by your .signature of:
This "joke" is clearly aimed at people who think they understand math/physics/science, so it won't be funny to most people. But, it also shows a complete lack of understanding about how equations should be interpreted. What the formula "money = work/knowledge" says to increase the amount of worked done, you need either more money or more knowledge. In other words, "the only substitute for knowledge is money", or "a fool and his money are soon parted". You are a case of "a little knowledge is a dangerous thing". By your own statments, you shouldn't vote.
The big problem with DNSSEC, if widely used, is that it prevents forgery of DNS responses. ISPs and internet cafes will not like this, since that means they can no longer forget DNS replies to missing domains or to force people through registration pages. I can see a *LOT* of push-back from having end-users using DNSSEC.
Trust is the same for DNSSEc, it's just that instead of using the root servers as a trust chain, you use a 3rd party that every domain owners had to pay for.
DNSCurve does not require you to pay any third parties, it is like DNSSEC where you publish your own information. Both technologies are (or in the case of DNSCurve, will be) free.
DNSCurve is much easier to implement than DNSSEC and and also advantages in term of cryptography speed and increase of traffic.
DNSSEC has many years of actual deployment, not as wide spread as it needs to be, but it has been out there and tested.
Can you point me to a single implementation of DNSCurve? Can you even point me to a specification of what exactly it is? I've looked, and the best that I can tell, there aren't any. More over, it doesn't appear that DJB's website has been updated since he proposed DNSCurve last year.
DNSCurve is interesting technology, but it has many problems, not the least of which is that it is mostly hype right now. It does not really replace DNSSEC in functionality, but rather, it is closer to TSIG. That is, instead of securing the actual DNS records, it secures the communication between name servers and resolvers. With DNSSEC, you can get your DNS records for a totally untrustworthy server, and yet be able to prove if they are valid or not, but there isn't any form of encryption so there isn't any privacy. DNSCurve encrypts the transactions, but you can often figure out what is there anyway by watching which name servers you are contacting and monitoring other things to figure out what you were looking up. I like DNSCurve, I hope it goes some where, but I also hope that DNSSEC takes off soon.
You may be a layman, but you appear to have far more clue about this stuff than most. Yes, once DNSSEC is deployed, anyone with a domain name can publish CERT records and have about the same security as a paid-for CERT. Granted the cert authorities right now require you to give your name and address and such, which publishing CERT records in the DNS won't require so they aren't exactly the same, but close enough considering how little checking the cert authorities do on such information
Most of the time, the email address in the "From:" header gets copied to the envelop "from". And, most importantly in this case, the envelop "from" is where bounces get sent to, so the bounces he receive could have been stopped if he had published an SPF record *and* everyone checked it.
Both SPF and DKIM have problems with this kind of thing. With SPF, you need to include these third parties in your SPF record (possibly using the SPF "include:" mechanism). As you mentioned, with DKIM, you have to send them a key.
Yes, tracking down all your mail servers is a huge problem for large organizations. SPF can help identify these sources by means of a "tracking exists: mechanism". This can then be used to help add IP addresses to SPF records or distribute keys for DKIM.
The IETF standard process runs by rough consensus. The IETF working group (MARID) that tried to standardize some of this stuff had everyone from Richard Stallman to Bill Gates trying to influence the outcome, along with a dozen proposals from "competing" technologies that would preferred no standard to theirs being excluded (this group included many folks who supported Domainkeys). SPF grabbed an early lead in deployment and mindshare by working outside the IETF, causing no great love with the IETF insiders.
It shouldn't be a big surprise that the MARID working group fail to reach consensus and was shut down without publishing any standards.
The folks involved with SPF (including me), and Microsoft both put forward their systems as "independent RFC submissions". These RFC submissions usually do not get put on the standard track and it could easily be argued that even being labeled "experimental" rather than "information" is a positive sign that the IETF sees at least some value in them.
By the time the DKIM working group was started, most of the hype had died away and a much smaller group of people were able to find consensus. DKIM was also backed by IETF insiders and most of the people from the SPF project that participated in the DKIM working group saw it as a complementary technology, rather than a competing one.
For email sent directly from one party to another, SPF, Sender ID, DomainKeys and DKIM all work very well.
In general, SPF (and Microsoft's Sender ID) has many more problems with email forwarders. There are ways around the problem using things like SRS or what gmail does and munges through the Received: headers to see which ones can be trusted and are forwarders. DKIM (and DomainKeys), on the other hand, rarely fail on forwarders, although it does happen when the forwarder adds trailers, do mime-rewriting, virus/worm scanning, etc.
In general, SPF has few problems with things like mailing lists. DKIM (and DomainKeys and Sender ID), on the otherhand frequently have problems with mailing lists.
Those of use that see SPF and DKIM/Domainkeys as complementary see domainkeys as very useful in identifying trustworthy forwarders which can then be used to make SPF more reliable, and SPF as a way of identifying trustworthy mailing lists, which can then be used to mail DKIM/DomainKeys more reliable.
In the 1970's, I used a the CDC PLATO system, which looked more like the modern internet than the internet in the 1970's looked like the modern internet. It had instant messaging (term-talk), email (pnotes), forums (notes, later to evolve into lotus notes), and chatrooms (0chat). No one talked about one replacing the other because they were all good for different things.
In the early 1980's, I used IBM's CMS system. It had instant messaging (#cp msg) and email, but sadly, no forums nor chat rooms. People talked about needing the later two.
In the mid 1980s to the early 1990's, I used unix. It had IM, email, forums and chat rooms.
Since the early 1990s', I've used unix on the internet. It has IM, email, forums and chat rooms.
Now, in the 2000's, people claim that IM will kill email? Huh? I don't see it. Did these people never have IM before?
Before "correcting" Karl Auerbach, you might want to to see just how many google RFC's he has been involved with, not to mention being kicked off the ICANN board for trying to stand up for the individual.
No, that isn't who DNS works. If a machine decides to send a query to the root name servers, they will send the complete domain name. The root name servers will then reply "I don't know the answer, try that name server over there". In theory, most machines should have the TLDs cached and not send the query to the root name server first, but there are a huge number of broken resolvers out there. The Measurement Factory has some published studies about just how much bogus crud gets sent to the root name servers, and there are a bunch of other studies that would require a little more work.
Seriously, yes, data mining the root name servers can be done. One of The Measurement Factory studies did just that. It could turn up a lot of interesting stuff.
Actually, wikipedia has a very good summary of when IPv4 address space exhaustion will likely happen. In particular, while the IPv4 allocation graphs made by Geoff Huston aren't as pretty, they are likely far more accurate than xkcd's. The only problem with Geoff's predictions is the exhaution date keeps getting moved forward so his dates are probably best-case predictions.
Basically, yes, the IPv4 space is running out. It is still 3-5 years out for IANA exhaustion and further for the RIRs and ISPs, but it is something that people need to start planning for. The predictions about IPv4 addresses running out back in the 90s was before the development of things like CIDR allocations, NAT, RFC1918 private network numbers, HTTP1.1's virtual hosts, DHCP, and the dot-com crash. There haven't been any new "gee, we can make the IPv4 space go a lot further if..." type ideas for years and it doesn't appear likely that any more large savings will happen before it is too late to deploy them.
It is still $200/employee/year, which is far far less than many other costs. The "spam problem" is no worse than the "heating/airconditioning problem", as far as costs go. Would any company seriously consider not heating/cooling their work place to the point of hurting productivity just because it costs a couple hundred per year per employee?
I guess I should have been clearer in my original post. The solutions to the spam problem have largely been known for many years now. What needs to happen is for wider implementation of those known solutions. Image-only spam has been around for years and it was predicted that once things like bayesian filtering became common, that this is the direction that spammers would move to. This is simply causing the anti-spam field to shift from "content-analysis" to the "sender's reputation" model. Only now, the industry has much better feed-back loops to let people know that a sender is sending unsolicited (or at least unwanted) email, and that they are sending it in bulk.
I never said that protecting your inbox from spam would be cheap, just the opposite, I said it would cost but that you shouldn't be penny wise and pound foolish. I also said that senders need to become much more careful about what they send, or they won't get their email delivered. However, as you say:
Agreed, this is another part of the spam solution. Law enforcement simply hasn't caught up with this new type of criminal activity.
Correct.
No, it is the network administrators not taking care of their networks, in particular the network administrators that have home desktop machines hooked up to their networks. Just because they may call themselves an "ISP" doesn't mean they don't run a network.
These network administrators either need to make sure that their customers don't get infected and that they don't have spammers as customers, or they need to make sure that their infected machines/spamming customers don't hurt the rest of the internet. Chances are that these network administrators don't want to do either of these because it is cheaper to let everyone else on the internet block the abuse and then blame everyone else on the internet when their customers complain about being blocked.
That comes to $200 per employee per year. How much is your employee's time worth? How much would be lost on paying employees to filter their spam themselves or having lots of false positives?
This is exactly what I was talking about. You are being penny wise and pound foolish if you don't think that $200/person/year is cheap. As you admit, the spam problem is largely solved for you.
OK, but here is the important point. If everyone in the world took as much effort to solve the spam problem as you, and many others, already do, then spam would largely disappear and spam filtering wouldn't be so expensive.
Let's see. The "theyworkforyou.com" domain doesn't have a valid rDNS pointer, something that the RFCs require for all hosts on the internet. They don't publish SPF records, nor use domainkeys.
If you are going to use things like Bayesian filters, you will need to train them. For spamassassin, you need to run an "sa-learn --ham" on a few of those emails. Many systems will whitelist senders that are in your address book, with spamassassin, you might need to add a "whitelist_from *@theyworkforyou.com" to your ~/.spamassassin/user_prefs. I also suspect that if theyworkforyou.com is getting that high of a spam-probability on the bayesian analysis, that they are sending email that looks spammy for other reasons.
Make sure your spamassassin is set up correctly. Enable the "network checks", some people turn them off because they are "too expensive". Also enable the auto-whitelist, even though that costs a little disk space and some people consider it to be "too expensive". Oh, and make sure you have a per-user dictionary for your bayesian analysis, not one for the entire domain. Again, it costs a little more, but it helps a great deal.
And, yes, picking a good domain name is important. If you are "Big Bank", don't go creating a brazillian different domain names, use subdomains.
I know people like to rant about the "spam problem" a lot, but for all practical purposes, the problem has been largely solved for several years now.
If you run reasonable spam filters, including many open source ones, you will not end up with much spam in your inbox. Yeah, there will be lots of spam still being sent, but the real, significant, cost of spam is really mostly people's time, not machines. Any ISP, company or person who gets "too much spam" is simply being penny wise and pound foolish. The same goes for systems that get too may "false positives", that is, legitimate emails being rejected. Almost all of that is due to trying to run "cheap" spam filters, or buying snake-oil systems. Upgrade your mail servers or switch to someone who runs reasonable spam filters.
The "spam problem" of today is really the "you can't do anything about spam" problem. Too many people are convinced that you can't stop spam, so you shouldn't try harder. The problem is low expectations. The problem is people cutting corners.
For email senders having problems getting caught in spam filters, some of this is due to people running bogus spam filters and that is the receiver's problem more than yours. Most of the rest is due to either you not running a standard-compliant mail server on a static IP address that can have a reputation built up for you being a good server, or because you really do send out spam, either due to "bad" customers or backscatter (bogus bounces, challenge/repsonse systems, autoresponders, etc.). Don't be cheap and think you can get away with not running spam filters on your outbound email and catching your "bad" customers. Don't be cheap and spew backscatter. Don't be cheap and say you can't afford to do port 25 blocking of dynamic IP addresses, or not allow customers to configure their reverse DNS.
The vast majority of knowledgable people in the area of spam do not munge their email addresses. The vast majority do not suffer either lots of spam in their inbox nor lots of false positives.
Sarcasm noted and you are right, spammers don't care at all if they are feared or not. That's not the point though.
The great thing about email is that people I don't really know, strangers, can easily contact me. Hiding your email address does kind of work, but it defeats one of the main advantages of email. Hiding your email address also only works as long as you never give it to anyone who gets their PC owned and turned into a zombie.
People who "need" to hide their email address are usually people who have not used any of the many good modern spam filters. I use spamassassin (sa-exim with graylisting, actually), but there are a lot of other good ones too. Use them. They work. Don't be cheap and say that they spam filters are "too expensive", because your time is worth a heck of a lot more than a few CPU cycles.
Indeed. I have noticed that almost everyone who is involved with stopping spam does not munge or hide their email addresses. Julian Haight is the only person that I can think off of-hand that does not publish his email address.
I've been publishing my email address since the late 80s, I'm not going to start hiding it now.
Ursula Le Guin? yep. Rowling? nope.
I suspect a lot of people here have never experienced the Internet without the web.
Let me tell you: it wasn't that bad!
Instead of forums and such, we had mailing lists and usenet. They both uses basically the same format for messages, so you could often use the same client to deal with both. They had some really nice advantages, such as almost all of the UI was done by client. You could easily change how stuff looked and worked anyway you wanted without changing the whole system and all the forums (newsgroups) worked the same. It isn't like web forums and blogs, where some have threading, and some don't, and each has a different navigation system, and most don't have nice ways of just reading new messages.
Before there were things like sourceforge and freshmeat, there were comp.sources.*. When the latest update of something you used was released, you just grabbed it off usenet as it rolled by. If you missed it, you could use ftp or even ftp-mail.
I'm not saying life was better before the web, just that it was quite livable. Almost all of the large multi-user systems I've used since the 1970s have had email, so I guess the web has only been around for less than half my "computer life".
It is no coincidence that spam pretty much killed usenet just went the internet was getting popular. I guess all the different blog/forum systems somewhat protect them from spam since you can't just write a simple program to cross-post your greencard spam to all the blogs/forums like you could with usenet. If spam does end up killing email like it did usenet, life is going to be rough for IM/blogs because spammers will be forced to put their full effort on them.
So do I, and I have for years now. It is called spamcop. One of the things they do is send all spam/phishing reports to another company called "cyveillance" that scans the spam for phishing, trademark violations (viagra, rollex, etc.), etc.
It is quick, and easy to do.
Yes, "making unwanted use of their property" is a form of trespassing, known as Trespass to chattels, which is a well defined legal concept that has been around for hundreds of years. "Chattel" is the archaic legal term for personal property, in contrast with land or real estate.
Having watched the talks given at the last several years of MIT Spam Conferences, I can safely say that the people involved with drafting Virginia's anti-spam laws and prosecuting this particular spammer have a very good understanding of technology in general, and email in particular. They probably have a better understanding than than the average slashdot user. As horrible as it may be for some geeks to imagine, yes, there are a lot of lawyers that are very smart and can learn very technical stuff.
You seem to have a very fuzzy concept of the internet and protocols. When someone puts a packet out on the net, they are, indeed, knowingly creating a process that will result in the packet ending up on the receiving computer's network port. It may not be the same exact electrons, but that is irrelevant. And, I assure you that AOL owns their servers and they are the ones that received the spam. Yes, customers of AOL rent the mailboxes from them, but AOL still has legal rights to the servers. This is no different than a hotel or apartment owner that rents out rooms/apartments. They still have legal rights to their property.
Not everyone likes the idea of applying the age old concept of Trespass to Chattels to the internet, for example, the EFF sees problems with it. I agree with the EFF on most things, and have contributed money to them, but in the area of spam, they act too much like chicken-little. The Virginia anti-spam law was narrowly taylored and well thought out. It is a shame that it large parts of it have been overridden by the much worse federal CAN-SPAM act.
Neither sending bounces to forged email addresses (backscatter) nor sending email to spam traps is a "spurious criteria" for getting listed. If you are doing either of these, you are part of the spam problem and you deserver to have your email blocked.
Configure your mail server to reject *during* the SMTP session, or only send bounces after the fact if the email passes something like SPF. Using SPF's "best guess" default record of "v=spf1 a mx ptr" when there isn't an SPF record may by safe, but you are still risking sending backscatter to innocent people. Domainkeys and DKIM validate the From: header, which isn't where you should send bounces to, so it can only be used when the Envelope-From and the From: header match. Really, the easiest thing to do is just to always reject during the SMTP session and not accept-then-bounce.
Greylisting will always be useful, at least to a certain degree. When you get email from an unrecognized source, one that you can't judge as being a source of legitimate email or if it is a spam source, greylisting lets the reputation systems like DNSBLs and DCC/Razor/etc. catch up. Also, good ISPs and mail admins have tools to watch for likely spam runs. Again, greylisting gives those tools time to act and purge the spam.
Sure, greylisting also gets rid of spam from spambots that don't retry email, and that's a plus but, as you mention, making sure the email gets retried isn't that hard.
Greylisting is not a Final and Ultimate Solution to the Spam Problem, but it will always be a useful tool.