It only holds something like 72 hours of DV. HDTV streams are somewhere in the vicinity of 10-25 Mbps (DV is 25 Mbps or roughly 15 Gb/hr).
That's actually not a lot of space once you get into multimedia.
But backup/recovery of a terabyte of data is not exactly trivial. Re-scanning and re-syncing a large disk array can take over a day. Moving that data across a 100mbps ethernet would require anywhere from 38 to 60 hours.
The cost isn't too bad (close to $1/Gb), but I'd prefer to see it reconfigured as a RAID5 unit.
FYI, AOL already does reverse-MX whitelisting. If your domain sends a large volume of e-mail into their system, you have to list the IP addresses of your outbound mail servers with them. (Otherwise the mail gets dropped to/dev/null.)
AOL's probably interested in SPF mainly because it means they won't have to do all of the manual processing that they do now. At least, they won't have to manually keep track of domains and outbound IPs. (Instead, they'll just query the SPF record for the domain.)
Backing of the large ISPs will definitely do a lot to either make/break any of the source verification systems such as DomainKeys or SPF.
The closest I get is because I use My Yahoo! as a home page and the (3) nearest big cities all have newsfeeds. But I agree, I'm well up on national/global news, but pretty sparse at what's going on 2 blocks from here.
I never watched local news, but back when I wasn't working from home I'd stop at a local diner on the way to work and always read the morning paper. Haven't read a newspaper in probably 2 years now, maybe 3.
While/. has it's downsides, it does serve as a useful place to:
- get multiple viewpoints or background, reasonably quickly
- check your worldview against others
- find out if your writing style is flamebait / trollish / or reasonable
The moderation levels are nice for clearing out the deadwood. I shudder when I go to a site like Groklaw / FPS forum / etc., see 200+ responses to a topic and know that half of them are going to be "me too!" posts.
Now if someone would just invent a +2 filter for dealing with the public...
If you mean e-mail that is purporting to be from yahoo! when it actually came from joe blow's spam factory trojan'd PC... then you're looking at SPF or one of the reverse-MX systems (or even Yahoo!'s).
Mail headers are created as e-mail is passed along from MTA to MTA. They are what they are...
He has a script running on his mail that replies to every email he gets with a confirmation code. When the end user replies with that confirmation code (all it takes is hitting ctrl-r and ctrl-enter) that email address is adding to his "verified email address" list, and the original email goes through.
Yes, you've just described a Challange/Response system. And right now, since domain / origin e-mail addresses are so easily forged - it's extremely annoying to the people who get those (forged) challenges. If it were widespread, it could currently be used as a DDoS attack against a victim of your choice. Just send out a spam with the victim's e-mail address as the FROM: address and watch their server go down in flames from all of the challenge messages (in addition to all of the bounces).
Your friend will get lambasted for using it sooner or later (probably sooner). And if mail clients could handle the C/R transaction transparently... well, that means it can be automated and spammers will just auto-add themselves to your friend's whitelist. Whoops, back to square one.
(Most C/R systems use a funky graphic image which has to be interpreted by human eyes... blind people need not apply... and which is not currently interpretable by a computer.)
You must be the one who's filter the spammers are trying to brute-force by putting random words in the subject line!
Whitelisting works... sorta... except that spammers can forge any address onto their e-mails that they want. So as your whitelist gets larger and less specific, the odds of them slipping one past you gets higher. (Heck, a lot of them just put *your* address in the FROM: line.)
However, it's an effective "first step" filter on a local level.
Microsoft's Penny Black does zip, zero, zilch about the problem of domain forgery. MS's solution is aimed at a way to rate-limit the amount of outbound SMTP e-mail that a client can send.
Yahoo!'s solution is aimed at the stopping of domain forgery and has nothing to do with rate-limiting outbound e-mail.
They're two seperate and distinct problems, hardly related at all.
Under SPF (or other reverse-MX proposals), e-mail that is purporting to be from domain X has to come from a limited set of IP addresses (typically the official, mail admin monitored, virus-checking, maybe rate-limiting, SMTP servers for a domain).
So in order for a spammer to spam someone using your actual e-mail address they have to:
1) hack into your domain's outbound mail server and send e-mail from there (nothing new in this risk)
2) hijack/trojan your machine or a machine in your organization and then route e-mails through the official SMTP server (same as what happens now, except that the mail admin is more likely to notice that customer 32432's account is sending gobs of e-mail)
3) poison the DNS SPF information (tough attack to pull off, can be combatted and might lead to new security in the DNS system)
4) spammer goes in search of a domain that doesn't have reverse-MX info and forges that domain onto their e-mails
5) spammer starts to use throw-away domains at $X each
#1 and #2 are the keys... SPF is designed to make it much more difficult to do domain forging or joe-jobbing.
Spam is like a telemarketer who's allowed to make all of their sales calls, collect. And they're allowed to munge/forge the caller ID information, or pretend to be a close friend of the family to get past whitelists and/or call screening.
It's all of the above: ECONOMIC, SOCIAL and TECHNICAL. No one approach is going to solve the issue.
I know it was mentioned a few days/weeks ago on Slashdot... Yahoo! Develops Anti-Spam Architecture . But no, I haven't seen any specifics... and some of the article wording indicates that there wouldn't be deliverable code until 6 months from now. (Making me wonder if specifics are even nailed down.)
I'm pretty sure one of the bayesian filters already does it that way, but I don't know which product. SpamBayes that I use is still single-word driven, but it does parse header/subject/from/to information and adds that to the database.
Which is essentially the same thing as a client-trained bayesian filter. Because your bayesian database isn't shared with anyone else, it's highly customized to what you consider to be ham/spam. Odds are that the spammer is going to guess wrong and get tagged as spam.
Just like whitelists/blacklists and a few other schemes, it's better to have a locally-controlled list rather then relying on a huge 3rd party list. When the huge 3rd party list is used by 1,000,000 clients, there's a large incentive to spend time hacking past it. OTOH, it's not worth the effort required to hack past a filter that only protects 100 clients.
Dunno, but I've tagged darn near every Newslinx e-mail newsletter that I've gotten in the past few months as "Not spam", yet Yahoo! Mail still puts it in the bulk folder.
You would expect that after 100 whacks with a rolled up newspaper (bad filter! bad filter!), it would get the idea that Newslinx is not spam...
I believe it's because Newslinx uses a new e-mail address every time (unlike the BBC daily news which never gets tagged as spam).
I probably should've been clearer... (I do realize that Yahoo!'s is open-source).
The original poster was saying that the Yahoo! system would need to be closed source in order to be secure. You and I both agree that a closed-source system does nothing to make Yahoo!'s system more secure and I was trying to point that out to the OP.
It's been highly educational (and sickening) to watch the machinations within the IETF and ASRG mailing lists. (I came to the conclusion last May that the ASRG probably would never actually fix anything...) Nothing surprising if you've ever worked around middle-management in a large corporation though.
Ummm... and who do you propose is going to do the licensing and regulations? What enforcement powers will they have over relays in another jurisdiction?
What's to stop the spammers from bribing officials to get their spam-relays "licensed"?
The main issue is that to protect its own workings, it would need to be closed source. There will be a slight problem with some system admins installing it in that event.
Why closed source?
Closed-source cryptographic systems (which is essentially what this is) are often very insecure if they are not peer-reviewed. In fact, Bruce Schneier argues often in his books that a properly designed cryptographic system is just as secure if the source/spec is open/published. Most problems are actually due to implementation weaknesses which argues for the "many eyes, bugs shallow" of open source code.
Go subscribe to Crypto-Gram or read up the back issues if you want to get a good background on what makes for secure systems.
This system, reverse-MX systems, and other systems will not be temporary.
The problem today with SMTP spam is that it's like being able to collect-call your target and they have to receive the call. Worse, the spammer is able to forge the caller ID (FROM:) information so that you can't simply use the caller ID info to decide whether to accept/reject the call/message.
This is the techical equivalent of a law requiring that caller ID information be accurate. It doesn't stop the telemarketing calls, but does let the receiver make a more informed decision about whether or not to accept the collect call.
Reverse MX and Yahoo!'s proposal, however, don't require widespread adoption at the start. In fact, the tipping point is probably only a few percentage points of the domain namespace.
After all, for just a few minutes worth of work (more if you don't already provide SMTP AUTH, or require users to VPN in to send e-mail already), you protect your domain against joe jobs and forged e-mail bounces. So there's a low cost-of-entry. (Yahoo!'s proposal requires more work then the simpler, less CPU-intensive SPF proposal.)
What happens next is that domain admins that publish keys/SPF information find that they're no longer getting joe-jobbed and they're able to block a higher percentage of spam then they used to. Word gets out and more folks sign on (second wave adopters).
Sometime after that, the big ISPs require your mail servers to publish SPF/keys if you want your e-mail to be delivered to their users. (FYI, this is very similar to AOL's whitelisting program, which is essential a privately-administered reverse-MX system where you tell AOL what IPs your e-mail is allowed to originate from.)
As a WAG about rate of pickup, early adopters have started, second wave folks will probably sign on in the spring/summer, and I wouldn't be surprised to see ISP-blocking by the end of the year.
You mean like "reverse MX" records... google for RMX, SMTP+SPF, DRIP, DMX. (SPF seems to have momentum at the moment)
However, reverse-MX solutions will not kill off spam (a common mis-conception). The goal of reverse-MX proposals is to stop domain forgery where spammers are able to, with complete impunity, to tack on any old domain name to their spams. Which means that the unfortunate organization who is forged gets to deal with the thousands of e-mail bounces and the irate phone calls / e-mails from people who think that the organization was the source of the spam. As a mail admin, I'm able to control which servers handle inbound e-mail for my domain through specifying MX records. Reverse MX allows me to have the same amount of control over outbound e-mail from my domain.
What will happen instead, once reverse-MX systems (or Yahoo!'s system or other sender-authentication systems) come into play. Spammers will have to change tactics and resort to either forging one of the remaining domains that don't have reverse-MX information published, or they will register throw-away domains by the hundreds. It will drive up their costs a tiny bit (much like the impact of bayesian and other filters requiring them to use randomization techniques).
But the real nice side-effect of reverse-MX, etc., is that you'll be able to more reliable whitelist based on domain name. And your bayesian filters will be able to assign high ham values to domain names.
It also puts a crimp in e-mail worms that attempt to use a built-in SMTP engine to avoid detection. Unless the worm forges a domain with no reverse-MX info published, the worm won't spread (most MTAs will drop the connection). Instead, the worm will have to route through the user domain's SMTP server, where the mail admin is more likely to catch the traffic (virus scanner on the SMTP server, or rate limiters).
It's bad if you have a different "From" address from what your SMTP server is, in which case I don't see how it could work for you.
1) Don't publish a key for your domain (downside is that you can still be joe-jobbed and nobody can verify that e-mail coming from your domain is authentic, or at least that it passed through an authorized server)
2) Use SMTP AUTH / VPN to connect to your domain's server, just as if you were in the office. (Most corporations, where you are acting as an agent of the corporation, would prefer this method.) The only time this is a problem is if you're behind a firewall of some sort, in which case a $9.95 dial-up account and unplugging a fax machine for a few minutes gets you past it. Or you can make use of the 3G hi-speed internet wireless services that are coming in a year or two.
There are options, and if your service provider doesn't provide alternatives, then you need to find another provider or bring pressure to bear on that provider. The ability to randomly forge any domain that you want onto your e-mail has been abused to death, hence the momentum behind things like Yahoo!'s proposal and the various reverse-MX / sender-authentication proposals. Most mail admins are tired enough of being joe-jobbed and dealing with bounces due to domain forgery that they're willing to make these changes.
Slashdot Mirror
It only holds something like 72 hours of DV. HDTV streams are somewhere in the vicinity of 10-25 Mbps (DV is 25 Mbps or roughly 15 Gb/hr).
That's actually not a lot of space once you get into multimedia.
But backup/recovery of a terabyte of data is not exactly trivial. Re-scanning and re-syncing a large disk array can take over a day. Moving that data across a 100mbps ethernet would require anywhere from 38 to 60 hours.
The cost isn't too bad (close to $1/Gb), but I'd prefer to see it reconfigured as a RAID5 unit.
FYI, AOL already does reverse-MX whitelisting. If your domain sends a large volume of e-mail into their system, you have to list the IP addresses of your outbound mail servers with them. (Otherwise the mail gets dropped to /dev/null.)
AOL's probably interested in SPF mainly because it means they won't have to do all of the manual processing that they do now. At least, they won't have to manually keep track of domains and outbound IPs. (Instead, they'll just query the SPF record for the domain.)
Backing of the large ISPs will definitely do a lot to either make/break any of the source verification systems such as DomainKeys or SPF.
The closest I get is because I use My Yahoo! as a home page and the (3) nearest big cities all have newsfeeds. But I agree, I'm well up on national/global news, but pretty sparse at what's going on 2 blocks from here.
I never watched local news, but back when I wasn't working from home I'd stop at a local diner on the way to work and always read the morning paper. Haven't read a newspaper in probably 2 years now, maybe 3.
While /. has it's downsides, it does serve as a useful place to:
- get multiple viewpoints or background, reasonably quickly
- check your worldview against others
- find out if your writing style is flamebait / trollish / or reasonable
The moderation levels are nice for clearing out the deadwood. I shudder when I go to a site like Groklaw / FPS forum / etc., see 200+ responses to a topic and know that half of them are going to be "me too!" posts.
Now if someone would just invent a +2 filter for dealing with the public...
Politics, politics, politics... a.k.a. making sure *my* name gets on the proposal and not my arch-enemy.
Go read the ASRG mailing list from last May-July and you'll get a glimpse of some of the people issues involved. (Everyone has a sacred cow.)
The solution may be largely technical, but it's a real social problem to get something implemented.
1) dictionary attacks
2) e-mail addresses in public records
3) common e-mail addresses that you have to monitor (john@domain, webmaster@, abuse@, postmaster@, root@)
4) friends who have posted your address online (good intentions...)
5) corporate espionage where someone makes a copy of a maillist for a spammer for $$$
6) spammer got lucky
Define "false headers"?
If you mean e-mail that is purporting to be from yahoo! when it actually came from joe blow's spam factory trojan'd PC... then you're looking at SPF or one of the reverse-MX systems (or even Yahoo!'s).
Mail headers are created as e-mail is passed along from MTA to MTA. They are what they are...
He has a script running on his mail that replies to every email he gets with a confirmation code. When the end user replies with that confirmation code (all it takes is hitting ctrl-r and ctrl-enter) that email address is adding to his "verified email address" list, and the original email goes through.
Yes, you've just described a Challange/Response system. And right now, since domain / origin e-mail addresses are so easily forged - it's extremely annoying to the people who get those (forged) challenges. If it were widespread, it could currently be used as a DDoS attack against a victim of your choice. Just send out a spam with the victim's e-mail address as the FROM: address and watch their server go down in flames from all of the challenge messages (in addition to all of the bounces).
Your friend will get lambasted for using it sooner or later (probably sooner). And if mail clients could handle the C/R transaction transparently... well, that means it can be automated and spammers will just auto-add themselves to your friend's whitelist. Whoops, back to square one.
(Most C/R systems use a funky graphic image which has to be interpreted by human eyes... blind people need not apply... and which is not currently interpretable by a computer.)
You must be the one who's filter the spammers are trying to brute-force by putting random words in the subject line!
Whitelisting works... sorta... except that spammers can forge any address onto their e-mails that they want. So as your whitelist gets larger and less specific, the odds of them slipping one past you gets higher. (Heck, a lot of them just put *your* address in the FROM: line.)
However, it's an effective "first step" filter on a local level.
No... I posted AC because it's not worth arguing with other AC's.
/. records the IP address of the submitter, so this is only pseudo-anonymous.
And
Microsoft's Penny Black does zip, zero, zilch about the problem of domain forgery. MS's solution is aimed at a way to rate-limit the amount of outbound SMTP e-mail that a client can send.
Yahoo!'s solution is aimed at the stopping of domain forgery and has nothing to do with rate-limiting outbound e-mail.
They're two seperate and distinct problems, hardly related at all.
Yes, you've missed the mark a bit...
Under SPF (or other reverse-MX proposals), e-mail that is purporting to be from domain X has to come from a limited set of IP addresses (typically the official, mail admin monitored, virus-checking, maybe rate-limiting, SMTP servers for a domain).
So in order for a spammer to spam someone using your actual e-mail address they have to:
1) hack into your domain's outbound mail server and send e-mail from there (nothing new in this risk)
2) hijack/trojan your machine or a machine in your organization and then route e-mails through the official SMTP server (same as what happens now, except that the mail admin is more likely to notice that customer 32432's account is sending gobs of e-mail)
3) poison the DNS SPF information (tough attack to pull off, can be combatted and might lead to new security in the DNS system)
4) spammer goes in search of a domain that doesn't have reverse-MX info and forges that domain onto their e-mails
5) spammer starts to use throw-away domains at $X each
#1 and #2 are the keys... SPF is designed to make it much more difficult to do domain forging or joe-jobbing.
Spam is like a telemarketer who's allowed to make all of their sales calls, collect. And they're allowed to munge/forge the caller ID information, or pretend to be a close friend of the family to get past whitelists and/or call screening.
It's all of the above: ECONOMIC, SOCIAL and TECHNICAL. No one approach is going to solve the issue.
I know it was mentioned a few days/weeks ago on Slashdot... Yahoo! Develops Anti-Spam Architecture . But no, I haven't seen any specifics... and some of the article wording indicates that there wouldn't be deliverable code until 6 months from now. (Making me wonder if specifics are even nailed down.)
B. CRLs don't scale. Period. There's a reason why PKIs hardly ever get past 100K users.
They scale enough though- the number of email domains is presumably much less than 100K.
Back in 2000-2001 there were already 15 million domains (estimated)
Another post from 2002 says that there are at least 36 million.
That's probably the next step in the arms race.
I'm pretty sure one of the bayesian filters already does it that way, but I don't know which product. SpamBayes that I use is still single-word driven, but it does parse header/subject/from/to information and adds that to the database.
Which is essentially the same thing as a client-trained bayesian filter. Because your bayesian database isn't shared with anyone else, it's highly customized to what you consider to be ham/spam. Odds are that the spammer is going to guess wrong and get tagged as spam.
Just like whitelists/blacklists and a few other schemes, it's better to have a locally-controlled list rather then relying on a huge 3rd party list. When the huge 3rd party list is used by 1,000,000 clients, there's a large incentive to spend time hacking past it. OTOH, it's not worth the effort required to hack past a filter that only protects 100 clients.
Dunno, but I've tagged darn near every Newslinx e-mail newsletter that I've gotten in the past few months as "Not spam", yet Yahoo! Mail still puts it in the bulk folder.
You would expect that after 100 whacks with a rolled up newspaper (bad filter! bad filter!), it would get the idea that Newslinx is not spam...
I believe it's because Newslinx uses a new e-mail address every time (unlike the BBC daily news which never gets tagged as spam).
I probably should've been clearer... (I do realize that Yahoo!'s is open-source).
The original poster was saying that the Yahoo! system would need to be closed source in order to be secure. You and I both agree that a closed-source system does nothing to make Yahoo!'s system more secure and I was trying to point that out to the OP.
It's been highly educational (and sickening) to watch the machinations within the IETF and ASRG mailing lists. (I came to the conclusion last May that the ASRG probably would never actually fix anything...) Nothing surprising if you've ever worked around middle-management in a large corporation though.
SMTP relays need to be licensed and regulated.
Ummm... and who do you propose is going to do the licensing and regulations? What enforcement powers will they have over relays in another jurisdiction?
What's to stop the spammers from bribing officials to get their spam-relays "licensed"?
The main issue is that to protect its own workings, it would need to be closed source. There will be a slight problem with some system admins installing it in that event.
Why closed source?
Closed-source cryptographic systems (which is essentially what this is) are often very insecure if they are not peer-reviewed. In fact, Bruce Schneier argues often in his books that a properly designed cryptographic system is just as secure if the source/spec is open/published. Most problems are actually due to implementation weaknesses which argues for the "many eyes, bugs shallow" of open source code.
Go subscribe to Crypto-Gram or read up the back issues if you want to get a good background on what makes for secure systems.
This system, reverse-MX systems, and other systems will not be temporary.
The problem today with SMTP spam is that it's like being able to collect-call your target and they have to receive the call. Worse, the spammer is able to forge the caller ID (FROM:) information so that you can't simply use the caller ID info to decide whether to accept/reject the call/message.
This is the techical equivalent of a law requiring that caller ID information be accurate. It doesn't stop the telemarketing calls, but does let the receiver make a more informed decision about whether or not to accept the collect call.
Pain is a powerful motivator...
Reverse MX and Yahoo!'s proposal, however, don't require widespread adoption at the start. In fact, the tipping point is probably only a few percentage points of the domain namespace.
After all, for just a few minutes worth of work (more if you don't already provide SMTP AUTH, or require users to VPN in to send e-mail already), you protect your domain against joe jobs and forged e-mail bounces. So there's a low cost-of-entry. (Yahoo!'s proposal requires more work then the simpler, less CPU-intensive SPF proposal.)
What happens next is that domain admins that publish keys/SPF information find that they're no longer getting joe-jobbed and they're able to block a higher percentage of spam then they used to. Word gets out and more folks sign on (second wave adopters).
Sometime after that, the big ISPs require your mail servers to publish SPF/keys if you want your e-mail to be delivered to their users. (FYI, this is very similar to AOL's whitelisting program, which is essential a privately-administered reverse-MX system where you tell AOL what IPs your e-mail is allowed to originate from.)
As a WAG about rate of pickup, early adopters have started, second wave folks will probably sign on in the spring/summer, and I wouldn't be surprised to see ISP-blocking by the end of the year.
You mean like "reverse MX" records... google for RMX, SMTP+SPF, DRIP, DMX. (SPF seems to have momentum at the moment)
However, reverse-MX solutions will not kill off spam (a common mis-conception). The goal of reverse-MX proposals is to stop domain forgery where spammers are able to, with complete impunity, to tack on any old domain name to their spams. Which means that the unfortunate organization who is forged gets to deal with the thousands of e-mail bounces and the irate phone calls / e-mails from people who think that the organization was the source of the spam. As a mail admin, I'm able to control which servers handle inbound e-mail for my domain through specifying MX records. Reverse MX allows me to have the same amount of control over outbound e-mail from my domain.
What will happen instead, once reverse-MX systems (or Yahoo!'s system or other sender-authentication systems) come into play. Spammers will have to change tactics and resort to either forging one of the remaining domains that don't have reverse-MX information published, or they will register throw-away domains by the hundreds. It will drive up their costs a tiny bit (much like the impact of bayesian and other filters requiring them to use randomization techniques).
But the real nice side-effect of reverse-MX, etc., is that you'll be able to more reliable whitelist based on domain name. And your bayesian filters will be able to assign high ham values to domain names.
It also puts a crimp in e-mail worms that attempt to use a built-in SMTP engine to avoid detection. Unless the worm forges a domain with no reverse-MX info published, the worm won't spread (most MTAs will drop the connection). Instead, the worm will have to route through the user domain's SMTP server, where the mail admin is more likely to catch the traffic (virus scanner on the SMTP server, or rate limiters).
It's bad if you have a different "From" address from what your SMTP server is, in which case I don't see how it could work for you.
1) Don't publish a key for your domain (downside is that you can still be joe-jobbed and nobody can verify that e-mail coming from your domain is authentic, or at least that it passed through an authorized server)
2) Use SMTP AUTH / VPN to connect to your domain's server, just as if you were in the office. (Most corporations, where you are acting as an agent of the corporation, would prefer this method.) The only time this is a problem is if you're behind a firewall of some sort, in which case a $9.95 dial-up account and unplugging a fax machine for a few minutes gets you past it. Or you can make use of the 3G hi-speed internet wireless services that are coming in a year or two.
There are options, and if your service provider doesn't provide alternatives, then you need to find another provider or bring pressure to bear on that provider. The ability to randomly forge any domain that you want onto your e-mail has been abused to death, hence the momentum behind things like Yahoo!'s proposal and the various reverse-MX / sender-authentication proposals. Most mail admins are tired enough of being joe-jobbed and dealing with bounces due to domain forgery that they're willing to make these changes.