Slashdot Mirror

← Back to Stories (view on slashdot.org)

Spam Solutions from an Expert

Posted by CowboyNeal on from the separating-the-wheat-from-the-chaff dept.

Mod N writes "SecurityFocus has posted a nice survey of anti-spam technologies by spam expert Neal Krawetz, in which he delves deeply into the specifics and pitfalls of the numerous proposed solutions. Krawetz makes it obvious that securing the email infrastructure is a very complex problem that many of the current (simple) solutions can't solve alone."

22 of 420 comments (clear)

  1. Proof? by monstroyer · · Score: 5, Interesting
    The marketing myth emphasizes two misconceptions: (1) a human must perform the challenge, and (2) these problems are too complex for automated solutions. In truth, most spam senders ignore these CR systems because they do not account for a large recipient base, not because the challenge is difficult. Many spam senders use valid email addresses for their scams or for validating mailing lists. When CR systems begin to interfere with spam operations, spammers will automate the responses to these challenges.

    Excuse me, what? Where's the proof? That's quite a brave statement to be making considering i've never seen this cracked, ever.

    I challenge someone to find an automated response to C/R.

    I did hear of a theory where C/R was being cracked by taking the C/R image, posting to a porn session, and letting a seeing person do the work. However, i've yet to witness this in practice. Show me the automated response to C/R that exists beyond a blog theory, and i'll believe. Until them, i hardly consider it "marketing hype".

    1. Re:Proof? by michaeltoe · · Score: 3, Interesting
      Yeah, true... I didn't really provide proof... but proof would require me to go out and program something fairly complicated, and I'm in the middle of midterm exams.

      The point I was making is that, while noone has done it yet, there's no theoretical reason why it shouldn't be possible.

      It's like saying "Oh, that mountain's to big, no one will ever climb it." -- If people are motivated enough, they can accomplish just about anything... and spammers seem clearly motivated.

    2. Re:Proof? by LostCluster · · Score: 4, Interesting

      Yes, but such a human-check is unlikely to be beaten by a computer 100% of the time. If a log of the failed challenge attempts is kept, the source of repeated failed challenges can be ruled out from getting any more challege attempts, or even just one failed challenge with hundreds of successful ones coming from the same IP space... then the hacker source cna be flagged and ruled out.

      The best defenses involve several lines so that when the first gets beaten, another one tightens up against whatever the first line learned from its defeat...

    3. Re:Proof? by michaeltoe · · Score: 3, Interesting
      True, or you could just come up with an authentication method that doesn't involve visually identifying numbers and letters... like, showing a picture of an apple, and having people type in the name of the fruit.

      Then again, that becomes less reliable and more ambiguous. You could keep on pushing for more difficult to interpret puzzles, and the technology to interpret it can just push back. People will just end up getting annoyed by it.

      Sooner or later that idea runs out of gas... it's only a temporary solution.

    4. Re:Proof? by chrisbtoo · · Score: 4, Interesting

      Well, this is by no means a proof, but maybe a method.

      1) Get image. I followed your link and got given this image.

      2) Pre-process. I loaded it into the GIMP and did Image->Mode->Greyscale, which yielded this image. Then I did Layer->Colours->Threshold, which yielded this image.

      3) Match characters. At this point, you have a monochrome image, in what appears to be a known font. The chars don't even appear to overlap, so a simple 1-for-1 match is achievable. Scan left-right, top-bottom until you see a 10x10 (or whatever) section with a black pixel. Scan down and right from that pixel until you see a character.

      I don't have the time to code it up right now, but if someone wanted to pay me to do it, I'm pretty sure it's acheievable - not least because a whole bunch of the more difficult code is available for me to use under the GPL.

      --
      Registering accounts later than some other chrisb since 1997
  2. Open Relays by QuePasaCalabaza · · Score: 4, Interesting

    The truth is 90% of spam comes from open relays, that is SMTP servers that can be tricked (a bit like lying to a 5 year old) into accepting and sending out massive ammounts of mail. Simply blocking open relays using The Open Relay Database at http://www.ordb.org/ or other open relay checking utility will save you lots of time if you run your own mailserver. When we can bascially negate the usefulness of open relays to spammers, they will then have to rely on their own bandwidth for the most part providing they cannot comprimise other "closed" relays.

  3. Let's use the Patriot Act for the benefit of good by mao+che+minh · · Score: 5, Interesting

    I am in full support of using the broad-powered, freedom crushing Patriot Act in apprehending and imprisoning spammers. We might as well get some good out of it.

  4. Good old fashioned riddles by KalvinB · · Score: 4, Interesting

    My free anonymous (as in they can only be traced back to a common e-mail account on my server) e-mailer uses a simple quiz to keep spammers out.

    The form page records the IP address of the visitor along the with the question number they were given in a file named with the IP address. That number is never sent to the client. When they hit submit the file of their IP is opened, the question number is read in and the answer given by the user is compared to the stored answer. The file is then deleted and if the answer was correct the e-mail is sent. Otherwise it's not.

    This forces my custom form to be used to be able to send the e-mails. And it's not possible to simply keep refreshing the submit page to keep sending the message.

    And the challenge is in the form of old riddles and a couple new ones like "what's your favorite color?"

    Things a bot would never get but that anyone who knows how to use Google can. Someone would have to program a custom bot with the answers in order to even attempt to spam. And even then since everything goes through my mail server nobody is going to sneak garbage past me for long and I know who your ISP is.

    I also include a disclaimer with every e-mail. It'd be quite silly for me not to.

    Ben

    --
    Work Safe Porn
  5. Fix SMTP! by schnarff · · Score: 4, Interesting

    Well, at the risk of sounding like a broken record, SMTP itself is the problem -- it's badly broken, security-wise, and needs to be fixed. It's going to be painful to move to a new mail standard, or to change SMTP so that it's not broken, but that's what needs to happen to stop spam. Thankfully, our friends the Russian Mafia and the ever-growing number of Windows zombie machines are making spam levels so great that, sometime soon, spam will represent such a large percentage of e-mail traffic that fixing SMTP will be necessary, not just something mail admins like myself wish for.

    BTW, does anybody have a good figure on what percentage of all e-mail spam represents these days? I'm talking about *all* traffic, too, not just what ends up in peoples' Inboxes after all the filtering going on out there has done its job.

    --
    How To Get Humans To Mars
  6. Having experience, I can answer 1.2.1 by snakecoder · · Score: 5, Interesting

    I am not recommending mailblocks, I belive there is a sourceforge project called TMDA which does the same thing. Having said that, my experience comes from using mailblocks:

    -cr deadlock: This does not exist because when you e-mail someone in a challenge and response system, it automatically assumes they are friendly. So if they have a challenge and response system, it will make it into your inbox, because you e-mailed them first

    -automated systems He is correct here. Personally I hate when friends submit my e-mail to third parties without my consent so I do not mind missing these e-mails. I have caught a few while searching my pending folder, and inform my friends I rather have them e-mail me directly.

    -interpretation challenge I believe he is wrong here because of a fundamental issue. When dealing with spam filters, the onus of working out refinements is left to the spamee, to make sure they filter out all spam. If a spammer adds a new technique, they get around the filter. With challenge systems, you have a few methods waiting as backup. When a spammer finally figures out how to read your words through AI, you simply change the challenge system and they are back to square 1 in trying to figure out how to defeat. As long as you have a few methods waiting in the wings, the spammers can easily be defeated, and have huge amounts of work to do.
    if you doubt this, write an AI system to defeat hotmails gifs. Now what if the next day instead of showing a word, they show you a picture of 3 fire trucks and 2 police cars and ask you how many police cars are in the picture, etc ...

    --
    -Nuke the moon
    1. Re:Having experience, I can answer 1.2.1 by vanyel · · Score: 3, Interesting

      He's also wrong about using certificates:

      1. certs don't require a connection to the cert authority. You get their CA cert ahead of time and then trust certs signed by it.

      2. Responsible CA's won't grant certs to spammers because people will stop trusting their certs

      3. If spam does come in signed, then they are trackable and the backlash will quickly shut them down.

  7. I managed to appall a colleague today... by Ungrounded+Lightning · · Score: 4, Interesting

    Was out to lunch with three colleagues today and the subject of anti-spam measures came up.

    I managed to appall the one from Berkeley by suggesting that the most practical solution was probably a moderate-size bomb.

    B-)

    But seriously:

    In an arms race, weapons eventually defeat armor. Spam will continue until two real-world things are BOTH brought to bear on spammers:

    - Economics
    - Muscle

    If a governmental solution applying both is not forthcoming soon, I predict that there WILL be vigilantism.

    In fact we're already seeing it.

    For instance: Subscribing the Detroit area spammer and his lawyer to enough real-world junkmail lists to bury his bills and other US Main correspondence in several daily truckloads of catalogues and other solicitations.

    Soon to come: Retaliatory information-war software directed at DDoSer / spammer zombi-net machines. (As discussed in a recent Slashdot article.)

    --
    Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
  8. dont forget ... by segment · · Score: 3, Interesting

    I don't bother getting too deep into downloading too many 'new improved!...' filters. I block entire damn countries/netblocks. Besides I don't know anyone in korea, brazil, china, nor any other one of the massive spamming countries. I configure postfix to filter out a lot and the minute I receive one spammed message, I always whois -h whois.apnic/arin/ripe/lacnic offender and block their entire range. I also have spam assassin running and I have to admit I get about maybe... maybe... 4 spams a week not kidding. Again though this is my personal machine.

    block return-icmp (8) in proto tcp from 24.76.0.0/14 to any port = 25
    block return-icmp (3) in proto tcp from 81.208.64.0/18 to any port = 25
    block return-icmp (4) in proto tcp from 163.121.163.0/22 to any port = 25
    block return-icmp (4) in proto tcp from 82.77.83.0/24 to any port = 25
    block return-icmp (4) in proto tcp from 61.247.224.0/19 to any port = 25
    block return-icmp (4) in proto tcp from 217.132.0.0/17 to any port = 25
    block return-icmp (4) in proto tcp from 62.103.204.32/27 to any port = 25
    block return-icmp (4) in proto tcp from 210.111.224.0/17 to any port = 25
    block return-icmp (4) in proto tcp from 144.135.0.0/8 to any port = 25
    block return-icmp (4) in proto tcp from 195.166.224.0/18 to any port = 25
    block return-icmp (4) in proto tcp from 61.228.0.0/8 to any port = 25
    block return-icmp (4) in proto tcp from 207.144.229.0/24 to any port = 25
    block return-icmp (4) in proto tcp from 193.252.22.160/28 to any port = 25
    block return-icmp (4) in proto tcp from 200.0.0.0/8 to any port = 25
    block return-icmp (4) in proto tcp from 209.202.192.0/18 to any port = 25
    block return-icmp (4) in proto tcp from 83.32.0.0/8 to any port = 25
    block return-icmp (4) in proto tcp from 68.38.64.0/8 to any port = 25
    block return-icmp (4) in proto tcp from 219.240.0.0/10 to any port = 25
    block return-icmp (4) in proto tcp from 195.57.218.0/25 to any port = 25
    block return-icmp (4) in proto tcp from 129.79.245.98 to any port = 25
    block return-icmp (4) in proto tcp from 24.150.0.0/19 to any port = 25
    block return-icmp (4) in proto tcp from 24.205.28.0/21 to any port = 25
    block return-icmp (4) in proto tcp from 220.116.0.0/8 to any port = 25
    block return-icmp (4) in proto tcp from 200.128.0.0/9 to any port = 25
    block return-icmp (4) in proto tcp from 212.81.64.0/17 to any port = 25
    block return-icmp (4) in proto tcp from 32.10.58.0/19 to any port = 25
    block return-icmp (4) in proto tcp from 210.183.110.0/20 to any port = 25
    block return-icmp (4) in proto tcp from 134.196.0.0/16 to any port = 25
    block return-icmp (4) in proto tcp from 24.60.88.0/23 to any port = 25
    block return-icmp (3) in proto tcp from 24.190.8.0/24 to any port = 25
    block return-icmp (2) in proto tcp from 24.98.77.0/23 to any port = 25
    block return-icmp (2) in proto tcp from 24.173.29.0/23 to any port = 25
    block return-icmp (2) in proto tcp from 205.206.176.0/23 to any port = 25
    block return-icmp (2) in proto tcp from 172.128.0.0/10 to any port = 25
    block return-icmp (2) in proto tcp from 200.171.99.0/24 to any port = 25
    block return-icmp (2) in proto tcp from 200.171.97.0/22 to any port = 25
    block return-icmp (2) in proto udp from 200.171.97.0/22 to any port = 25
    block return-icmp (2) in proto tcp from 68.62.80.128/25 to any port = 25
    block return-icmp (2) in proto udp from 68.62.80.128/25 to any port = 25
    block return-icmp (2) in proto tcp from 218.76.0.0/17 to any port = 25
    block return-icmp (2) in proto udp from 218.76.0.0/17 to any port = 25

    --
    MoFscker
  9. Maintenace the problem by powerpuffgirls · · Score: 4, Interesting

    As stated in the article's summary, the main problem with most spam-filter is the need for constant maintenance. We need a solution that requires ZERO maintenance by the joe-users, and yet cost-effective enough to implement.

    My ISP seems to have a so-called "Watch Dog" spam filter, where they actually hire people to read spams and filter them manually, that's probably the most effective way to filter spam, but I wonder if it is cost-effective though.

  10. Do not call ... by Ephboy · · Score: 5, Interesting

    Prior to this October, telemarketing calls were a national scourge. Amazingly, since we signed up for the Do-Not-Call list, we've only received 2 illegal calls. I'm rather surprised, in fact, at the relatively uniform acquiescing to this law. While spam, coming from all corners of the earth and is more anonymous, will be harder to enforce, some law with real teeth may be a good start.

  11. Reputations by grotgrot · · Score: 4, Interesting

    The only thing that will work in the end is some sort of distributed reputation management system. To a certain extent that is what RBLs do, except they are on or off. SpamAssassin does offer shades of grey to the RBLs (differening weights to each one).

    To a certain extent this is what we already do in real life. We 'judge a book by its cover' as a first pass (for example people will often walk past a beggar in the street completely ignoring them) and then include other factors. How polite they appear, where they are from, recommendations from friends etc

    All other mechanisms suffer from a determined spammer being able to get around them as the article pointed out. Any mechanism that prevents some spammers makes things more lucrative for the rest.

    1. Re:Reputations by leviramsey · · Score: 4, Interesting

      I just devised a setup that might be interesting:

      • Users (sysadmins) of the blacklist submit two lists of IPs, good (non-spammers) or bad (spammers).
      • When a server receives a mail, it checks with the list to see on which lists the IP appears as good and on which it appears as bad.
      • The user marks the mail as ham or as spam. A Bayesian algorithm then determines which lists are trustworthy for marking spam hosts.
      • Filters could then /dev/null mail based on this bayesian score

      The idea is essentially to allow a collaboratively developed decentralized blacklist and whitelist to develop. Spammers will either submit the IPs they use to this list or not submit them; if they do submit them, then a "good" report from them will eventually be taken as a strong sign of spamminess. If they don't, then nothing happens, but presumably "trustworthy" blacklists would list them.

      Thus, a user in Brazil, where they would be receiving lots of legit mail from Brazilian IPs would not find a blacklist that listed all of LACNIC to be a strong indicator of spamminess. The effects of blacklisters who maliciously put enemies into their blacklist would also be reduced, if not eliminated.

      A suggested implementation detail on the blocking would be to make it random; that is to say that 100% of the mail with a 100% probability of being spam gets dropped, 99% of mail with a 99% probability gets dropped, 97% of mail with a 98% probability gets dropped, 94% of mail with a 97% probability gets dropped, 90% of mail with a 96% probability gets dropped, etc. according to this function:

      d(p) = d(p+1)*p/100, where d(100) = 100, and 73<=p<=100

      This would allow for a degree of "retraining" in the event of false positives (since a /dev/null'd mail cannot be retrained from!).

  12. SPF Anyone? by ignoramus · · Score: 3, Interesting

    One proposed solution I would love to see getting more attention is SPF ("Sender Policy Framework"), which allows each domain admin to specify their email sending policy using existing infrastructure.

    See the SPF site or read this month's Linux Journal to find out more.

    Executive summary of SPF: Just use DNS to specify where mail from your domain may originate from. If everyone used this, we could have domain blacklists that actually work.

    Do an "nslookup -type=txt psychogenic.com" to see an example entry. And if you manage any domains, please consider doing the same.

  13. challenge-response handling being outsourced by SuperBanana · · Score: 3, Interesting
    I did hear of a theory where C/R was being cracked by taking the C/R image, posting to a porn session, and letting a seeing person do the work.

    I had a chat with a Veep that was hired on to a company I used to work at. Very down to earth guy, very friendly. We got to talking about spams and semi-legitimate emailings to customers, etc.

    He had one very interesting tidbit; stick with me for a sec here. Most companies outsource their semi-legit stuff because they get reported as spammers and whatnot, or it bogs down their email server/network, etc. No surprise there- however, the interesting tidbit is that the outsourcing companies turn around and outsource to Indian firms for handling the bounces. There's literally a room full of people in India, sitting there answering those challenge/responses and updating the client's customer email list(unlike spammers, it really is in their best interests to minimize failed deliveries). It sounds "expensive", but it's not, considering how few people use challenge/response systems. Further- a reasonably smart human can get familiar with all the various systems quickly(an hour or two, I'd guess, tops) and probably process close to a message every few seconds with a client program set up to do that limited functionality smoothly. Best part- if your client does several mailings, unless the recipient goes in and removes you, you're clear for future emailings.

    --
    Please help metamoderate.
  14. Interview with a spammer by dbIII · · Score: 3, Interesting
    The truth is 90% of spam comes from open relays, that is SMTP servers that can be tricked
    A couple of years ago I didn't have a job - and a government sponsored job database came up with a listing for a job using computers in the "adult" industry. I went along to an the job interview, and found the job would have been setting up a dozen modems on a linux box and writing a program to scan for open relays (he'd apparently paid US$10,000 for a list). All income would have been undeclared, and some dodgy accounting involving payroll in the name of tourists would have gone on. Some background checks on the employer turned up a few interesting things as well as birthplace, education, home address etc. It looked like a may have had a choice between becoming a spammer and never getting paid for it, or losing my unemployment benefits (the consequence of turning down a job offer in my country). Another, actual legitimate job came up for a dying dot-com, so I never had to argue with beuracrats as to why I had turned down a job.

    Oddly enough the spammers name was "Fagin", as in the Oliver Twist villain, and he was born with that name.

  15. SPAMfighter works for me... by Alex_Ionescu · · Score: 3, Interesting

    The big problem with mail filters, as the article mentions, is that they need to be updated when new spam technologies appear... and there's also a lot of false positives... I gave SPAMfighter a try (from www.spamfighter.com) and although it was a bit worse at finding spam (At first), I never got any false positives. The way it works is that the "filters" are actually some kind of hash that users submit whenever they block or unblock an email (it analyses the whole content I think, not just the text). So if a new type of spam technique appears, the users will just block it. And unlike many other client-side plugins, it actually works on Outlook Express.

    Another one I recomment is Spambayes...but there's the problem with false positives. All the other ones I've tried are utter crap.

    Best regards,
    Alex Ionescu
    Relsoft Technologies

  16. CR deadlock by Skim123 · · Score: 3, Interesting
    Another deadlock case, which happened too many times in my experiences with C/R:

    • Alice sends a message to Bob. Alice is not in Bob's whitelist, so Bob's C/R anti-spam system sends a challenge to Alice.
    • Alice doesn't use C/R, but rather a filter. Her filter, unfortunately, marks Bob's challenge as spam. Since Alice is only a computer novice, she does not know how to check his Junk Email folder, and therefore never receives Bob's challenge, hence Bob never gets Alice's email. Alice, who is blissfully ignorant of the "behind-the-scenes" happenings, thinks Bob just is trying to ignore her. So she sends another email, which is, of course, not received by Bob. And she sends another. Still, no response from Bob. Alice takes it personally and decides if she does ever hear from Bob again she won't be going on a second date with him no matter what.
    --

    I could not justify my existence if I were a turkey farmer. Would I terminate myself? Undoubtably, yes.