Ask Jonathan Zdziarski

You may recognize the name Jonathan Zdziarski from a recent Slashdot book review of his book Ending Spam. Aside from his DSPAM spam filter Jonathan has also contributed several other projects to the open source community under the GNU General Public License. These projects include Verizon-Compatible SMIL Multimedia Gateway, The Reactive Automated Blackhole List Server, Apache DoS Evasive Maneuvers Module, and several others. Want to know how to effectively contribute projects to the open source community? Curious to ask another programmer about his history? Now is the time to ask. Moderators will select the top few questions that we will forward on to Jonathan sometime tomorrow. The answers to the questions will be displayed next Tuesday when we will encourage Jonathan to participate in the discussion as time permits.

Comments

Nomenclature

[Anonymous Coward]

How do you pronounce your name?

Re:Nomenclature

[Ann Elk]

Polish 'z' sounds like English 'z' as in "zoom".
Polish 'dzi' trigraph sounds more-or-less like English 'j' as in "jam".
Polish 'a' sounds like English 'a' as in "call".
Polish 'r' sounds something like an English 'r' as in "read", but it's rolled (more like a Spanish 'r').
Polish 's' sound like English 's' as in "say".
Polish 'k' sounds like English 'k' as in "kit".
Polish 'i' sounds like English 'y' as in "fully".

"Z-dzi-a-r-s-k-i" is prounounced (very roughly) "Ze-jarrsky". At least, in theory. I'm not Polish, so he may have a different opinion.

How do you pronounce your last name?

[winkydink]

that's my question.

Re:How do you pronounce your last name?

[halltk1983]

Exactly as it's spelled! ;-)

GPL 3?

[Anonymous Coward]

Seeing how Johnathan has put much of his time and effort into Open Source projects over the years, it would seem he is a good canadate for this question: What do you think about the proposed change to the GPL with the upcoming GPL 3? Is it a welcomed breath of fresh air to the Open Source Community, or will it just be a reiteration of the previous GPL? What are your thoughts and comments on the GPL 3?

--
Do you get those pesky Nigerian 419 emails? Post them here, and watch the database grow! : http://urgentmessage.org/

Future of Anti-Spam Techniques

[mxmasster]

Most antispam software seems to be fairly reactionary - wither it is based on keyword patters, urls, sender, ip, or the checksum of the message a certain amount of spam has to first be sent and identified before additional messages will be tagged and blocked. Spf, domainkeys, etc... requires a certain percentage of the Internet to adopt before they will be truely effective.

What do you see on the horizon as the next big technique to battle spam? How will this affect legitimate users on the Internet?

DIY Spam Filtering

[Anonymous Coward]

Mr. Zdziarski, it appears as if you are a supporter of use of statistical methods to filter out spam. But these filtering methods have limitations, in that there are ways of getting around these filters. Since human beings can recognize spam better than any software filter, do you not believe that more emphasis should be put on developing software that facilitates DIY spam filtering?

Re:DIY Spam Filtering

[LnxAddct]

Actually, filters do a much better job then humans. One human can't deal with the job, its too overwhelming in many cases, therefore one human isn't capable of filtering effectively, even if they were capable, most would rather pay someone else to do it for them. The next solution, say AOL highered 1,000 people to filter spam, each one of them would disagree with what is spam and what isn't. Some people might want to get car deals, and sports information, or porn and viagra, others won't. The spam filter is often personalized, where as someone sifting through thousands of different people's emails can't be. So let's review, you don't have the time, others don't have the knowledge, the only solution is a filter that learns your habits and works in milliseconds. The only thing better then a filter would be you, no other human being, therefore saying humans are better is not accurate. You couldn't just sit a random human infront of a random person's inbox and say filter this.
Regards,
Steve

see any decrease in spam lately?

[krelyk]

Have you noticed any decrease in the amount of spam since a few of the hardcore spammers have finally been prosecuted? I always wonder if scare tactics will work against these guys, or if they will just move their colo to some small country offshore where it becomes harder to press charges.

SpamAssassin Tools - AK-47s, Knives, or Nukes?

[billstewart]

So when you're trying to assassinate spammers, do you favor precisely targeted tools like knives, medium-scale tools like AK-47s, or nuke-them-from-orbit solutions?

I guess the more serious version of this question is the tradeoff of precision and false negatives vs. overkill and false-positives. For instance, my email provider lets me pick country-blacklists, so I reject all email from China, Korea, and Nigeria, where I don't know anybody, and Japan gets accepted with extra filtering, because I know a couple people there who normally don't send me mail - it's not quite a nuke-Asia-from-orbit approach, because people who actually do want mail from people in China can accept it, but people who don't can reject it all and lose the occasional message from a friend at a cybercafe.

Insights into the corporate mentality for OS / GNU

[RealisticCanadian]

Jon, your acheivements thus far are impressive. I am personally most impressed by your adherence to Open Source Solutions in a corporate environment.

I myself have had numerous interactions with less-than-technically-savvy management-types. Any time I bring up solutions that are quite obviously a better technical and financial choice over software-giant-type solutions; conversation seems to hit a brick wall. The ignorance of these people on such topics is astounding, and I find many approaches I have tried seem to yield no results in the short term. "Well, yes, your example proves that we would save $500,000 per year using that Open Source solution. But We've decided to go the Microsoft (or what-have-you) route."

With your track record, I can only assume you have found some ways to overcome this closed-mindedness.

I would greatly appreciate any input you have on this; from the perspective of someone who has overcome this obstacle before.

Spam Delays

[Malyven]

How do you deal with spam checking software causing a delay at the point where you do the spam filtering? As communication backup becomes more important in the business place you have some companys dealing with literally millions if not billions of emails a day. Even an efficent filter will take to go through that many emails, How do you deal with this?

Bogofilter And Standardized Bayesian Testing

[Goo.cc]

I have two questions:

1. In your new book, you basically state that Bogofilter is not a bayesian filter, which was news to some of the Bogofilter people I have spoken to. Can you explain why you feel that Bogofilter is not a bayesian filter?

2. Bayesian filters have been around for some time now but there still seems to be no standardized testing methods for determining how well filters work in comparison to one another. Do you think that comparitive testing would be useful and if so, how should it be performed?

Thanks Jonathan.

Re:Bogofilter And Standardized Bayesian Testing

[dozer]

From his paper:

http://www.nuclearelephant.com/papers/justifying.h tml

"This family of filters includes the now-popular Bayesian filters (pronounced "bay zee in") as well as other filters using statistical analysis to filter spam (such as Markovian classifier CRM114 and Chi-Square Bogofilter)."

That's why Bogofilter is not Bayesian.

I definitely like the second question.

What about legitimate mailing list software

[skazatmebaby]

Jonathan,

I develop and manage a lightweight Open Source Application that's used to send announce only and discussion mailing lists, similar to the Mailman and Majordomo projects. It's very popular and has a loyal following.

What advice do you have as a developer of this program to:

* Help my users send legitimate messages (either by education (specifically) or by programming techniques)

* Help Spam Filtering Software check the messages my program sends out for possible abuse

* Be a part of the solution to sending legitimate messages to many people, rather than perhaps be part of the problem.

I understand that any tool can be circumvented and abused and I do believe context always plays a part in how to judge something as Good or Bad. I'm sure like many different types of software, Spammers are a problem for my business as well.

I find myself in an interesting position, where I can change how many email messages are sent out. If I can send "better" email messages that are not filtered as spam if they are legitimate and can stop possible abuse of my program, I can help in a solution to people who would like to send out announce only and discussion email messages.

Thanks for your time.

Simple Question

[jnaujok]

The SMTP standard that we use for mail transfer was developed in the late 70's - early 80's and has, for the most part, never been updated. In that time period, the idea of hordes of spam flowing through the net wasn't even considered.

It has always been the most obvious solution to me that what we really need is SMTP 2.0, where a server only accepts mail from a user that can authenticate themselves with a name and password. A server can also accept mail from another server, but only for mail directed at legitimate users on it's system. Mail servers would have to register with a central authority, and must include their active IP address in that registration. Any attempt to deliver mail from an unregistered server is bounced.

Wouldn't this simple fix stop 99% of spammers in their tracks? Isn't it about time we updated the SMTP standard?

Re:Simple Question

[stoborrobots]

Wouldn't this simple fix stop 99% of spammers in their tracks?

No, it wouldn't.

Firstly, what this prevents is the direct sending of mail from unregistered IPs to a destination host, or via an open relay. However, the bulk of the spam out there today (not this time last year, when the profile was completely different...) does not come from open relays. Eliminating both open relays and direct port 25 connections from non-mailserver IPs would only eliminate one simple route for spam.

The bulk of todays spam comes from trojaned machines (botnets) which are able to spew forth spams as directed by their controlling server. Given that these bots are able to hook into things like MAPI and/or read configuration files for kmail/evolution/mutt which can contain smarthost IPs and login details, they are able to send as much mail as they like pretending to be the authorised owner of the machine in question. At this point, there is nothing left to distinguish a spam email from any other email originating on that computer.

Until you can prevent every machine out there from being compromised thus, or convince the entire world that clicking the "save password" button is evil, you cannot prevent spam disguising itself as legitimate mail.

Further, assuming that we don't consider these smart viruses which pick up the user/password settings, there's nothing preventing the spammers from registering "sdfkjwnwfsinlsd.biz", configuring the "official authorised mailserver IP" to be that of a compromised machine somewhere (or an army of them) and having those spew forth spams for 24 hours (or even 1 hour... ) Getting his 100million messages out there, then cancelling the domain, and leaving no trace of anything worth blocking.

Greylisting comes the closest to being an effective spam blocker, but it would be trivial to implement a spam-bot which played the greylisting game... and once it passed the greylist test, it could then spam that mailserver for a while, confident that its messages were getting through.

Spam cannot be avoided by purely technological means. As long as a human can message you using only a computer, a spammer can make that computer spam you.

Re:my thesis

[hobbesx]

For my thesis ... I am testing some additions and tweaks ... what are some ... ?

Additionally, please format any comments of said tests in a double-spaced Word document in at least 1,500 words. Please cite references!
Thanks again!

Freedom of speech.

[Sheetrock]

In the past, I've heard it suggested that anti-spam techniques often go too far, culling good e-mail with the bad and perhaps even curtailing 1st Amendment rights. Clearly this depends on what end of the speculum you're on, but recent developments have given me pause for thought on the matter.

For example, certain spam blacklists would censor more than was strictly necessary (a subjective opinion, I realize) to block a spammer -- sometimes blocking a whole Class C to get one individual. This would cause other innocent users in that netspace to have their e-mail to hosts using the blacklists silently dropped without any option of fixing the problem besides switching ISPs.

This is an extreme example, but most anti-spam approaches have the following characteristics:

• They are implemented on a mailserver without fully informing the users of the ramifications (or really informing them at all)
• They block messages without notification to the sender, causing things to be silently dropped
• Even if the recipient becomes aware of the problem, few or no options are given for the recipient to alter this "service"

Recently I had to fix an installation where daily messages from a particular host stopped appearing in a mailbox. This system was connecting with an ISP that had offered no spam filtering and had been using a client-based Bayesian classifier with great success, but suddenly the mail coming into the system had scaled back by a factor of ten. Sure enough, the ISP installed a server-based spam filter which took out most of the spam and a good deal of the legitimate mail -- they had a (not well publicized) means of accessing the account settings and turning off the filter, and a holding tank for mail classified as spam, but beyond the last two weeks everything was thrown out.

I'm curious about what you think about server-based approaches vs. client-based approaches to spam classification and filtering and if, maybe, the cure is worse than the disease.

Christian Beliefs

[Anonymous Coward]

This is arguably out of scope for this interview, but I still feel it's something many Slashdotters would be interested in hearing about.

On your webpage you have an essay describing your Christian beliefs and why you have them. You say many things that most Slashdotters (and nerds and scientist in general) regard as utterly ridiculous. You think the earth is no more than 10,000 years old, you think Christianity is logical, you regard the Bible as a historial document, etc.

No doubt you are aware of the fact that most nerds disagree with you on these things. Indeed, they might even consider you "crazy" for holding them.

Without going into the truths of the beliefs in question, which I'm sure will be debated enough in the Slashdot thread anyway (and I hope you'll join in), what do you think the reason is that so many scientists, nerds and people otherwise rather similar to you think your beliefs are obviously incorrect? Do you think they are all deluded? Do you agree that there might be a possibility that your beliefs are not rational (again, without going into whether or not they are so)?

Best regards,
an AC

Re:Christian Beliefs

[adrianmonk]

Do you agree that there might be a possibility that your beliefs are not rational (again, without going into whether or not they are so)?

I think he sort of answers that question in the essay you linked to. He says that "it is true that Christianity is ultimately based on faith".

There are many philosophical viewpoints on what are valid ways of obtaining knowledge. Some people think the only valid source of knowledge is empirical observation and rational thought. Others think that, if there is such a thing as a supernatural being, that being could impart knowledge to people through some sort of mystical revelation, i.e. God controls the universe, so if he wants, he can make you know things. (It's a pretty reasonable conclusion if you first assume God does exist and does control the universe.) Some people think empirical observation and spiritual revelation are both valid, but if the two are in conflict, revelation should take precedence.

These different viewpoints are differences in philosophy. I learned in computer science class (when I learned about diagonalization and the incompleteness theorem) that logical thought cannot give you all the answers. There are true statements that logic can't lead you to and can't support. Many people during The Enlightenment believed that the forward march of reason was inexorable, and reason could, given enough time, solve any problem. They were wrong, although they weren't even proven wrong until less than 100 years ago.

My point is, the question of what avenues for obtaining knowledge are valid is an open philosophical question. (One might even say a timeless question.) There have been relatively recent developments that have changed our views of this question.

In light of that, is it wrong thinking to believe in faith over reason? Maybe it is, or maybe it isn't. But many Christians have a pretty simple philosophy on it: they believe in faith and revelation over reason, but they do investigate Christianity in an intellectual sense enough to be sure that it's a defensible, basically consistent point of view. (It doesn't have to be perfectly consistent and complete, because none of the other views of the world are either.)

history of DSPAM

[passion]

I recall hearing a story that you created DSPAM as a response to the trashy emails that your religious leader was receiving. I also see that your religion plays a large role in your life. I'm curious, how a thinking, logical, Christian such as yourself feels about the "intelligent design" movement?

Is this a misinterpretation of scripture? A reaction filled with fear against science? An attempt to distance ourselves from animals so that the atrocities occuring in modern industrial-meat production can be justified? Or is it a revival of much-needed spiritual values in our country?

In addition, I'm curious what your take is on the Intelligent Falling theory?