Slashdot Mirror
Jonathan Zdziarski Answers
Wednesday we requested questions for Jonathan Zdziarski, an open source contributor and author of the recently reviewed book "Ending Spam." Jonathan seems to have taken great care in answering your questions, which you will find published below. We have also invited Jonathan to take part in the discussion if he has time so if your question didn't make the cut perhaps there is still hope.
Winkydink asks:
How do you pronounce your name?
Jonathan Responds:
Hi. Well first off, I'm sticking to the pronunciation 'Jarsky', however many of my relatives still pronounce it 'Zarsky' or "Za-Jarsky". As far as I can tell, my last name was originally 'Dziarstac' when the first generation of my family came over, which would have been pronounced with a 'J'. It's of polish decent, but I'm afraid I'm not very in tune with my ancestors on this side of the family. The other side of my family is mostly Italian, and they drink a lot, organize crime, and generally have more fun - so they are much more interesting to hang out with. For the past 29 years of my life, giving my last name to anyone has included the obligatory explanation of its pronunciation, history, and snickering at puns they think I'm hearing for the first time (-1, Redundant), so don't feel too bad for asking.
As far as who I am and why you should care - I guess that depends on what kind of geek you are. I've never appeared in a Star Trek series or anything (I've been too busy coding and being a real geek), so I guess that eliminates me as a candidate for public worship in some circles. I guess if you're into coding, open source, hacking all kinds of Verizon gear, or eradicating spam, then some of my recent projects may be of interest. If you at least hate me by the end of the interview, I'll have accomplished something.
An Anonymous Coward asks:
What do you think about the proposed change to the GPL with the upcoming GPL 3? Is it a welcomed breath of fresh air to the Open Source Community, or will it just be a reiteration of the previous GPL? What are your thoughts and comments on the GPL 3?
Jonathan Responds:
Based on the scattered information I've read about some potentially targeted areas in GPLv3 and the religious fervor with which some of these discussions have been reported, all I can say is I hope common sense prevails. Actually there's much more I can, and will, say about the subject below, but I think it's probably a good idea to summarize in advance as you may not make it through the list of details in one sitting. So in summary of all my points to come: I hope common sense prevails.
One of the things I've heard, which doesn't make much sense to me, is the idea of changing the GPL to deal with 'use' rather than 'distribution', which would affect companies like Google and Amazon. The argument seems to be that some people feel building your infrastructure on open source should demand a company release all of their proprietary source code which links to or builds on existing GPL projects. They argue that the open source community hasn't benefited from companies like Google and Amazon. Well, from a source code perspective that might be somewhat true - but if you take into consideration the fact that we all have a good quality, freely accessible search engine, cheap books, and employment for many local developers (many of whom write open source applications), the benefits seem to balance out the deficiency. Does anybody remember what the world was like before Google? None of us do, primarily because we couldn't find it - we couldn't find much of anything we were looking for on the Internet as a matter of fact, including other people's open source projects. You might not be getting "free as in beer" or "free as in freedom", but you are getting "free as in searches" and "free as in heavily discounted but not quite free books" in exchange. That's a pretty good trade. It's certainly better than having to look at pages of advertising before completing your order, or subscribing to a Google search membership. On top of this, you probably wouldn't want to see half of the source code that's out there being integrated (internally) into these projects. While I haven't seen Google or Amazon's mods specifically, I do heavily suspect that, if they are like any other large corporate environment, there are many disgusting and miserable hacks that should under all circumstances remain hidden from sight forever - many of which are probably helping ensure job security for the developers that performed the ugly hacks in the first place. Just how useful would they be to your project anyway? Probably little. And if you really believe in free software ("free as in freedom"), then the idea that someone should be required to contribute back to your project in order to use it is contradictory to that belief - you might just as well be developing under an EULA instead of the GPL.
With that said, there's a difference between freedom and stealing. I've heard that GPLv3 will attempt to address the mixing of GPL and non-GPL software. I think this clarification might be a good thing. For one, because I've seen far too many pseudo-open source tin cans and CDs being resold commercially out there, distributing many different F/OSS tools with painfully obvious closed commercial code, and finding ways to easily loophole around this part of the GPL, and secondly because it's based around implementation guidelines that really aren't any of the GPL's business. At the moment, mixing uses a very archaic guideline, which is - in its simplest terms -based on whether or not your code shares the same execution space as the GPL code. I think this needs to be reworked to give authors the flexibility to define "public" and "private" interfaces in a project manifest. We're already defining these anyway if we believe in secure coding practices. Closed source projects may then use whatever public interfaces the author has declared public (such as command line execution, protocols, etcetera) but private interfaces are off limits. One particular area where this would come in handy is in GPL kernel drivers, which need this ability to avoid tainted-kernel situations. If the author wants, they can declare dynamic linking to a library as a public interface and even make their code more widely useful without having to switch to the GPL's red-headed stepchild, the LGPL. It would also be nice to be able to restrict proprietary protocols (such as one between a client piece and a server piece, which may have originally been designed to function together) to only other GPL projects, which would essentially create GPL-bonded protocol interfaces. This won't restrict use in any way - only what closed-source projects are limited to interfacing with when redistributed.
I would also like to see the GPL's integration clause tightened down quite a bit. There are some companies out there abusing the GPL with "dual licensing". I've considered dual licensing myself in some commercial products, and I just don't believe it's being done in the right spirit much, if at all. Doing away with the possibility of integrating the GPL into a dual license could help strengthen the GPL.
Finally, I'd say mentioning a few words in the GPLv3 about submission practices to help stave off problems like this whole Sco and Linux® fiasco from ever happening again would be a good thing. People generally don't want to limit usage, but if you're going to submit code, there should be at least some submission guidelines. I suspect much of this can (and should) be done outside of the GPL, but at least covering the basics might be appropriate. It should be understood that if you're going to contribute code to the GPL, it had better be unencumbered. It's definitely something every project should already be considering already.
An Anonymous Coward asks:
Do you have any suggestions for the enthusiastic yet inexperienced? Perhaps a listing of projects in need of developers, with some indication of the level of experience suggested (as well as languages required).
Jonathan Responds:
The best projects I've seen were those started from someone with a passion for what it is they're coding. Open source development is the internship of the 21st century, and working on projects is tedious, frustrating, and likely to make you want burn out if you haven't developed perseverance. I usually suggest to people to come up with ideas for some projects they feel passionately about and make those their first couple of goals. Even if it's completely useless to anyone else, you're still likely to benefit from it yourself. Just look at my Australian C programming macros. Who would have thought that people wouldn't want to use "int gday(int argc, char *argv[])" in their code. I'm sure I learned something from that project, though I still can't remember what.
Instead of spending idle time looking for other projects to jump on, I'd spend as much time as I could in man pages, books, and coding up my own little concoctions. Even if they're stupid ones, you're likely to learn something, or even better - come up with another neat little idea you can spin off of it. Necessity is the mother of invention, so I try and figure out what it is I need, and then do it myself. That usually works. If you still can't think of anything, see if you can catch a vision for something someone else needs. I wouldn't touch anything that you're not 100% bought into and excited about for your first projects.
RealisticCanadian asks:
I myself have had numerous interactions with less-than-technically-savvy management-types. Any time I bring up solutions that are quite obviously a better technical and financial choice over software-giant-type solutions; conversation seems to hit a brick wall. The ignorance of these people on such topics is astounding, and I find many approaches I have tried seem to yield no results in the short term. "Well, yes, your example proves that we would save $500,000 per year using that Open Source solution. But We've decided to go the Microsoft (or what-have-you) route." With your track record, I can only assume you have found some ways to overcome this closed-mindedness.
Jonathan Responds:
I'm not so sure that I have convinced anyone open source was better inasmuch as I've convinced people that other people's projects were better than what Microsoft had to offer, and that's not hard for anyone to accomplish. I can strongly justify some open source projects to people because they are already superior to their commercial counterparts, but there are also a lot of crummy projects out there that should be shot and put out of my misery. I'm not one to advocate a terribly written project, even if it is open source. The good projects can usually speak for themselves with only a little bit of yelling and biting from me. So if you want to become a respected open source advocate at your place of business, I'd say the first rule of thumb is not to try and advocate crap projects for the mere reason that they're open source. Advocating the good ones will help you build a reputation. It also helps if you read a lot of Dilbert so you'll understand the intellectual challenges you'll be facing.
Some other things that I've found can help include what managers love to call a "decision matrix" which is a spreadsheet designed to make difficult decisions for them. For your benefit, this should consist of a series of features and considerations that the competitor doesn't have, with a big stream of checkboxes down the row corresponding to your favorite open source project. Nobody's interested in knowing what the projects have in common anyway, so tell them (with visual cues) what features your open source solution has over the competitor. And if you really want to get your point across clearly to your manager, do the spreadsheet in OpenOffice so they'll have to download and install an open source project to read it.
Once you've done that, and if you're still employed by now, the next thing to put together is an ROI (return on investment) comparison, which not only addresses the costs of the different solutions, but costs to support both solutions in the long run, cost of inaccuracy (if this is a spam solution for example), cost of training, customizations, and resources to manage each product. This is a great opportunity to size machines and manpower and include that in a budget forecast. Many managers are sensitive to knowing just how much extra dough it's going to cost to implement the commercial solution. At the very least, you ought to be able to prove many commercial solutions don't actually make the company much money in the long run. If speaking of cash isn't enough to convince your manager then a full analysis of low-level technical aspects will be necessary. This is simply a dreadful process, and where most open source attempts fail - because a lot of people are just too lazy to learn about the technical details of both projects and complete their due diligence. If you take the time, though, you're likely to either convince your boss or utterly confuse him - either one is very satisfying.
The biggest challenge in justifying many open source projects I've run into is finding solid support channels that your boss can rely on if you get hit by a bus (or in his mind, fired). Support is, in many cases, a requirement but not all good open source projects see the benefit in offering support. A lot of companies are willing to pay just to have someone they can call when they have a problem. So if you can find a project that's got a pool of support you can draw out of, you can not only use that to justify the project to your manager, but kick a few bucks back into the open source community. I started offering support contracts for dspam primarily because people needed them in order to get the filter approved as a solution. I think I do a good job supporting my clients that do need help, but at least half of them just pay for a contract and never use it. I certainly don't have a problem with that, and it supports the project as well as the people investing time in it.
Goo.cc asks a two parter:
1. In your new book, you basically state that Bogofilter is not a bayesian filter, which was news to some of the Bogofilter people I have spoken to. Can you explain why you feel that Bogofilter is not a bayesian filter?
Jonathan Responds:
Bogofilter uses an alternative algorithm known as Fisher-Robinson's Chi-Square. Gary Robinson (Transpose) basically built upon Fisher's Inverse Chi-Square algorithm for spam filtering, which provided some competition for the previously widely accepted Bayesian approach to this. Therefore, Bogofilter is not technically a Bayesian filter. The term, "Bayesian", however is commonly a buzzword known to most people to describe statistical content filtering in general (even if it isn't Bayesian), and so Bogofilter often gets thrown into the same bucket. CRM114 is another good example of this - many people throw it in the same bucket as a Bayesian filter, but it is configured (by default, at least) to be a Markovian-based filter which is "almost entirely nothing like Bayesian filtering". Technically, CRM114 isn't a filter at all, but a filtering-language JIT compiler (it can be any filter). I cover all of these mathematical approaches in Ending Spam, so grab a copy if you're interested in learning about their specific differences.
2. Bayesian filters have been around for some time now but there still seems to be no standardized testing methods for determining how well filters work in comparison to one another. Do you think that comparative testing would be useful and if so, how should it be performed?
Jonathan Responds:
Part of the reason there's no standardized testing methodology is because there's no standardized filter interface. A few individuals have attempted to build spam "jigs" for testing filters, but the bigger problem is really lack of an interface. About a year ago, the ASRG was reportedly working on developing such a standard - but as things usually turn out, it's an extremely long and painful process to get anything done when you've got a committee building it (take the mule, for instance, which was a horse built by a committee). This is probably why filter authors have also been hesitant to try and accommodate their filters to a particular testing jig. Incidentally, this is how I surmise that SPF could not have possibly made it through the ASRG - the fact that it made it out at all suggests that it never went in.
I think it's of some interest to compare the different filters out there, but it's also somewhat of a pointless process too. Since these systems learn, and learn based on the environment around them, only a simulation and not a test, will really identify the true accuracy of these filters - and even if you can build a rock solid simulation, it will only tell you how well each filter compared for the test subject's email. If we are to have a bake-off of sorts, it definitely ought to include ten or more different corpora from different individuals, from different walks of life. Even the best test out there can't predict how a filter might react to your specific mail, and for all we know the test subjects may have been secretly into ASCII donkey porn (which will, in fact, complicate your filtering).
This is why some people misunderstand my explanations of dspam's accuracy. All I've said in the past is "this is the accuracy I get", and "this is the accuracy this dude got". Which is the equivalent of "our lab mice ate this and grew breasts". There's no guarantee anybody else is going to get those results, though I'm sure many would try (with the mice, that is). In general, though, I try to publish what I think are good "average" levels for users on my own system, and they are usually around 99.5% - 99.8%. In other words: your mileage may vary. So try it anyway. Incidentally, I've been working with Gordon Cormack to try and figure out what the heck went wrong with his first set of dspam tests. So far, we've made progress and ran a successful test with an overall accuracy of 99.23% (not bad for a simulation).
What would be far more interesting to me would be a well-put together bakeoff between commercial solutions and open source solutions. The open source community around spam filtering really has got the upper hand in this area of technology, and I'm quite confident F/OSS solutions can blow away most commercial solutions in terms of accuracy (and efficacy).
Mxmasster asks:
Most antispam software seems to be fairly reactionary - wither it is based on keyword patters, urls, sender, ip, or the checksum of the message a certain amount of spam has to first be sent and identified before additional messages will be tagged and blocked. Spf, domainkeys, etc... requires a certain percentage of the Internet to adopt before they will be truely effective. What do you see on the horizon as the next big technique to battle spam? How will this affect legitimate users on the Internet?
Jonathan Responds:
That's the problem with most spam solutions, and why I wrote Ending Spam. Bayesian content filtering, commonly thrown into this mix, has the unique ability to grow out of your typical reactive state and become a proactive tool in fighting spam. I get about one spam per month now at the most, and DSPAM is learning many new variants of spam as it catches them; I'd call that pretty proactive. Spam, phishing, viruses, and even intrusion detection are all areas that can benefit greatly from this approach to machine learning. They will likely never become perfect, but these filters have the ability to not only adapt to new kinds of spam, but to also learn them proactively before it makes it into your inbox. Some of this is done through what is called "unsupervised learning" and not traditional training, while other tools, such as message inoculation and honey pots, can help automate the sharing of new spam and virus strains before anyone has to worry about seeing them. We haven't thoroughly explored statistical analysis enough yet for there to be a "next big technique" beyond this. The next big techniques seem to be trying to change email permanently, and I don't quite feel excited about that. Statistical tools are where I think the technology is at and it needs to become commonplace and easier to setup and run.
The problem seems to be in the myth that statistical filtering is ineffective or incomplete. Many commercial solutions pass themselves off as statistical(ish) and it seem to be contributing to this myth by failing to do justice to the levels of accuracy many of the true (and open source) statistical filters are reflecting. Any commercial solution that claims to be an adaptive, content-based solution (like Bayesian filters are) really ought to deliver better than 95% or 99% accuracy. Part of the problem is just bad marketing - most of these tools are not true "Bayesian" devices; they just threw a Bayesian filter in there somewhere so they could use the buzzword. Another problem is design philosophy and the idea that you need an arsenal of other, less accurate tests, to be bolted in front of the statistical piece. If you're going to train a Bayesian filter with something other than a human being, whatever it is that's training it ought to be at least as smart as a human being. Blacklist-trained Bayesian filters are being fed with about 60% accurate data, (whereas a human is about 99.8% accurate). So it's no surprise to me that Blacklist-trained filters are severely crippled - what a dumb combination. If you really want to combine a bunch of tools for identifying spam, build a reputation system instead. They do a very good job of cutting spam off at the border, are generally more scalable than content-based filtering, and most large networks can justify their accuracy by their precision.
Not all commercial content-based filters are junk. Death2Spam is one exception to this, and delivers around 99.9% accuracy, which is in the right neighborhood for a statistical filter. Not all reputation systems are junk either. CipherTrust's TrustedSource is one example of what I call a well-thought out system. If you must have a commercial solution, either of these I suspect will make you quite happy. As for (most of) the rest, quit screwing around and build something original that actually works.
Jnaujok asks:
The SMTP standard that we use for mail transfer was developed in the late 70's - early 80's and has, for the most part, never been updated. In that time period, the idea of hordes of spam flowing through the net wasn't even considered. It has always been the most obvious solution to me that what we really need is SMTP 2.0. Isn't it about time we updated the SMTP standard?
Jonathan Responds:
You're talking about an authenticated approach to email, and there have been many different standards proposed to do this. First let me say that, even though SMTP was drafted a few decades ago, it's still successful in performing its function, which is a public message delivery system - key word being public. There exist many private message delivery systems already, which you could opt to use, including bonded sender and even rolling your own using PGP signatures and mailbox rules. I have reservations about forcing such a solution on everybody and breaking down anonymity for the sake of preventing junk mail. Until you can sell a company like Microsoft on absolute anonymity in bonded sender and sell ISPs into putting up initial bonds for their customers (so that a ten-year old gradeschool student can still use email), I see a very large threat (especially by the government) in globalizing this as a replacement for the 'public' system. With services like gmail, where you can store an entire life's worth of email, the idea that everything you've ever said could be sufficiently traced back to you and used against you, I would rather deal with the spam. Why? Let me pull out my tinfoil hat...
It's been advertised plenty of times on Slashdot that Google stores everything about all of its queries. It wouldn't surprise me if they already have government contracts in place to perform data mining on specific individuals. How would you like, in the future, all of your email to be mined and correlated with other personal data to determine whether or not you should be allowed to fly? Buy a firearm? Rent a car? We're not very far off from that, and even less so once this correlation is made possible.
So abstract some level of anonymity at the ISP-level you say? That's just not going to happen. For one, that makes it just as simple for a spammer to abuse somebody's network and then we've gone and redesigned SMTP for no good reason. Remember, business has to be able to set up shop online fairly easily and spammers are a type of shop. So we are always going to balance between free enterprise and letting spammers roam on the network. Should we employ a CA, how much would it cost to run your own email server? More importantly - does this perhaps open the door for per-email taxes? I'd much rather just deal with spam the way we are now. For another thing, abstracted identity architectures would only give you a level of anonymity parallel to the level of anonymity you have when you purchase a firearm (where the forms are stored by your dealer, rather than filed to a central government agency). See how long it takes for the feds to trace your handgun back to you if you leave it at the scene of a crime.
You can't leave it in the ISP's control anyway. The sad truth is that most ISPs still don't care about managing outgoing spam on their network; so new spammers are being given a nurturing environment to break into this new and exciting business. I had a recent bout with XO Communications about one such new spammer who had run a full-blown business on their network since 1997 and recently decided he'd like to start spamming under the "CAN-SPAM" act (which he was convinced defended his right to spam). He included his phone number, address, and web address in the spam - I called him up and verified he was who he said he was (the owner of this business, and spamming). Provided all of this information (over a phone call) to the XO abuse rep (let's call him "Ted"), even filed a police report, and XO still to this day has done nothing. His site is even still there, selling the same crap he spams for. This happens every day at ISPs out there.
The consequences outweigh the benefits. The people who drafted the SMTP protocol probably thought of most of these issues too. A public system can't exist without the freedom to remain anonymous, ambiguous, and the right to change your virtual identity whenever the heck you like.
Sheetrock asks a two parter:
1. In the past, I've heard it suggested that anti-spam techniques often go too far, culling good e-mail with the bad and perhaps even curtailing 1st Amendment rights. Clearly this depends on what end of the spectrum you're on, but recent developments have given me pause for thought on the matter. For example, certain spam blacklists would censor more than was strictly necessary (a subjective opinion, I realize) to block a spammer -- sometimes blocking a whole Class C to get one individual. This would cause other innocent users in that net space to have their e-mail to hosts using the blacklists silently dropped without any option of fixing the problem besides switching ISPs.
Jonathan Responds:
A lot of blacklists have started taking on a vigilante agenda, or at the very least rather questionable ethical practices. Spamhaus' recent blacklisting of all Yahoo! Store URLs (and Paul Graham's website) is a prime example of this. As long as you're subscribed to human-operated blacklists, you're going to suffer from someone's politics. That's one of the reasons I coded up the RABL, which is a machine-automated blacklist. There is also another called the WPBL (weighted private block list). As the politics of the organizations running human-maintained lists get worse, I think more of these automated lists will start to pop up. Machine-automated blacklists don't have an agenda - they have a sensitivity threshold. It's much easier to find the right list with the right threshold than it is to find the right politics (and then keep tabs on them to make sure they don't change). The RABL, for example, measures network spread rather than number of complaints. If a spammer has affected more than X networks, they are automatically added to the system, and removed after being clear for six hours (no messy cleanup). Another nice thing about machine-automated blacklists is that they are really real-time blacklists, and capable of catching zombies and other such evils with great precision.
NOTE: I haven't had time yet to bring the RABL into full production, but am interested in finding more participants to bring us out of testing.
2. This is an extreme example, but most anti-spam approaches have the following characteristics: They are implemented on a mail server without fully informing the users of the ramifications (or really informing them at all). They block messages without notification to the sender, causing things to be silently dropped. Even if the recipient becomes aware of the problem, few or no options are given for the recipient to alter this "service".
Jonathan Responds:
I've run into issues like this with my ISP (Alltel), and I agree with a lot of what you're saying. In the case of Alltel, not only are they filtering inbound messages using blacklisting techniques and other approaches they don't care to tell me about, but they are filtering outbound messages as well. I had to eventually give up using their mail server because I could not adequately train my own spam filter (Alltel would block messages I forwarded to it). To make matters worse, there is no way to opt out of this type of filtering on their network, even though I offered to give them the IP address of my remote mail server. This clearly does affect their customers, and I feel there are censorship, violation of privacy and denial of service issues all going on here. (Somebody please sue them by the way).
Fortunately, I don't think this issue is as wide spread as you might think. Many of the ISPs and Colleges I've worked with are, unlike Alltel, very dedicated to ensuring that their tools only provide a way for their users to censor themselves. I think this ought to be a requirement for any publicly used system. Specifically...
1.The user must be able to opt in or out of all aspects of filtering
2.All filtering components and their general function must be fully disclosed
3.The user must be able to review and recover messages the system filtered
Opting out of RBLs is as easy as having two separate mail servers and homing on the box you want. I would strongly advise to ensure that your solution is capable of receiving instruction from a user to improve its results, but it is still very difficult to scale this to millions users. At the very least should be fully disclosed, recoverable, and removable.
An Anonymous Coward asks:
Without going into the truths of the beliefs in question, which I'm sure will be debated enough in the Slashdot thread anyway (and I hope you'll join in), what do you think the reason is that so many scientists, nerds and people otherwise rather similar to you think your beliefs are obviously incorrect? Do you think they are all deluded? Do you agree that there might be a possibility that your beliefs are not rational?
Jonathan Responds:
The beliefs I hold as a Christian aren't always the popular ones, but they're certainly valid arguments for anyone who cares to ask about them (not that that has happened). When you read about someone's beliefs, you have the option to engage in discussion, or to filter his or her beliefs through your own belief system. The former option involves cognitive thought, however the latter is how most people today respond to anything that even smells religious. And I say this coming from the position of someone who hasn't tried to shove my beliefs down anyone's throat - I merely documented them on my personal website. That tells me that some people don't believe I have the right to my own beliefs - how asinine is that?
But to address the question, my beliefs aren't based on some religious intellectual suicide. In fact, the Bible teaches that you should know what you believe and why, and that you should even be prepared to give a defense for your faith - so the Bible encourages sound thinking and not some pontificated ideal structure as many quickly dismiss it as. I didn't dumb down when I became a Christian. In fact, it felt more like I began to think more clearly. I was raised in the same public school system as everyone else and didn't even know who Jesus Christ was until around my junior or senior year of high school. I've read from my early days in Kindergarten how "billions of years ago, when dinosaurs roamed the earth" and I've been taught the theory of evolution like everyone else. The problem, though, is that no matter how credible or not a particular area of science is, much of what is out there is taught based on authority. I find it very ironic to be flamed by anyone who thinks I'm an idiot for not believing in a theory that's never been proven by scientific process. It's recently become a "religious act" to question science in any capacity, but isn't questioning science the only way we can tell the good science from the bad science? And there is a lot of great science out there - even in public schools. But there's no longer a way for students to evaluate the credibility of what they're being taught. That seems to be degrading the quality of the subject. Science should be a quest for the truth, with no presuppositions, and appropriate understanding between hypotheses vs. theories vs. laws. When a theory is presented in the classroom as law and it's not held accountable to method, it's degenerated into mere conditioning.
I've spent a considerable amount of time studying topics such as the age of the earth and the theory of evolution, and I could probably argue it quite well if so inclined to engage in a discussion. That's important if you're going to believe anything really - including whatever the mainstreamed secular agenda happens to be.
Just as an example, I've recently looked into Carbon-14 dating and found that in cross-referencing it to Egyptian history (which dates back as far as 3500 B.C. and is held to be in very high regard by archaeologists and scientists alike), there is evidence that Carbon dating may be inaccurate beyond around 1800 B.C. For someone not to consider that would be ignoring science. My point here is that my beliefs aren't merely unfounded, eccentric ideas. Just because microevolution is feasable, that doesn't mean I'm going to sweep macroevolution under the rug and not test it - the two are actually worlds apart, just cleverly bundled. The Bible has given me a perspective that seems to offer a reasonable and sensible way to put the different pieces of good science together. No matter what you believe, I strongly feel that you should have some factual foundation to support whatever it is, and if you don't, then be man enough to admit you only have a theory put together.
No matter what side of the camp you are on, your beliefs require a certain amount of faith, as neither side is at present proven scientifically. I don't have all the answers, but I don't think science in its present state does either. At the end of the day, you can't prove the existence of God factually, and so whatever you believe is still based on faith. But at least the Christians can admit that - I just wish the evolutionists would too.
How do you pronounce your name?
Jonathan Responds:
Hi. Well first off, I'm sticking to the pronunciation 'Jarsky', however many of my relatives still pronounce it 'Zarsky' or "Za-Jarsky". As far as I can tell, my last name was originally 'Dziarstac' when the first generation of my family came over, which would have been pronounced with a 'J'. It's of polish decent, but I'm afraid I'm not very in tune with my ancestors on this side of the family. The other side of my family is mostly Italian, and they drink a lot, organize crime, and generally have more fun - so they are much more interesting to hang out with. For the past 29 years of my life, giving my last name to anyone has included the obligatory explanation of its pronunciation, history, and snickering at puns they think I'm hearing for the first time (-1, Redundant), so don't feel too bad for asking.
As far as who I am and why you should care - I guess that depends on what kind of geek you are. I've never appeared in a Star Trek series or anything (I've been too busy coding and being a real geek), so I guess that eliminates me as a candidate for public worship in some circles. I guess if you're into coding, open source, hacking all kinds of Verizon gear, or eradicating spam, then some of my recent projects may be of interest. If you at least hate me by the end of the interview, I'll have accomplished something.
An Anonymous Coward asks:
What do you think about the proposed change to the GPL with the upcoming GPL 3? Is it a welcomed breath of fresh air to the Open Source Community, or will it just be a reiteration of the previous GPL? What are your thoughts and comments on the GPL 3?
Jonathan Responds:
Based on the scattered information I've read about some potentially targeted areas in GPLv3 and the religious fervor with which some of these discussions have been reported, all I can say is I hope common sense prevails. Actually there's much more I can, and will, say about the subject below, but I think it's probably a good idea to summarize in advance as you may not make it through the list of details in one sitting. So in summary of all my points to come: I hope common sense prevails.
One of the things I've heard, which doesn't make much sense to me, is the idea of changing the GPL to deal with 'use' rather than 'distribution', which would affect companies like Google and Amazon. The argument seems to be that some people feel building your infrastructure on open source should demand a company release all of their proprietary source code which links to or builds on existing GPL projects. They argue that the open source community hasn't benefited from companies like Google and Amazon. Well, from a source code perspective that might be somewhat true - but if you take into consideration the fact that we all have a good quality, freely accessible search engine, cheap books, and employment for many local developers (many of whom write open source applications), the benefits seem to balance out the deficiency. Does anybody remember what the world was like before Google? None of us do, primarily because we couldn't find it - we couldn't find much of anything we were looking for on the Internet as a matter of fact, including other people's open source projects. You might not be getting "free as in beer" or "free as in freedom", but you are getting "free as in searches" and "free as in heavily discounted but not quite free books" in exchange. That's a pretty good trade. It's certainly better than having to look at pages of advertising before completing your order, or subscribing to a Google search membership. On top of this, you probably wouldn't want to see half of the source code that's out there being integrated (internally) into these projects. While I haven't seen Google or Amazon's mods specifically, I do heavily suspect that, if they are like any other large corporate environment, there are many disgusting and miserable hacks that should under all circumstances remain hidden from sight forever - many of which are probably helping ensure job security for the developers that performed the ugly hacks in the first place. Just how useful would they be to your project anyway? Probably little. And if you really believe in free software ("free as in freedom"), then the idea that someone should be required to contribute back to your project in order to use it is contradictory to that belief - you might just as well be developing under an EULA instead of the GPL.
With that said, there's a difference between freedom and stealing. I've heard that GPLv3 will attempt to address the mixing of GPL and non-GPL software. I think this clarification might be a good thing. For one, because I've seen far too many pseudo-open source tin cans and CDs being resold commercially out there, distributing many different F/OSS tools with painfully obvious closed commercial code, and finding ways to easily loophole around this part of the GPL, and secondly because it's based around implementation guidelines that really aren't any of the GPL's business. At the moment, mixing uses a very archaic guideline, which is - in its simplest terms -based on whether or not your code shares the same execution space as the GPL code. I think this needs to be reworked to give authors the flexibility to define "public" and "private" interfaces in a project manifest. We're already defining these anyway if we believe in secure coding practices. Closed source projects may then use whatever public interfaces the author has declared public (such as command line execution, protocols, etcetera) but private interfaces are off limits. One particular area where this would come in handy is in GPL kernel drivers, which need this ability to avoid tainted-kernel situations. If the author wants, they can declare dynamic linking to a library as a public interface and even make their code more widely useful without having to switch to the GPL's red-headed stepchild, the LGPL. It would also be nice to be able to restrict proprietary protocols (such as one between a client piece and a server piece, which may have originally been designed to function together) to only other GPL projects, which would essentially create GPL-bonded protocol interfaces. This won't restrict use in any way - only what closed-source projects are limited to interfacing with when redistributed.
I would also like to see the GPL's integration clause tightened down quite a bit. There are some companies out there abusing the GPL with "dual licensing". I've considered dual licensing myself in some commercial products, and I just don't believe it's being done in the right spirit much, if at all. Doing away with the possibility of integrating the GPL into a dual license could help strengthen the GPL.
Finally, I'd say mentioning a few words in the GPLv3 about submission practices to help stave off problems like this whole Sco and Linux® fiasco from ever happening again would be a good thing. People generally don't want to limit usage, but if you're going to submit code, there should be at least some submission guidelines. I suspect much of this can (and should) be done outside of the GPL, but at least covering the basics might be appropriate. It should be understood that if you're going to contribute code to the GPL, it had better be unencumbered. It's definitely something every project should already be considering already.
An Anonymous Coward asks:
Do you have any suggestions for the enthusiastic yet inexperienced? Perhaps a listing of projects in need of developers, with some indication of the level of experience suggested (as well as languages required).
Jonathan Responds:
The best projects I've seen were those started from someone with a passion for what it is they're coding. Open source development is the internship of the 21st century, and working on projects is tedious, frustrating, and likely to make you want burn out if you haven't developed perseverance. I usually suggest to people to come up with ideas for some projects they feel passionately about and make those their first couple of goals. Even if it's completely useless to anyone else, you're still likely to benefit from it yourself. Just look at my Australian C programming macros. Who would have thought that people wouldn't want to use "int gday(int argc, char *argv[])" in their code. I'm sure I learned something from that project, though I still can't remember what.
Instead of spending idle time looking for other projects to jump on, I'd spend as much time as I could in man pages, books, and coding up my own little concoctions. Even if they're stupid ones, you're likely to learn something, or even better - come up with another neat little idea you can spin off of it. Necessity is the mother of invention, so I try and figure out what it is I need, and then do it myself. That usually works. If you still can't think of anything, see if you can catch a vision for something someone else needs. I wouldn't touch anything that you're not 100% bought into and excited about for your first projects.
RealisticCanadian asks:
I myself have had numerous interactions with less-than-technically-savvy management-types. Any time I bring up solutions that are quite obviously a better technical and financial choice over software-giant-type solutions; conversation seems to hit a brick wall. The ignorance of these people on such topics is astounding, and I find many approaches I have tried seem to yield no results in the short term. "Well, yes, your example proves that we would save $500,000 per year using that Open Source solution. But We've decided to go the Microsoft (or what-have-you) route." With your track record, I can only assume you have found some ways to overcome this closed-mindedness.
Jonathan Responds:
I'm not so sure that I have convinced anyone open source was better inasmuch as I've convinced people that other people's projects were better than what Microsoft had to offer, and that's not hard for anyone to accomplish. I can strongly justify some open source projects to people because they are already superior to their commercial counterparts, but there are also a lot of crummy projects out there that should be shot and put out of my misery. I'm not one to advocate a terribly written project, even if it is open source. The good projects can usually speak for themselves with only a little bit of yelling and biting from me. So if you want to become a respected open source advocate at your place of business, I'd say the first rule of thumb is not to try and advocate crap projects for the mere reason that they're open source. Advocating the good ones will help you build a reputation. It also helps if you read a lot of Dilbert so you'll understand the intellectual challenges you'll be facing.
Some other things that I've found can help include what managers love to call a "decision matrix" which is a spreadsheet designed to make difficult decisions for them. For your benefit, this should consist of a series of features and considerations that the competitor doesn't have, with a big stream of checkboxes down the row corresponding to your favorite open source project. Nobody's interested in knowing what the projects have in common anyway, so tell them (with visual cues) what features your open source solution has over the competitor. And if you really want to get your point across clearly to your manager, do the spreadsheet in OpenOffice so they'll have to download and install an open source project to read it.
Once you've done that, and if you're still employed by now, the next thing to put together is an ROI (return on investment) comparison, which not only addresses the costs of the different solutions, but costs to support both solutions in the long run, cost of inaccuracy (if this is a spam solution for example), cost of training, customizations, and resources to manage each product. This is a great opportunity to size machines and manpower and include that in a budget forecast. Many managers are sensitive to knowing just how much extra dough it's going to cost to implement the commercial solution. At the very least, you ought to be able to prove many commercial solutions don't actually make the company much money in the long run. If speaking of cash isn't enough to convince your manager then a full analysis of low-level technical aspects will be necessary. This is simply a dreadful process, and where most open source attempts fail - because a lot of people are just too lazy to learn about the technical details of both projects and complete their due diligence. If you take the time, though, you're likely to either convince your boss or utterly confuse him - either one is very satisfying.
The biggest challenge in justifying many open source projects I've run into is finding solid support channels that your boss can rely on if you get hit by a bus (or in his mind, fired). Support is, in many cases, a requirement but not all good open source projects see the benefit in offering support. A lot of companies are willing to pay just to have someone they can call when they have a problem. So if you can find a project that's got a pool of support you can draw out of, you can not only use that to justify the project to your manager, but kick a few bucks back into the open source community. I started offering support contracts for dspam primarily because people needed them in order to get the filter approved as a solution. I think I do a good job supporting my clients that do need help, but at least half of them just pay for a contract and never use it. I certainly don't have a problem with that, and it supports the project as well as the people investing time in it.
Goo.cc asks a two parter:
1. In your new book, you basically state that Bogofilter is not a bayesian filter, which was news to some of the Bogofilter people I have spoken to. Can you explain why you feel that Bogofilter is not a bayesian filter?
Jonathan Responds:
Bogofilter uses an alternative algorithm known as Fisher-Robinson's Chi-Square. Gary Robinson (Transpose) basically built upon Fisher's Inverse Chi-Square algorithm for spam filtering, which provided some competition for the previously widely accepted Bayesian approach to this. Therefore, Bogofilter is not technically a Bayesian filter. The term, "Bayesian", however is commonly a buzzword known to most people to describe statistical content filtering in general (even if it isn't Bayesian), and so Bogofilter often gets thrown into the same bucket. CRM114 is another good example of this - many people throw it in the same bucket as a Bayesian filter, but it is configured (by default, at least) to be a Markovian-based filter which is "almost entirely nothing like Bayesian filtering". Technically, CRM114 isn't a filter at all, but a filtering-language JIT compiler (it can be any filter). I cover all of these mathematical approaches in Ending Spam, so grab a copy if you're interested in learning about their specific differences.
2. Bayesian filters have been around for some time now but there still seems to be no standardized testing methods for determining how well filters work in comparison to one another. Do you think that comparative testing would be useful and if so, how should it be performed?
Jonathan Responds:
Part of the reason there's no standardized testing methodology is because there's no standardized filter interface. A few individuals have attempted to build spam "jigs" for testing filters, but the bigger problem is really lack of an interface. About a year ago, the ASRG was reportedly working on developing such a standard - but as things usually turn out, it's an extremely long and painful process to get anything done when you've got a committee building it (take the mule, for instance, which was a horse built by a committee). This is probably why filter authors have also been hesitant to try and accommodate their filters to a particular testing jig. Incidentally, this is how I surmise that SPF could not have possibly made it through the ASRG - the fact that it made it out at all suggests that it never went in.
I think it's of some interest to compare the different filters out there, but it's also somewhat of a pointless process too. Since these systems learn, and learn based on the environment around them, only a simulation and not a test, will really identify the true accuracy of these filters - and even if you can build a rock solid simulation, it will only tell you how well each filter compared for the test subject's email. If we are to have a bake-off of sorts, it definitely ought to include ten or more different corpora from different individuals, from different walks of life. Even the best test out there can't predict how a filter might react to your specific mail, and for all we know the test subjects may have been secretly into ASCII donkey porn (which will, in fact, complicate your filtering).
This is why some people misunderstand my explanations of dspam's accuracy. All I've said in the past is "this is the accuracy I get", and "this is the accuracy this dude got". Which is the equivalent of "our lab mice ate this and grew breasts". There's no guarantee anybody else is going to get those results, though I'm sure many would try (with the mice, that is). In general, though, I try to publish what I think are good "average" levels for users on my own system, and they are usually around 99.5% - 99.8%. In other words: your mileage may vary. So try it anyway. Incidentally, I've been working with Gordon Cormack to try and figure out what the heck went wrong with his first set of dspam tests. So far, we've made progress and ran a successful test with an overall accuracy of 99.23% (not bad for a simulation).
What would be far more interesting to me would be a well-put together bakeoff between commercial solutions and open source solutions. The open source community around spam filtering really has got the upper hand in this area of technology, and I'm quite confident F/OSS solutions can blow away most commercial solutions in terms of accuracy (and efficacy).
Mxmasster asks:
Most antispam software seems to be fairly reactionary - wither it is based on keyword patters, urls, sender, ip, or the checksum of the message a certain amount of spam has to first be sent and identified before additional messages will be tagged and blocked. Spf, domainkeys, etc... requires a certain percentage of the Internet to adopt before they will be truely effective. What do you see on the horizon as the next big technique to battle spam? How will this affect legitimate users on the Internet?
Jonathan Responds:
That's the problem with most spam solutions, and why I wrote Ending Spam. Bayesian content filtering, commonly thrown into this mix, has the unique ability to grow out of your typical reactive state and become a proactive tool in fighting spam. I get about one spam per month now at the most, and DSPAM is learning many new variants of spam as it catches them; I'd call that pretty proactive. Spam, phishing, viruses, and even intrusion detection are all areas that can benefit greatly from this approach to machine learning. They will likely never become perfect, but these filters have the ability to not only adapt to new kinds of spam, but to also learn them proactively before it makes it into your inbox. Some of this is done through what is called "unsupervised learning" and not traditional training, while other tools, such as message inoculation and honey pots, can help automate the sharing of new spam and virus strains before anyone has to worry about seeing them. We haven't thoroughly explored statistical analysis enough yet for there to be a "next big technique" beyond this. The next big techniques seem to be trying to change email permanently, and I don't quite feel excited about that. Statistical tools are where I think the technology is at and it needs to become commonplace and easier to setup and run.
The problem seems to be in the myth that statistical filtering is ineffective or incomplete. Many commercial solutions pass themselves off as statistical(ish) and it seem to be contributing to this myth by failing to do justice to the levels of accuracy many of the true (and open source) statistical filters are reflecting. Any commercial solution that claims to be an adaptive, content-based solution (like Bayesian filters are) really ought to deliver better than 95% or 99% accuracy. Part of the problem is just bad marketing - most of these tools are not true "Bayesian" devices; they just threw a Bayesian filter in there somewhere so they could use the buzzword. Another problem is design philosophy and the idea that you need an arsenal of other, less accurate tests, to be bolted in front of the statistical piece. If you're going to train a Bayesian filter with something other than a human being, whatever it is that's training it ought to be at least as smart as a human being. Blacklist-trained Bayesian filters are being fed with about 60% accurate data, (whereas a human is about 99.8% accurate). So it's no surprise to me that Blacklist-trained filters are severely crippled - what a dumb combination. If you really want to combine a bunch of tools for identifying spam, build a reputation system instead. They do a very good job of cutting spam off at the border, are generally more scalable than content-based filtering, and most large networks can justify their accuracy by their precision.
Not all commercial content-based filters are junk. Death2Spam is one exception to this, and delivers around 99.9% accuracy, which is in the right neighborhood for a statistical filter. Not all reputation systems are junk either. CipherTrust's TrustedSource is one example of what I call a well-thought out system. If you must have a commercial solution, either of these I suspect will make you quite happy. As for (most of) the rest, quit screwing around and build something original that actually works.
Jnaujok asks:
The SMTP standard that we use for mail transfer was developed in the late 70's - early 80's and has, for the most part, never been updated. In that time period, the idea of hordes of spam flowing through the net wasn't even considered. It has always been the most obvious solution to me that what we really need is SMTP 2.0. Isn't it about time we updated the SMTP standard?
Jonathan Responds:
You're talking about an authenticated approach to email, and there have been many different standards proposed to do this. First let me say that, even though SMTP was drafted a few decades ago, it's still successful in performing its function, which is a public message delivery system - key word being public. There exist many private message delivery systems already, which you could opt to use, including bonded sender and even rolling your own using PGP signatures and mailbox rules. I have reservations about forcing such a solution on everybody and breaking down anonymity for the sake of preventing junk mail. Until you can sell a company like Microsoft on absolute anonymity in bonded sender and sell ISPs into putting up initial bonds for their customers (so that a ten-year old gradeschool student can still use email), I see a very large threat (especially by the government) in globalizing this as a replacement for the 'public' system. With services like gmail, where you can store an entire life's worth of email, the idea that everything you've ever said could be sufficiently traced back to you and used against you, I would rather deal with the spam. Why? Let me pull out my tinfoil hat...
It's been advertised plenty of times on Slashdot that Google stores everything about all of its queries. It wouldn't surprise me if they already have government contracts in place to perform data mining on specific individuals. How would you like, in the future, all of your email to be mined and correlated with other personal data to determine whether or not you should be allowed to fly? Buy a firearm? Rent a car? We're not very far off from that, and even less so once this correlation is made possible.
So abstract some level of anonymity at the ISP-level you say? That's just not going to happen. For one, that makes it just as simple for a spammer to abuse somebody's network and then we've gone and redesigned SMTP for no good reason. Remember, business has to be able to set up shop online fairly easily and spammers are a type of shop. So we are always going to balance between free enterprise and letting spammers roam on the network. Should we employ a CA, how much would it cost to run your own email server? More importantly - does this perhaps open the door for per-email taxes? I'd much rather just deal with spam the way we are now. For another thing, abstracted identity architectures would only give you a level of anonymity parallel to the level of anonymity you have when you purchase a firearm (where the forms are stored by your dealer, rather than filed to a central government agency). See how long it takes for the feds to trace your handgun back to you if you leave it at the scene of a crime.
You can't leave it in the ISP's control anyway. The sad truth is that most ISPs still don't care about managing outgoing spam on their network; so new spammers are being given a nurturing environment to break into this new and exciting business. I had a recent bout with XO Communications about one such new spammer who had run a full-blown business on their network since 1997 and recently decided he'd like to start spamming under the "CAN-SPAM" act (which he was convinced defended his right to spam). He included his phone number, address, and web address in the spam - I called him up and verified he was who he said he was (the owner of this business, and spamming). Provided all of this information (over a phone call) to the XO abuse rep (let's call him "Ted"), even filed a police report, and XO still to this day has done nothing. His site is even still there, selling the same crap he spams for. This happens every day at ISPs out there.
The consequences outweigh the benefits. The people who drafted the SMTP protocol probably thought of most of these issues too. A public system can't exist without the freedom to remain anonymous, ambiguous, and the right to change your virtual identity whenever the heck you like.
Sheetrock asks a two parter:
1. In the past, I've heard it suggested that anti-spam techniques often go too far, culling good e-mail with the bad and perhaps even curtailing 1st Amendment rights. Clearly this depends on what end of the spectrum you're on, but recent developments have given me pause for thought on the matter. For example, certain spam blacklists would censor more than was strictly necessary (a subjective opinion, I realize) to block a spammer -- sometimes blocking a whole Class C to get one individual. This would cause other innocent users in that net space to have their e-mail to hosts using the blacklists silently dropped without any option of fixing the problem besides switching ISPs.
Jonathan Responds:
A lot of blacklists have started taking on a vigilante agenda, or at the very least rather questionable ethical practices. Spamhaus' recent blacklisting of all Yahoo! Store URLs (and Paul Graham's website) is a prime example of this. As long as you're subscribed to human-operated blacklists, you're going to suffer from someone's politics. That's one of the reasons I coded up the RABL, which is a machine-automated blacklist. There is also another called the WPBL (weighted private block list). As the politics of the organizations running human-maintained lists get worse, I think more of these automated lists will start to pop up. Machine-automated blacklists don't have an agenda - they have a sensitivity threshold. It's much easier to find the right list with the right threshold than it is to find the right politics (and then keep tabs on them to make sure they don't change). The RABL, for example, measures network spread rather than number of complaints. If a spammer has affected more than X networks, they are automatically added to the system, and removed after being clear for six hours (no messy cleanup). Another nice thing about machine-automated blacklists is that they are really real-time blacklists, and capable of catching zombies and other such evils with great precision.
NOTE: I haven't had time yet to bring the RABL into full production, but am interested in finding more participants to bring us out of testing.
2. This is an extreme example, but most anti-spam approaches have the following characteristics: They are implemented on a mail server without fully informing the users of the ramifications (or really informing them at all). They block messages without notification to the sender, causing things to be silently dropped. Even if the recipient becomes aware of the problem, few or no options are given for the recipient to alter this "service".
Jonathan Responds:
I've run into issues like this with my ISP (Alltel), and I agree with a lot of what you're saying. In the case of Alltel, not only are they filtering inbound messages using blacklisting techniques and other approaches they don't care to tell me about, but they are filtering outbound messages as well. I had to eventually give up using their mail server because I could not adequately train my own spam filter (Alltel would block messages I forwarded to it). To make matters worse, there is no way to opt out of this type of filtering on their network, even though I offered to give them the IP address of my remote mail server. This clearly does affect their customers, and I feel there are censorship, violation of privacy and denial of service issues all going on here. (Somebody please sue them by the way).
Fortunately, I don't think this issue is as wide spread as you might think. Many of the ISPs and Colleges I've worked with are, unlike Alltel, very dedicated to ensuring that their tools only provide a way for their users to censor themselves. I think this ought to be a requirement for any publicly used system. Specifically...
1.The user must be able to opt in or out of all aspects of filtering
2.All filtering components and their general function must be fully disclosed
3.The user must be able to review and recover messages the system filtered
Opting out of RBLs is as easy as having two separate mail servers and homing on the box you want. I would strongly advise to ensure that your solution is capable of receiving instruction from a user to improve its results, but it is still very difficult to scale this to millions users. At the very least should be fully disclosed, recoverable, and removable.
An Anonymous Coward asks:
Without going into the truths of the beliefs in question, which I'm sure will be debated enough in the Slashdot thread anyway (and I hope you'll join in), what do you think the reason is that so many scientists, nerds and people otherwise rather similar to you think your beliefs are obviously incorrect? Do you think they are all deluded? Do you agree that there might be a possibility that your beliefs are not rational?
Jonathan Responds:
The beliefs I hold as a Christian aren't always the popular ones, but they're certainly valid arguments for anyone who cares to ask about them (not that that has happened). When you read about someone's beliefs, you have the option to engage in discussion, or to filter his or her beliefs through your own belief system. The former option involves cognitive thought, however the latter is how most people today respond to anything that even smells religious. And I say this coming from the position of someone who hasn't tried to shove my beliefs down anyone's throat - I merely documented them on my personal website. That tells me that some people don't believe I have the right to my own beliefs - how asinine is that?
But to address the question, my beliefs aren't based on some religious intellectual suicide. In fact, the Bible teaches that you should know what you believe and why, and that you should even be prepared to give a defense for your faith - so the Bible encourages sound thinking and not some pontificated ideal structure as many quickly dismiss it as. I didn't dumb down when I became a Christian. In fact, it felt more like I began to think more clearly. I was raised in the same public school system as everyone else and didn't even know who Jesus Christ was until around my junior or senior year of high school. I've read from my early days in Kindergarten how "billions of years ago, when dinosaurs roamed the earth" and I've been taught the theory of evolution like everyone else. The problem, though, is that no matter how credible or not a particular area of science is, much of what is out there is taught based on authority. I find it very ironic to be flamed by anyone who thinks I'm an idiot for not believing in a theory that's never been proven by scientific process. It's recently become a "religious act" to question science in any capacity, but isn't questioning science the only way we can tell the good science from the bad science? And there is a lot of great science out there - even in public schools. But there's no longer a way for students to evaluate the credibility of what they're being taught. That seems to be degrading the quality of the subject. Science should be a quest for the truth, with no presuppositions, and appropriate understanding between hypotheses vs. theories vs. laws. When a theory is presented in the classroom as law and it's not held accountable to method, it's degenerated into mere conditioning.
I've spent a considerable amount of time studying topics such as the age of the earth and the theory of evolution, and I could probably argue it quite well if so inclined to engage in a discussion. That's important if you're going to believe anything really - including whatever the mainstreamed secular agenda happens to be.
Just as an example, I've recently looked into Carbon-14 dating and found that in cross-referencing it to Egyptian history (which dates back as far as 3500 B.C. and is held to be in very high regard by archaeologists and scientists alike), there is evidence that Carbon dating may be inaccurate beyond around 1800 B.C. For someone not to consider that would be ignoring science. My point here is that my beliefs aren't merely unfounded, eccentric ideas. Just because microevolution is feasable, that doesn't mean I'm going to sweep macroevolution under the rug and not test it - the two are actually worlds apart, just cleverly bundled. The Bible has given me a perspective that seems to offer a reasonable and sensible way to put the different pieces of good science together. No matter what you believe, I strongly feel that you should have some factual foundation to support whatever it is, and if you don't, then be man enough to admit you only have a theory put together.
No matter what side of the camp you are on, your beliefs require a certain amount of faith, as neither side is at present proven scientifically. I don't have all the answers, but I don't think science in its present state does either. At the end of the day, you can't prove the existence of God factually, and so whatever you believe is still based on faith. But at least the Christians can admit that - I just wish the evolutionists would too.
First Question: How do you pronounce your name?
Linux Video Tutorial Project, Tutoring the masses.
Very, VERY interesting. I have to say thank you to him, for the fact that he made a good statement about faith. Very brave, and very good man.
Yay, I have a sig.
Does anybody remember what the world was like before Google? None of us do, primarily because we couldn't find iy
YESI do remember you noob.
Google is nothing new, before them there were a few engines that did the job fine. There was even an web based FTP search engine Where is that google, where is that.
If i write some code and I licence it under the GPL and something else what is the problem?
You can take the GPL code and do what you like with it under the GPL, but I choose to licence what i have written under BSD (say) as well then what is the problem? It is going way OTT to take that away from me if I am gifting my work back to the community with the GPL. This is why I always stipulate that my code is licenced under GPL v2 and not any subsequent version - no self-appointed guardian has the right to take away my freedom to dual licence code.
At least his first name isn't Jathan. He's be stuck explaining his name for another five minutes to people as well.
Me: My name is Jathan.
Response: Woah. Were your parents stoned when they named you?
Me: haha, yeah that's funny. It's kind of like saying Jason with a lisp.
Response: Thats great caus I half a lithsp
Me: Oh, sorry, it's like Nathan with a J then.
I feel your pain Mr.Zdziarski....
// no
Typically what is done is to select a range of filters/learners that you want to evaluate. A test dataset is also selected (in this case, it would be an archive of spam and nonspam messages, correctly classified). An M-way N-fold cross validation is performed. What this means is that the data set is split into N parts, and N runs are conducted for each classifier, training using N-1 of the parts. The remaining part is used to test the learner. This is repeated, each time holding out a different part of the test set.
This ENTIRE procedure is repeated M times. This gives, ultimately, M*N results. Each column pair of results from a specific pair of learners has a T-test applied to it. This tells the statistical significance of variations in performance. Usually, a 5% or 1% threshold of significance is used.
Once that is completed, something called a WLT table is computed. Each time a learner defeats another learner on a given test, its W ("Win") counter is incremented. Likewise, when a learner loses, the L ("Loss") counter is incremented. When two learners tie (i.e., when the variation is not statistically significant), the T ("Tie") counter is incremented.
The overall "winner" of the comparison is the learner with the maximum value of W-L.
This sounds complicated and bloated, but it is, in fact, how machine learners are tested in academia. The cross validation method, along with checks for statistical significance, is critical to achieving a valid comparison. Simply running the tests once and saying "This filter got 98% correct, and this other filter got 95% correct -- therefore the first filter is better" is NOT sufficient.
Jesus Christ how old are you?
He was about 33 when he died.
Web Design Tips
I use SpamAssassin (server) and SpamPal (client). They're both quite accurate and I'm very happy with them.
However, I've had unacceptably high false-positive rates. Saying that you only get one spam a day is fine--I can deal with that. Are you sure that no legitimate e-mail is being tagged though? I have the subject lines prefixed with [SPAM] and so I just go through and look for anything that looks like it might not be spam. This process takes about 10 minutes a day, which is 10 minutes more than I would care to spend.
I give the anti-spam developers credit for their hard work, but I believe that the best solution would not be filter-based, for mere fact that if 1 spam gets through a day, and the volume of spam increases 100x in the next 2 years, then you're back up to ~100 spams a day. It's a temporary solution to a permanent problem.
Just my $0.02.
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird. --Nietzsche
Well, it wasn't worthless to me, and it looks like the editors thought that my question was a bit more coherent than a improperly written href.
If there's something that you really want to know, have you considered contacting him using this newfangled technoloigy called email?
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
Bingo. One of my managers said it very well at my former employment: nobody ever got fired for choosing IBM- and he's 100% right. I know a few companies spending millions to have services offered by Dell, IBM, Microsoft, etc who could get their services for thousands from clone computer makers, and Linux- but who would they?
Choose IBM and loose a few million, and you 'missed the market'. Choose open source and loose a few million and 'your solution wasn't up to par'. Choose open source and succeed and you make millions...
Is it worth the risk for the second situation? Most managers who want to leave with a hefty bonus and a good referral woulds say no.
PS: Agree 100% with almost everything he said. Smart man.
-M
when you see the word 'Linux', drink!
What advice do you have as a developer of this program to: * Help my users send legitimate messages (either by education (specifically) or by programming techniques) * Help Spam Filtering Software check the messages my program sends out for possible abuse * Be a part of the solution to sending legitimate messages to many people, rather than perhaps be part of the problem.
I had written up an answer to this one, but it turned out not to appear in the interview questions, so it got bitcanned. I believe the summary was to incorporate functionality such as listid, to help a filter identify that the message was coming from a mailing list. This way, whether it's spam or ham, the filter can better identify it.
Perhaps other areas that might make it very sysadmin-friendly in terms of management would be putting in filter interfaces to allow one to check outgoing messages to the list, and potentially flag suspect messages for additional approval.
As far as making people send good messages, there's not much hope for you. People are going to abuse your software no matter how hard you try and educate them. That's part of free software - giving spammers the freedom to use it too. Hopefully incorporating as many list identification features by default will at least help the dimmer of the bunch make their spam more available.
My spam problem is the reverse of most people's. Using grey listing I get basically no actual spam. It's wonderful - works very well. But that's not my problem. My problem is that some a$$hole spammer has decided to start using my domain as a from address in his spams. So I'm currently getting deluged by bounce messages for mails I never sent. I've published SPF records and that's helped a bit, but not a lot.
Anyone got any good suggestions?
---- Den ene knappen er powerknapp, den andre er Bender voice knapp "Bite My Shiny Metal Ass"
Sure looks like he didn't take his good time on researching carbon-14 and find out that to date dino-bones we aren't using carbon-14 that much.
http://science.howstuffworks.com/carbon-142.htm
Hard-core Christians complain that we aren't researching their opinions, but I see way too much that it is the same the other way around. If you believe in Carbon-14 then you have to agree that the other science behind the chemistry also works. And in that case that argument for the age of dinosaurs so fall apart for those Christians.
(Disclaimer: I believe in God, Jesus, and Science - The Bible has good things but it is too man made and narrow minded towards the real world, IMHO)
Not trying to judge anyone,
But you're *supposed* to judge people. Jesus wants you not to consort with disbelievers. How would you know they're disbelievers unless you judge them as such?
The basic problem is that you're trying to impose your own definition of Christian on everyone. I'm a true Christian and I don't believe the same things as you, and you can't make me stop calling myself a Christian.
The real metric of whether someone is a Christian or not is their actions, not their claims. Ye shall know them by their fruits.
Don't piss off The Angry Economist
i always find it laughable when 'intelligent' people counter the 'theory of evolution' by rolilng over with a statement like:
"I find it very ironic to be flamed by anyone who thinks I'm an idiot for not believing in a theory that's never been proven by scientific process."
since 'the theory of evolution' falls under the Scientific definition of theory...
a plausible or scientifically acceptable general principle or body of principles offered to explain phenomena
here is a good one Jonathan, explain the 'flounder' w/o using evolution as a base...
PEOPLE!! STOP. Stop using "theory" in regards to science as a theory in the common definition. A scientific theory is very well backed up by facts.
"In layman's terms, if something is said to be "just a theory," it usually means that it is a mere guess, or is unproved. It might even lack credibility. But in scientific terms, a theory implies that something has been proven and is generally accepted as being true."
I will personally stangle the next person who does this. Pisses me off.
Scientific Theory = Scientific Fact. Bitches.
http://wilstar.com/theories.htm
I am truely sick of people who call themselves Christians but are really practising some whacky supersitious religion that has no place for critical thought.
To quote Mr Zdziarski's homepage:
"to teach and to defend what I have come to find is a scientifically beautiful piece of logical harmony - the Bible"
Ah so science is a book that is thousands of years old and most of it is not corroborated in secondary sources? A book that is known to have been selectively edited through out its history for political reasons? So Jesus violating the laws of physics in his numerous miracles is science?
It certainly has great bits of logic and moral teaching in it (Do unto others, as you would have them do unto you), but it is not science. For someone to call it science shows that they have no understanding of science at all and it is no surprise that he thinks creationism and evolution should be taught in science class. I was taught creationism at school but in thelogy class.
I spent my entire education in christian schools. I have spent the last three weeks going to church to reconnect with God. Science does not preclude God. Just because God didn't have to make Adam from mud, after he made the world in six days, doesn't mean there is no God. No matter who much scientific knowledge we get there will always be room for God (What came before the Big Bang? And how did matter get the properties it has?).
For me God is the ultimate programmer. No sense doing all the work by hand when you can write some perl scripts to do it for you.
Science tells us what we can do and how. Religion tells us if we should.
========
CINC, 4th Penguin Legion
I second that. What you believe is one thing, but if you abandon scepticism and value religious doctrine higher than tested scientific theory, then you've got a problem.
Incidentally, I'm sad to see that Zdziarski tries to pull the same old stunt again that most supporters of creationism try to pull - namely, deliberately misunderstanding the meaning of "theory" in the context of science and equating it with an unproven hypothesis. Everyone who knows a bit about science (which no doubt includes Zdziarski) will know that that's not true, of course, but the general public often doesn't, which is why this kind of tactic is so despicable.
I'd really like to see a supporter of creationism who says "I don't believe in evolution, but I still acknowledge that it explains the observed facts and has made falsifiable predictions that were, in turn, shown to be correct". But I guess that's something you just won't hear from someone who puts his personal faith above the scientific method, as far as the search for scientific truth is concerned.
quidquid latine dictum sit altum videtur.
Breakfast served all day!
"One of the weaker points of religions, is that they base all their facts in one initial fact: God exists, and so, from that all the other knowledge is generated."
:)
Just to be clear, science is based on at least 2 equally unproven hypotheses (possibly others):
1) For every physical-observable effect there is a physical-observable cause. (physical-observable was the short-hand my Physics profs used for anything that can be measured or detected directly or indirectly)
2) Physical Law (that is the way the universe behaves) is consistent everywhere in the universe.
I am not claiming either of these is false. But from a purely philosophical point of view, the idea of "proving" things is a relative notion. In science, we can hypothesize, test, and improve our theories. But there is always room for the universe to prove us wrong.
Faith and Science start from different sets of assumptions. My own opinion is that:
a) People of faith have nothing to fear from Science, since God surely understands the true nature of the universe, which we understand in progressive approximation through science.
b) Scientists are foolish to become defensive about science in the face of Faith, as this often leads to pompously accepting as immutable that which is a progressive approximation of the true nature of the universe.
Of course, the Jonathan Zdziarski's responses re: F/OSS and SPAM were also quite thought provoking
==>Andrew Bauserman!
"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled."
- Richard P. Feynman
(Appendix to the Rogers Commission Report on the Challenger Space Shuttle Accident)
"A philosopher once said, 'It is necessary for the very existence of science that the same conditions always produce the same results'. Well, they do not... Yet science goes on in spite of it...
"What is necessary 'for the very existence of science', and what the characteristics of nature are, are not to be determined by pompous preconditions, they are determined always by the material with which we work, by nature herself.
"In fact, it is necessary for the very existence of science that minds exist which do not allow that nature must satisfy some preconceived conditions, like those of our philosopher."
- Richard Feynman, _The_Character_of_Physical_Law_
First, a note: I am not religious. Just wanted to clarify that so you know my position isn't biased in this rebuttal.
"Religious persons acts differently, they beleive in a certain religion because their parents did."
While this is indeed true for the vast majority of people all through-out history, it isn't _always_ true. Jonathan's responses even explicitly state that he hadn't heard of Jesus until much later in his life.
"Also, religion doesn't evolve, science does. The catholic church once stated"
Religion has evolved *tons*. Christianity today looks very little like what it was 2000 years ago, and even the Catholic church is different. The simple fact that we *have* multiple versions of Christianity is only a couple hundred years old. The Church (which was never called "Catholic" until after the Reformation) has had many changes in its beliefs. The belief that the Son is of the Father (ie, that Jesus is divine, more than a man) didn't come about until several hundred years after the Church had formed. That's but one of many, many examples.
"the same than it was 1.7k years ago, science changes daily,"
What's wrong with not changing? If it changes, that would mean that it was wrong. Likewise, when science changes, that means that it was, previously, wrong. 2000 year old scientific facts are impressive. This week's hot scientific theory is really no different than the nut-jobs jumping on the latest fad religion. (Kids with black lipstick calling themselves Wiccans: I'm refering to you.)
If you clarify "change" to mean "refine," then we once again come back to the fact that Christianity has refined itself greatly over the last two millenia, not to mention the refinements that Judaism likely underwent long before it branched into Christianity. Any major religion undergoes refinements in its beliefs and tenants, and simply studying history and works of antiquity will make that evolution (forgive the pun) of the religions clear.
"religion doesn't change, and doesn't add new knowledge."
Religion isn't about knowledge. If all you are interested in is being smart, then sure, a religion like Christianity isn't for you. That doesn't mean Religion is wrong. Science has not disproven God. It has not disproven Creation. It has not disproven anything in the Christian religion, and likewise many other religions have elements which have not been proven nor disproven by science.
I would say that the real purpose of religion is to make up for what knowledge can't give you.
It is key to keep knowledge in mind, though. For a personal pet peeve, I don't care what the book says, the Bible *is not* the word of God. That's something you can prove, I might note - just pick up two editions of the Bible and note the differences. There are in fact many factual discrepencies. Perhaps, long long long ago, some particular stories were the word of God, but they have been retold, transcribed, edited, translated, and artistically recast so many times that whatever copy of the Bible you have access to now cannot be wholly and firmly trusted to be true. In many cases, slight changes of wording can make a phrase still say the same thing, yet mean something new. There are known and documented errors in most popular versions of the Bible, and many more documented passages of the modern Bible eidtions which are believed to be flat out incorrect. Many of today's versions of various Bible passages which may have once been analogies could now be considered literal truth due to poor translation or simply a loss of contextual understanding since the passages' original inception. I.e., is the "seven days" of the creation a measure of actual time as we perceive it, or an unclear translation of a literary device used to explain the passing of seven units of time of unknown size? You can't be sure, and there are no original manuscripts of the Bible to study; indeed, there never were original manuscripts of most of it, as much of the Bi
The beliefs I hold as a Christian aren't always the popular ones, but they're certainly valid arguments for anyone who cares to ask about them (not that that has happened).
Yes, but as anybody with a clue would point out, a perfectly valid argument can still be completely wrong. The problem scientific-types have with Christianity isn't that it's not a valid argument - it's that the axioms are wrong.
Point out a scientist who claims that, assuming the Bible is the incorruptible word of God, Christianity is not a valid argument. I don't think that's a common attitude. What I do think is a common attitude is disagreeing that we should use that as an axiom in the first place.
When you read about someone's beliefs, you have the option to engage in discussion, or to filter his or her beliefs through your own belief system. The former option involves cognitive thought, however the latter is how most people today respond to anything that even smells religious. And I say this coming from the position of someone who hasn't tried to shove my beliefs down anyone's throat - I merely documented them on my personal website. That tells me that some people don't believe I have the right to my own beliefs - how asinine is that?
That's completely asinine. It's also a straw man argument. So people filter what you say through their own beliefs system before responding - how does that in any way whatsoever tell you that they don't think you have a right to your own beliefs?
the Bible encourages sound thinking
Correct me if I'm wrong (really). The Bible claims that it is the incorruptible word of God, and that you should believe this because the Bible says so. The Bible also says that if you don't believe, you go to hell.
I consider these two things to be antithetical to sound thinking. If, however, I am wrong, and the Bible doesn't claim these things, then you have to explain how vast numbers of Christians say that it does. The only explanation I can see is that they have misinterpreted it. In which case, you are left with the unenviable position of claiming that you are interpreting it right and they aren't - so much for the incorruptible word of God.
The problem, though, is that no matter how credible or not a particular area of science is, much of what is out there is taught based on authority.
Everything that is taught is done so based on authority. But the practice of science is based upon the rejection of authority. You can't practice science if you blindly accept things as the truth. That's not how science works.
It's recently become a "religious act" to question science in any capacity
Bullshit. Scientists question science all the time. That's their job. You can't do science without it.
Just because microevolution is feasable, that doesn't mean I'm going to sweep macroevolution under the rug and not test it - the two are actually worlds apart, just cleverly bundled.
Speciation has been observed a number of times. If you aren't referring to speciation, then I suggest you use proper terminology instead of the terms cooked up by creationists.
No matter what side of the camp you are on, your beliefs require a certain amount of faith
I agree. I have faith in the relative fidelity of my memory. I have faith that my senses are not being tampered with. I have faith that the laws of nature don't change behind my back every few minutes. I have faith that the rest of human society isn't engaged in a giant conspiracy to deceive me. I consider that to be the minimum you must have faith in to make any sense of the world. If I suspect that any particular piece of science is wrong, those faiths above are all I need to check one way or the other.
You, on the other hand, have faith that a magical
deliberately misunderstanding the meaning of "theory" in the context of science and equating it with an unproven hypothesis.
Oh I don't misunderstand the difference. However, most public schools do, and they teach a theory as if it were a law (such as the laws of thermodynamics or the laws of gravity). I think a lot of people misinterpret trying to bring the theory of evolution down to a "theory" as trying to convert it into a hypothetis. This just isn't the desired intent.
don't believe in evolution, but I still acknowledge that it explains the observed facts and has made falsifiable predictions that were, in turn, shown to be correct
How's this:
I don't believe in evolution, but I still acknowledge that it makes an attempt to explain the observed facts and has made predictions that were, in turn, shown to be correct by those who believe the theory of evolution is in fact falsible.
Unfortunately, most scientists today view evolution as infalsible. The excuse is to simply "keep digging". I believe there is more than enough information to suggest that we move on and find some other scientific explanations.
I find it very ironic to be flamed by anyone who thinks I'm an idiot for not believing in a theory that's never been proven by scientific process.
In actuality, I believe you're an idiot because you don't understand that the theory of evolution does not attempt to be a proof. That's just more propaganda from the radical right.
Science attempts to explain why things are and, by extrapolation and interpolation, why they might change. Evolution does this very well. It does not require you to "believe". It simply states what is. You are still free to believe in God in any form you care to, just don't expect to be able to predict what will come next with any accuracy.
It's simple: I demand prosecution for torture.
Untrue. See, for example, the writings of the Ante-Nicene Fathers.
You're right that certain beliefs and lines of thinking have changed and grown over time -- especially in Roman Catholicism with regard to Augustine's Aristotelianism, but the implication that the roots of Christianity weren't present in written form by at least the end of the first century doesn't fit the historical record.
(I am an armchair historian.)
how to invest, a novice's guide
But you're *supposed* to judge people. Jesus wants you not to consort with disbelievers.
C'mon, you call yourself a Christian with this belief? Firstly, "Do not judge, or you too will be judged" (Matthew 7:1). That is from the mouth of Jesus himself.
Second, Jesus does not want you to consort with disbelievers?! Look at his whole life. He spent most of His time with "sinners and tax gatherers." Jesus wants us to be like Himself, so how can we do that and judge people? There are two possibilities, one is that Jesus contradicts Himself here, or you don't know what you're talking about.
Seeing as I have provided some context and you have not, I will leave this exercise up to the reader.
Lynam and I said that DSPAM 2.8.3, in its default configuration, achieved 98.15% accuracy on the same corpus to which Zdziarski refers above. The report also argued that accuracy was a very poor measure of filter performance and that a false positive rate such as the 1.28% demonstrated by DSPAM would likely be unacceptable to an email user.
In recent correspondence, Zdziarski suggested three configurations of DSPAM (available here on the web) that achieved the following results:
dspam(tum) fpr 1.81% fnr 0.80% accuracy 99.20%
dspam(toe) fpr 1.94% fnr 0.59% accuracy 99.16%
dspam(teft) frp 1.85% fnr 0.53% accuracy 99.32%
More detailed results and comparisons will be made available when our current study is complete. Don't take my word (or Jonathan's) for anything; run this filter and others on your own email. But please take great care in constructing your gold standard.
Gordon Cormack
You can say, "here is a theory." Yes, but do you believe the theory is correct?
Another interesting point is that ID is falsifiable. Successful experiments could falsify the claims of irreducible complexity. Evolution seems unfalsifiable, on the other hand.
Celebrate the finer things in life
Spoken like a "True Believer". It seems to me that many (most?) True Believers can't understand that science isn't about believing in anything, and that science never, ever claims to prove anything. Shame, too, on the non-Believers who say that it does.Again, a clear misunderstanding of science. Science is all about questioning everything. I.e. why are we here? How did we get here? Where are we going? What is thunder? It's only a "religious act" when the questioner is an avowed (or covert) Believer who is offering zero actual evidence supporting any alternate theory.
Religion, generally, and Chistianity specifically, appears to be all about answers. I.e., to answer the questions above, because god made us. God put us here. To heaven if you're good, hell if your not. God is angry. See the difference?
Does this mean I think less of his thoughts on email? No, I don't dismiss someone because of their beliefs (think Stallman
Jesus Christ how old are you? 5?
You can't accurately measure how old he is. You see, while his birth certificate says he is 5, nobody here actually witnessed his birth to testify to the fact that the birth certificate is accurate. We take birth certificates on higher authority, and don't subject them to real scientific standards. Why, I read an article recently that indicated that clocks weren't accurate back before 2002!
In all seriousness now, lets just clear up his misconceptions really quickly here:
The beliefs I hold as a Christian aren't always the popular ones, but they're certainly valid arguments for anyone who cares to ask about them
Many creationists, strangely, think that "Christian" viewpoints equals "Creationist" viewpoints. Heck, even the Vatican accepts evolution.
That tells me that some people don't believe I have the right to my own beliefs - how asinine is that?
If someone told you the world was flat, you'd question their better judgement, now wouldn't you? Nobody is telling you what you can or cannot believe, and don't pretend that they are. But you can't tell people that you believe in something contrary to all available evidence and expect them not to care.
It doesn't get into your knowledge of spam filtering, by the way, so don't take this as an insult.
In fact, the Bible teaches that you should know what you believe and why, and that you should even be prepared to give a defense for your faith
What are you referring to? Paul regularly launches diatribes against the learned, tells followers not to listen them, and to believe in faith alone. For example, read Corinthians sometimes. It's one long diatribe against the educated after another.
Take the old testament, too. It, too, says not to even listen to unbelievers, lest they make you doubt your faith. So, I'd like to see where you draw these "intellectual debate-encouraging" lines from.
I've read from my early days in Kindergarten how "billions of years ago, when dinosaurs roamed the earth"
Then you had the worst Kindergarten class on Earth. "Dinosaurs" roughly range from the Triassic to the Cretaceous, i.e. 248 to 65 mya. "Billions of years ago" was Precambrian, where single cellular and simple multicellular organisms roamed the earth.
The problem, though, is that no matter how credible or not a particular area of science is, much of what is out there is taught based on authority.
It doesn't matter what it's "taught" by. What matters is how the evidence was derived. By the way, did you not find the irony of the fact that your belief is based on an "ultimate authority source"?
I find it very ironic to be flamed by anyone who thinks I'm an idiot for not believing in a theory that's never been proven by scientific process.
Quite to the contrary. First, a bit of background for you. When the study of geology began in earnest, almost all mainstream scientists were biblical literalists. The "flood theory" of strata layers was essentially unquestioned; finding fossils in layers only furthered their belief in the flood theory. That was, until they hit a brick wall: Sorting.
The fossils that they kept finding were extremely confined. You never find trilobytes in the same layer as clams, or grasses in the same layer as Alethopteris, etc: *ever*, the world over, despite how incredibly common and distributed these species were.
The scientists who discovered this were at a loss to explain it. Gradually, a new theory rose to explain it. Evolution, right?
Nope! The "multiple creation, multiple flood" theory. They were so convinced that the bible had to be literally accurate, that they simply adapted it. This theory was dominant in European scientific thought for several hundred years. However, as time progressed, it kept falling apart. They kept having to add in more creations and more floods; by the time of Darwin, there were widely viewed beliefs of hal
Rock Us, Dukakis.
Just don't pass it while they are 3. If you teach your children to THINK, and teach him your specific beleives when he is old enough to decide for himself, it's ok. If you tell your child about your political/religious ideas when s/he's 15, you are leting him choose, but if you put that shit in his head since the day s/he was born ...
Also, the way religion is teached is not actual teaching, but brainwashing. When you learn algebra, you are told how it works, and that's it. When you are teached religion, they tell you how it works, and then they tell you that if you don't beleive in it and follow it exactly, you will go to hell, to be punished forever, when this is told to a 5 years old child, what do you think will happen?
WTF am I doing replying to an AC at 5 A.M on a Friday night?
Ye shall know them by their fruits.
And from their ever so fashionable use of Early Modern English.
The nit-pickey aspects of things like how cosmology, evolution and relativity (or, for that matter, the macro aspects of weather systems) were beyond the understanding of people of that period (and even beyond the understanding of most of the people today). The only people for whom things like quantum theory regularly make a practical difference are the people who do things like design (as opposed to use) cpu circuits.
Does the question of how ducks came to be affect the question of whether I should respect this jewel of an earth that mankind was given as a gift. Does it affect the question of whether I should kill my neighbour for the gas his truck? no. For the most part, evolution vs creationism is just an irrelevant academic exercise and a distraction from more real issues like love, respect, trust, honouring the humanity and beauty around us, and being true to our responsibilities.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Oh I don't misunderstand the difference. However, most public schools do, and they teach a theory as if it were a law (such as the laws of thermodynamics or the laws of gravity). I think a lot of people misinterpret trying to bring the theory of evolution down to a "theory" as trying to convert it into a hypothetis. This just isn't the desired intent.
Which laws of gravity are you talking about? the newtonian laws always were just a theory too, which has since been disproven and replaced by einsteins theories. The "laws" are still used because they seem to be a good aproximation.
There really are no absolute laws in nature that we can discover.
While you can observe a falling stone again and again, you can never proove that it will fall down the next time and not float straight up.
The "real laws" of gravity could for example be that a stone will only fall down 99,99...%, and only occams razor can give you some hint that this is not the case. But you can never be sure.
Now, thermodynamics is a little bit different. given a few simple axioms about the statistic behaviour of a system you can actually prove a few theorems, which boil down to the laws of thermodynamics. Of course, it is not possible to proove that the axioms really apply to our world, so we cannot proove the laws of thermodynamics either. And of course the statistical nature of the laws of thermodynamics means that they can be violated on a small scale anyway.
First off, Behe is a Christian:
"Scott refers to me as an intelligent design "creationist," even though I clearly write in my book Darwin's Black Box (which Scott cites) that I am not a creationist and have no reason to doubt common descent. In fact, my own views fit quite comfortably with the 40% of scientists that Scott acknowledges think "evolution occurred, but was guided by God." Where I and others run afoul of Scott and the National Center for Science Education (NCSE) is simply in arguing that intelligent design in biology is not invisible, it is empirically detectable."
Secondly, if you apply Behe's arguments to Rainbow Bridge, it'd declare that it never could have formed naturally. It did. It also would declare that bubbles are impossible in nature (remove any part of the bubble, it pops), fire can't be started naturally (remove any of the ingredients, and it goes out), ice dams can't form (remove any part of the ice dam, and the water rushes out behind it destroying the dam), and geysers aren't natural (alter any part of the geyser's path, and it stops erupting).
"Irreducible complexity" is a one-way process. Structures come into being for reasons other than their initial functionality (and for the above cases that I mentioned, for no reason at all related to the end aesthetics or functionality).
For example, a commonly cited case is the bombardier beetle. They produce hydroquinones and H2O2 which collect in a resevoir. The resevoir opens into a tough, thick-walled reaction chamber that produces catalyases and peroxidases; this breaks down the H2O2 and produces heat and helps break down the hydroquinone to p-quinones. A fifth of the mixture is vaporized, and propels the burning mixture through a series of valves and nozzles that spray it onto the target.
Irreducibly complex to say the least, right? Gish sure thought so. Yet, the intermediary stages already exist for most stages, and the others are obvious progressions. From here:
1. Quinones are produced by epidermal cells for tanning the cuticle. This exists commonly in arthropods. [Dettner, 1987]
2. Some of the quinones don't get used up, but sit on the epidermis, making the arthropod distasteful. (Quinones are used as defensive secretions in a variety of modern arthropods, from beetles to millipedes. [Eisner, 1970])
3. Small invaginations develop in the epidermis between sclerites (plates of cuticle). By wiggling, the insect can squeeze more quinones onto its surface when they're needed.
4. The invaginations deepen. Muscles are moved around slightly, allowing them to help expel the quinones from some of them. (Many ants have glands similar to this near the end of their abdomen. [Holldobler & Wilson, 1990, pp. 233-237])
5. A couple invaginations (now reservoirs) become so deep that the others are inconsequential by comparison. Those gradually revert to the original epidermis.
6. In various insects, different defensive chemicals besides quinones appear. (See Eisner, 1970, for a review.) This helps those insects defend against predators which have evolved resistance to quinones. One of the new defensive chemicals is hydroquinone.
7. Cells that secrete the hydroquinones develop in multiple layers over part of the reservoir, allowing more hydroquinones to be produced. Channels between cells allow hydroquinones from all layers to reach the reservior.
8. The channels become a duct, specialized for transporting the chemicals. The secretory cells withdraw from the reservoir surface, ultimately becoming a separate organ.
This stage -- secretory glands connected by ducts to reservoirs -- exists in many beetles. The particular configuration of glands and reservoirs that bombardier beetles have is common to the other beetles in t
Rock Us, Dukakis.