My own spam filters are in fact written in it, and I've been hacking them daily lately. Arc is fairly usable. It's painful when I have to switch to Common Lisp for something, which is a good sign.
The problem is that our current implementation is appallingly slow. We're working on a faster interpreter; then we'll have another round of language redesign; then we may release something.
IE doesn't do real spam filtering yet, but MSN 8 now does content-based filtering that learns by example. Since they brag that it uses a "patented" algorithm, I assume they're using this Bayesian filtering algorithm.
Before everyone starts worrying that MSFT has patented Bayesian filtering, (a) I don't think the patent would hold up well in court, because e.g. ifile is older and (b) patents are not a problem for open-source projects anyway.
This is very well put. Do you have this letter posted anywhere on the Web where I could link to it? It would serve as a good model for anyone who wants to write to politicians.
That is an interesting idea. In fact, it might be useful not just to study spam but to improve filtering. If you looked at diffs of word frequencies over time, you could use this to bump up the probabilities for words whose use seemed to be accelerating.
You say no spammer will reply to mail asking for authentication, but if your solution were adopted as part of a widely-used piece of software, instead of something you cooked up for yourself, spammers would would be quick to automate answering such requests.
Probably quicker than some human users would be, which brings us to the second problem: getting authentication requests is so annoying that some senders won't bother. In that case your filtering solution is effectively generating false positives, which is the big no-no.
In this case, one thing that makes it hard for the spammers is that nonspam words will vary from one individual user to another. To take advantage of that you do have to filter individually for each user, though.
It might be possible to make a general purpose spam filter that you could just plug into your network like a router, but I am less optimistic about the chances of that working long term.
Though I did write this, I didn't post it here, and would not have. I wrote this to figure out for myself why Java seemed suspicious. Posting it on a forum like slashdot amounts to a troll.
I think this sort of article is barely worth writing (if something does suck, time will take care of it), and that's why on the site I put it near the bottom.
You are right. The original title of this article was "Inside Orbitz", but within slashdot they edit the title and description you submit, and in the process all the references to Orbitz got changed to Sabre. They are different programs. Sorry if this has confused anyone.
I was at this workshop and it it really felt like something unusual was happening. I think everyone who was there will remember it.
Most conferences are deadly boring, because people go to them to present things they've already thought of. At this one you had a feeling that people were having new ideas right there.
There was also a strong feeling that the topic was an important one, where new things are happening every day. That, I think, is because new programming languages matter now more than they used to. In the 1970s and 80s, there was a vogue for inventing new languages, usually to prove a point about types, and they all just disappeared into a black hole, because no one used them for anything.
If you make a new language now, people can actually use it. In 1980, if you wanted to use a new language, you had to convince your sysadmin to install it. Now everyone (at least everyone who reads slashdot) is sysadmin of their own server. That changes everything: if you build it, they will come. I think that will make the next few years a period of rapid innovation in programming languages.
(At the very least, we'll probably see more Lisp features in Perl and Python.;-)
You make a good point when you ask "Whose Lisp?" Lisp is a family of languages. Common Lisp, which is arguably the leading dialect now, has major flaws. It was designed by a committee, and not especially liked even by them. Common Lisp is like a big ugly old wrestler: powerful but lumpy and ill-mannered (i.e. impolite to the OS and other applications).
Disputes about Lisp are often about two different things. Those who attack Lisp are usually attacking Common Lisp. Those who defend Lisp are usually defending Lisp, the family.
Lisp the family sounds like a vague concept, but there is a solid core there. You could approximate it as either Common Lisp minus the crap, or as Scheme (another family member) plus more data structures and libraries.
Lisp would certainly look better if it had a better representative to send to the Language Beauty Contest than the lumpy old wrestler. It is about time someone made a nice new dialect. In the meantime I'll still take the wrestler over the alternatives, but the price I have to pay is using a language that is considered unfashionable.
It's interesting to hear that lots of people describe Python as Lisp without parentheses. That's exactly what I thought when I, as a long-time Lisp hacker, first looked at Python. Python even duplicates accidental historical features of Lisp that many Lisp hackers now consider a bad idea. They did not get there by accident.
The problem is, what "Lisp without parentheses" means is a dumbed-down Lisp. There is a reason for those parentheses. Lisp without parentheses means Lisp without the core idea of Lisp: that the programs are made out of Lisp data structures. Lisp without parentheses means Lisp without macros, and that means a Lisp that is no longer (as John Foderaro famously called it) the "programmable programming language."
It could have been done in Perl, in the worst
(and fairly common) case by creating another
instance of Greenspun's Tenth Law.
That's literally what would have been required
if someone used Perl 4, which was what Perl was
when we started in 1995. The Viaweb editor
depended very heavily on lexical closures,
which at that time only Lisp and Smalltalk had.
And yet, you know, I kind of think you'd be
saying the same thing if Perl was Perl 4.
For more details of why we needed Lisp
see these excerpts
from a talk I gave recently.
A couple people have mentioned that I didn't
explain in sufficient detail what we did
with Lisp. Sorry about that; I had been
planning (and still plan) to write a second,
more technical, article. In the meantime, there
are more technical details in these
excerpts
from a talk I gave recently at BBN.
-- pg
for fast, efficient development of an application
like the Yahoo store builder, you sure as hell don't need a
language which permits the definition of new syntactic forms.
Since I wrote it, I can tell you that you're
mistaken. Any big Lisp program will
tend to use a fair number of macros, because
the way to write a big Lisp program is to
start by writing a language for programs
like yours, and then to write yours on top
of it. Viaweb was, effectively, written in
a language for building online apps. The
core operator was not merely a macro, but
a macro-defining macro.
This bottom-up style of programming is not
limited to Lisp, of course, but Lisp is very
well suited to it and most good Lisp hackers
work this way.
There were also specific individual tasks I
could not have done without macros. For
example, the software does not use cgi
scripts, but instead has a hash-table of
closures stored in the server, which are
generated on the fly when links are. (If
you try out the editor you'll notice the
urls are just hash keys.) These closures
often needed to refer to variables from
the surrounding context. I could not
have done that without macros.
It's true, incidentally, that everyone
thinks their favorite language is great.
That doesn't mean they're all wrong.
In many cities, fans think their team
will win the world series this year.
In one city they are right.
Slashdot Mirror
My own spam filters are in fact written in it, and I've been hacking them daily lately. Arc is fairly usable. It's painful when I have to switch to Common Lisp for something, which is a good sign.
The problem is that our current implementation is appallingly slow. We're working on a faster interpreter; then we'll have another round of language redesign; then we may release something.
There's no rush. We'd rather get things right.
IE doesn't do real spam filtering yet, but MSN 8 now does content-based filtering that learns by example. Since they brag that it uses a "patented" algorithm, I assume they're using this Bayesian filtering algorithm.
Before everyone starts worrying that MSFT has patented Bayesian filtering, (a) I don't think the patent would hold up well in court, because e.g. ifile is older and (b) patents are not a problem for open-source projects anyway.
This is very well put. Do you have this letter posted anywhere on the Web where I could link to it? It would serve as a good model for anyone who wants to write to politicians.
That is an interesting idea. In fact, it might be useful not just to study spam but to improve filtering. If you looked at diffs of word frequencies over time, you could use this to bump up the probabilities for words whose use seemed to be accelerating.
There are two problems with this approach.
You say no spammer will reply to mail asking for authentication, but if your solution were adopted as part of a widely-used piece of software, instead of something you cooked up for yourself, spammers would would be quick to automate answering such requests.
Probably quicker than some human users would be, which brings us to the second problem: getting authentication requests is so annoying that some senders won't bother. In that case your filtering solution is effectively generating false positives, which is the big no-no.
In this case, one thing that makes it hard for the spammers is that nonspam words will vary from one individual user to another. To take advantage of that you do have to filter individually for each user, though.
It might be possible to make a general purpose spam filter that you could just plug into your network like a router, but I am less optimistic about the chances of that working long term.
Though I did write this, I didn't post it here, and would not have. I wrote this to figure out for myself why Java seemed suspicious. Posting it on a forum like slashdot amounts to a troll.
I think this sort of article is barely worth writing (if something does suck, time will take care of it), and that's why on the site I put it near the bottom.
You are right. The original title of this article was "Inside Orbitz", but within slashdot they edit the title and description you submit, and in the process all the references to Orbitz got changed to Sabre. They are different programs. Sorry if this has confused anyone.
--pg
I was at this workshop and it it really felt like something unusual was happening. I think everyone who was there will remember it.
;-)
Most conferences are deadly boring, because people go to them to present things they've already thought of. At this one you had a feeling that people were having new ideas right there.
There was also a strong feeling that the topic was an important one, where new things are happening every day. That, I think, is because new programming languages matter now more than they used to. In the 1970s and 80s, there was a vogue for inventing new languages, usually to prove a point about types, and they all just disappeared into a black hole, because no one used them for anything.
If you make a new language now, people can actually use it. In 1980, if you wanted to use a new language, you had to convince your sysadmin to install it. Now everyone (at least everyone who reads slashdot) is sysadmin of their own server. That changes everything: if you build it, they will come. I think that will make the next few years a period of rapid innovation in programming languages.
(At the very least, we'll probably see more Lisp features in Perl and Python.
You make a good point when you ask "Whose Lisp?" Lisp is a family of languages. Common Lisp, which is arguably the leading dialect now, has major flaws. It was designed by a committee, and not especially liked even by them. Common Lisp is like a big ugly old wrestler: powerful but lumpy and ill-mannered (i.e. impolite to the OS and other applications).
Disputes about Lisp are often about two different things. Those who attack Lisp are usually attacking Common Lisp. Those who defend Lisp are usually defending Lisp, the family.
Lisp the family sounds like a vague concept, but there is a solid core there. You could approximate it as either Common Lisp minus the crap, or as Scheme (another family member) plus more data structures and libraries.
Lisp would certainly look better if it had a better representative to send to the Language Beauty Contest than the lumpy old wrestler. It is about time someone made a nice new dialect. In the meantime I'll still take the wrestler over the alternatives, but the price I have to pay is using a language that is considered unfashionable.
It's interesting to hear that lots of people describe Python as Lisp without parentheses. That's exactly what I thought when I, as a long-time Lisp hacker, first looked at Python. Python even duplicates accidental historical features of Lisp that many Lisp hackers now consider a bad idea. They did not get there by accident.
The problem is, what "Lisp without parentheses" means is a dumbed-down Lisp. There is a reason for those parentheses. Lisp without parentheses means Lisp without the core idea of Lisp: that the programs are made out of Lisp data structures. Lisp without parentheses means Lisp without macros, and that means a Lisp that is no longer (as John Foderaro famously called it) the "programmable programming language."
That's literally what would have been required if someone used Perl 4, which was what Perl was when we started in 1995. The Viaweb editor depended very heavily on lexical closures, which at that time only Lisp and Smalltalk had.
And yet, you know, I kind of think you'd be saying the same thing if Perl was Perl 4.
For more details of why we needed Lisp see these excerpts from a talk I gave recently.
--pg
A couple people have mentioned that I didn't explain in sufficient detail what we did with Lisp. Sorry about that; I had been planning (and still plan) to write a second, more technical, article. In the meantime, there are more technical details in these excerpts from a talk I gave recently at BBN. -- pg
Since I wrote it, I can tell you that you're mistaken. Any big Lisp program will tend to use a fair number of macros, because the way to write a big Lisp program is to start by writing a language for programs like yours, and then to write yours on top of it. Viaweb was, effectively, written in a language for building online apps. The core operator was not merely a macro, but a macro-defining macro.
This bottom-up style of programming is not limited to Lisp, of course, but Lisp is very well suited to it and most good Lisp hackers work this way.
There were also specific individual tasks I could not have done without macros. For example, the software does not use cgi scripts, but instead has a hash-table of closures stored in the server, which are generated on the fly when links are. (If you try out the editor you'll notice the urls are just hash keys.) These closures often needed to refer to variables from the surrounding context. I could not have done that without macros.
It's true, incidentally, that everyone thinks their favorite language is great. That doesn't mean they're all wrong. In many cities, fans think their team will win the world series this year. In one city they are right.
-- Paul Graham