In response to the discussion of the paper "Lisp as an Alternative To
Java" by Erran Gat, 1999:
I have a lot of experience with both Common Lisp and Java. I like
both langauges. Although I spent many years of my life as an
enthusiastic Lisp booster, if I were to start a new project now, under
most circumstances that I can imagine, I would select Java. But the
reasons don't have that much to do with deep programming language
concepts. (More about my credentials later.)
In the following, when I say "Lisp" I mean "contemporary Common Lisp
including CLOS" if I don't say otherwise. I talk about Lisp as it is
now. Lisp's history is an interesting subject but quite irrelevant to
the questions brought up by the paper.
>> The Erran Gat paper of 1999
The experiment and results in this paper don't persuade me of
anything. The programmers were self-selected; the Java programmers
were apparently quite inexperienced; the sample size is just too
small; judging a whole langauge based on a single programming problem
is too narrow. A small programming exercise like this tests only a
small fraction of the interesting aspects of programming.
The paper is valuable in that it makes you think harder about the
"which langauge is faster" question. Remember, langauages don't have
speeds; langauge implementations have speeds. And what you have heard
may not be true. Be skeptical of "common wisdom" about what's slow
and what's fast; such "common wisdom" is often outdated, limited to
particular contexts, or just plain wrong.
>> Why isn't Lisp more popular?
A language's popularity is strongly influenced by "network effects":
that is, as more people use a language, it becomes more desirable to
use. It is hard for a new language to "break through", and once it
does there is a strong positive-feedback effect. Breaking through is
very difficult and depends a lot on timing, luck, and often on
forceful publicity and marketing.
Java managed to do it, due to a confluence of many factors. It was in
the right place at the right time. Java's early success had a lot to
do with the rise of the World Wide Web, the decision of Netscape to
incorporate Java into their browser, the politics of Microsoft's entry
into the Internet area, the use of Sun's marketing resources, Sun's
decision to give out the implementation for free, and many other
factors on top of the technical merits (and demerits) of the language
and its then-available implementation.
Lisp's attempts to break through didn't succeed because the right
confluence didn't happen, partly due to luck and partly due to
ineptness of all of us who were hoping to promote it. It did not help
that Lisp was marketed primarily on the coat-tails of the "AI
industry" of the 1980's, which did not succeed as an industry
(although many parts of the AI technology are alive, well, and making
money today). It was also hurt by claims that Lisp could only run
well on special-purpose hardware, by its unusual syntax that puts off
so many people initially, by the lack of good free implementations (at
the time); I could go on and on.
It has helped Java's cause that there is one organization promoting
and defining the langauge and establishing standard API's in so many
areas. Many Lisp enthusiasts put their energy into refining and
improving the language (resulting in excellent technology such as
Scheme and Dylan) rather than all concentrating on stabilizing and
developing one standard.
The extensive set of standard API's created by Sun and the Java
Community Process is very valuable. There just isn't any Lisp
equivalent standard API for JMS, JDBC, Enterprise JavaBeans, and so on.
The positive-feedback "network" effects is extremely valuable to
Java's cause. It's much easier to find trained Java programmers than
trained Lisp programmers. All kinds of tools and libraries are
available (many free), many more than for Lisp. There are lots of
books available about Java, so many that even the subset that are
*good* books is pretty large, and not just the core Java language but
facilities such as RMI, Enterprise JavaBeans, JDBC, JMS, and on and
on.
For example, there are many commercial producers of messaging
subsystems that implement the JMS specification, and competition
between them is driving higher functionality and lower prices.
There's nothing like that going on for messaging subsystems and Lisp.
This has nothing to do with any technical features of Java and Lisp as
languages.
>> Lisp as a General-Purpose Language -- Lisp is not "exotic"
"Lisp" means different things to different people. If you read "The
Structure and Interpretation of Programming Langauages", you're going
to see Scheme code that is fundamentally different from the way code
looks in most langauges. I think this is all extremely interesting
and valuable, but it's not what I have spent my time on.
To me, Lisp is general-purpose programming language in which I have
done all kinds of system programming. In my experience, Lisp is a lot
less "exotic" than some people might expect. For example, the whole
"Lisp is functional, not procedural" business is largely irrelevant.
The control structure and overall organization of a program in Common
Lisp is extremely similar to that of a Java or C++ program. All these
languages have subroutine calling, object-oriented programming,
iteration, recursion, data structures, structured programming,
exceptions, multithreading, etc.
Any claims that groups can't program in Lisp, or that Lisp programs
are inherently unmaintainable, are nonsense. There is nothing about
writing a program in Lisp that makes it harder to maintain than a
program in C++ or Java.
>> Static Typing
Static typing is one of the biggest differences. I agree that Common
Lisp basically does not have static typing (yes, I know about
"declare"). Some languages have static typing that is so restrictive
that it seriously gets in the way of getting useful work done, and in
the Old Days we of the Lisp world were very much reacting to
constricting type systems. I am pretty happy with the Java concept of
types, which I feel provides useful expression of intent in the code,
and useful compile-time error checking, and doesn't get in your way
very much. The parameterized typing coming to Java (in JDK 1.5, last
I heard) should improve the situation further. Still in all, it's not
that big of a deal, and if I went back to using Lisp without static
typing I don't thihk my life would be all that different.
>> Parentheses and Macros
Lisp's unusual syntax, with the parentheses and what used to be called
"Polish notation" (i.e. no infix operators), bothers some people. It
looks unusual and unwieldy to those who are unaccustomed to it.
However, once you get used to it, and if you use a decent programming
environment (particularly a text editor that can "indent for Lisp" and
otherwise knows Lisp syntax, such as but not necessarily Emacs), you
quickly get used to it and it seems perfectly normal. (It is also
possible to make a Lisp-family langauge with a more conventional
syntax, as the Dylan designers decided to do.)
Lisp's ayntax allow programmers to see programs in the form of a
simple data structure, which is the basis upon which is built the Lisp
"macro" facility, one of the truly different and powerful features of
Lisp. Lisp macros make Lisp an extensible language, in profound sense
of the word "extensible". They are a form of programming abstraction
that, properly used, can help make complex systems more simple and
understandable. Now that I used Java, I do miss Lisp macros. On the
other hand, it turns out it's not such a big deal, and if I had the
power of Lisp macros in Java it would not really change my life all
that much.
>> My Experience
Lisp: I wrote the first Emacs-written-in-Lisp (known variously as
EINE, ZWEI, and Zmacs). It was in fact the second Emacs ever,
developed concurrently with the original Emacs (written in TECO). I
have also written in Lisp an interactive debugger, a local area
network control program, a compiler, and an object-oriented database
management system. I worked with small and medium-sized groups. I
maintained lots of code written by people other than myself. I was
one of the five co-authors of "Common Lisp: The Language". I used
Lisp, in an implementation that eventually evolved into Common Lisp,
between 1976 and 1988.
Java: I was one of the reviewers of "The Java Language Specification".
I edited early drafts for Bill Joy and Guy Steele Jr. I co-wrote the
Java part of the ObjectStore database management system, and a
transactional data manager called "PSE Pro for Java", both from Object
Design. Currently I am developing business-to-business integration
software in Java at the same company, now known as eXcelon. I have
been using Java since early 1996. (In between I did C++.)
I'd like to go into the question of development environments but this
is long enough as it is. In a nutshell, I really wish I had my good
old Lisp Machine development environment again; what I'm using for
Java nowadays is stone knives and bearskins.
Regarding the "scathing criticism" of "neural
networks": I believe you're referring to the
book "Perceptrons" by Minsky and Papert.
Minsky has been saying for many years that
he regrets being responsible for the cutoff
of this line of research. What he and Papert
proved was true of "perceptrons", but what
we now call "neural nets" are more general
than "perceptrons" and the proofs in the
"Perceptrons" book don't apply to them.
In fact, for the last twenty years at least,
Marvin has been supportive of a very wide
range of approaches to AI research. He wants
to see people trying all kinds of methods and
philosophies. In fact he wants nothing more
than for a younger generation of AI researchers
to come in and vigorously try different
approaches. He's been talking that way ever
since the 1980's. So your portrayal of him is
really inaccurate.
And your implication that he is somehow holding
back the field of AI is not too plausible, either.
It's hardly as if he is controlling AI research
all over the world. He's not even controlling
AI research at MIT. If neural networks haven't
yet taken over the world, you can hardly lay the
blame at Marvin's doorstep.
This book is absolutely top-notch. I've been
working with SSL lately and have found the book
invaluable. The writing is crystal clear and
the author really knows what he's talking about.
It's worth reading even if you don't need to
know SSL, just for general education and
edification (although if you're not using SSL
you can skip over the sections that go into
detail). It doesn't get any better than this.
I think your attitude is very healthy, and I
think you're quite wise not to base any financial
plans on your options.
Nevertheless, that's not the same thing as the
answer to the question "How do you value your
options?". The question really means "How
much would I have to pay you for you to give
up your options?" If you really valued your
options at "nothing", you'd be willing to give
them up for $1, which you are presumably too
wise to do. On the other hand, if someone
offered you $1,000,000, you'd probably be
wise to accept. Somewhere in between is where
you value your options.
Luckily for you, nobody's offering, so you don't
have to decide.
Indeed, ObjectStore (and maybe other OODBMS's,
I genuinely don't know) is being used as a
middle-tier storage platform quite a lot these
days. This has turned out to be one of the
best fits for its technology. We have a
product called Javlin which provides various
enhancements for ObjectStore that are particularly
relevant to this kind of use.
I am one of the co-founders of Object Design Inc. (now known as
eXcelon Corp.) and one of the original architects of ObjectStore.
The term "object-oriented database" means many different things to
different people, so there is no one way to reply to the wide range of
things being said here about them. The best I can do is present my
own personal view, which should not be taken as an official
statement of my employer.
The most important thing I want to say is that I don't agree with
the original article's assertion that, in general, people who are
using relational database systems ought to be using object-oriented
database systems.
Way back in 1988 when the original engineering team first started to
design ObjectStore, we had no intentions (or illusions) of replacing
relational database systems. Rather, our purpose was to create a data
management system suitable for certain applications that, so far, had
not found existing data management systems to be appropriate. There
were many examples of these, but just to keep things simple I'll focus
on the one that was most representative: computed-aided design (CAD).
In 1988, no leading CAD system was using relational database systems
to model their most fundamental data, such as transistors, connections
between electronic parts, traces on PC boards, electronic schematic
diagrams, and so on. As far as I know, production CAD systems still
aren't using relational databases for this purpose.
Our goal was to design a data management system suitable for CAD (and
other applications) for which RDBMS's didn't seem to be doing the job.
Why? Because they had a very different set of requirements, and they
needed a different tool to do a different job. The only way to
understand what we were up to and where we were coming from is to
think carefully about our target
market, and how different it was from the traditional RDBMS target
market.
ObjectStore was originally designed with goals in mind such as:
(1) Be familiar to the programmers who develop things like CAD
systems. At the time, C-based object-oriented programming langauges
such as C++ and Objective C were starting to emerge, as the
programming world started to see the benefits of "object-oriented
programming", or, at least, the useful data modelling capabilities
provided by OOP. We bet that C++ would probably catch on in our
target markets, and I think we turned out to be right. So our goal
was to make a system that would be easy to use for programmers trained
in C++. We wanted to say that you could program very much as you are
accustomted to program, and with a little extra effort, it can all be
stored away automatically. (We didn't care about people trained in
SQL: they were not our target market.) Also, make it as easy as
possible to use your existing base of C++ code with the new data
management system (i.e. we cared a lot about pre-existing code; the
issue of pre-existing relationsl databases wasn't an issue for our
target market). Integration with the programming language was way up
high in our list of goals.
(2) Optimize performance for cases in which there was excellent
spatial locality of reference and excellent temporal locality of
reference. This is true of the typical CAD application. An example
of the kind of thing we wanted to benchmark well on was: "Redisplay
the screen, with lots and lots of transistors and resistors and
capacitors and wires connecting them all" (or whatever, depending on
the electronics technology in question). (We didn't care about TPC
benchmarks: that was not our target market.) Imagine what the code
looks like to redisplay the screen. Now imagine if every time you
wanted to know "what wires are connected to this transistor", you had
to put together an SQL string, the SQL string had to be sent over a
network to a database server, the server had to parse the SQL string,
the server had to run the query optimizer over the SQL string, the
server had to find the appropriate pages in its data cache, etc, etc,
etc. In ObjectStore the whole thing happens in a few CPU
instructions. This one example of where we get huge performance
gains.
(3) Provide real ACID transactions. (Yes, of course programmers had
to think about transactions, and where the transaction boundaries
should be, and what transactional consistency meant in their system:
that's necessary for dealing with concurrency and recovery
consideration. We didn't promise that you'd program exactly as you
did pre-database-system; but we made it as close as possible.)
(4) Provide simple associative queries, with automatic optimization,
subject to goal (1), i.e. make the query language natural for C++
programmers. It was not a goal to support some of the fancier queries
that SQL can do; that didn't seem to be too important to our target
market.
(5) Be able to provide fast navigation through database objects, just
as any modern programming langauge lets you navigate quickly through
programming language objects. Many of the most basic, inner-loop
computations that programers in our target market wanted to do would,
if expressed in SQL, use five-way or seven-way or ten-way joins, which
existing commercial RDBMS's handled very poorly. (Ten-way joins look
just find in Codd's mathematical structures, and you could conceive
of ways to implement an RDBMS that would do much better at them, but
it would be hard, and as far as I know it hasn't been done; Oracle's
traditional target market didn't much care about them.)
(6) Distribute the use of CPU resources. The CAD people were moving
in the direction of using what were then called "workstation"-class
computers, connected by local-area networks, in which the "client"
workstations had a great deal of processing power and memory (by the
standards of the time), and it made sense to take advantage of that.
We were not the only people thinking this way. There were many, many
papers published by university and corporate researchers, saying the
same things. Our principal competitors, as far as I've ever been able
to tell, were using the same general principals. The benchmarks that
our customers used to evaluate us were obviously designed around these
same assumptions.
Let me point out again that "object-oriented database system" means
many things to many people: the list of goals that I gave above is
certainly not what a lot of people mean by OODBMS!
Now, as it happens, most of the leading CAD vendors did not end up
buying ObjectStore and building CAD systems based on it. If you look,
you'll find that they are using pretty much the same technology now
that they were using twelve years ago. At the time, many of them were
talking about upgrading to use some kind of data mangement system,
very much like what we were talking about: we were in touch with the
CAD company engineers and spent a lot of time talking to them about
what they wanted. Unfortunately, for various business reasons, most
of them did not move ahead with major advances to their technical
underpinnings. (That said, there were some new CAD companies that did
created CAD systems based on ObjectStore, some of which are still
being sold today.)
In particular, the locality-related optimizations (point 2 above) were
fundamental in our overall design. In our classic use case, one user
would access a hunk of information, and pretty much all of it would
get cached on his powerful workstation, and would be operated on at
the powerful local workstation for a considerable period of time. So
it made a lot of sense to move a good deal of processing to the client
side, and try to minimize the burden on the (shared) server.
There are plenty of tradeoffs involved in these optimizations. For
applications without that kind of locality, they aren't optimizations
at all. Also, basic pure ObjectStore does a lot less runtime
error-checking than it might, because the CAD engineers specifically
did not want to pay for that kind of runtime error-checking, at least
from what we learned. (Note that there are other OODBMS's that make
very different choices here; much of what I'm saying, especially about
performance issues, should not be taken to apply to anything besides
ObjectStore.)
We built a different tool to solve a different problem. So all the
arguments of the form "relational database experts are easier to find
and hire than OO database experts" are true, and probably good answers
to the original article, but aren't relevant to our business, because
we were not trying to replace relational databases for what relational
databases are good at.
Similarly, if I may indulge in sarcasm for a moment, not every
collection of human beings on the planet Earth have (a) already
learned about expressing data in normal form (which is really quite
un-obvious until you study it) and already learned SQL, and (b) have
no idea how to program in any modern programming language. Some
people are skilled C++ programmers who have never used a relational
database system in their life.
I still think we did a pretty good job of what we originally set out
to do. As things developed, though, we ended up with a far more
diverse and heterogeneous set of customers than we had originally
expected: lots of telecommunications, lots of financial applications,
and all sorts of exotic stuff. Our customers have tried to use
ObjectStore as part of all kinds of software systems, many of them
really cool and innovative and not falling into any traditional
category at all. Some of them had requirements that were pretty much
like the ones above; some had very different requirements. We did a
good job for some of them and a not-so-good job for others. In my
personal opinion, we had trouble developing ObjectStore because our
requirement set became so diffuse; that is, everybody was asking for
something different.
There turned out to be some applications in a middle ground between
what we designed for, and what relational database have traditionally
been used for. So this complicated things a lot, and meant that
sometimes ObjectStore was being asked to do things too far away from
what its original design was intended for. One thing I'll 'fess up to
is that I think in the past our company sometimes tried to sell
ObjectStore for applications that were too far away from the original
design goals, and so sometimes it would turn out that ObjectStore just
wasn't the right tool for the job. I think we're a lot better about
that these days then we used to be. And we've found more areas in
which our kind of system turns out to be a good and appropriate tool.
I can't reply to everything everyone has said. And a lot of what has
been said, rightly, has been directed at the original article, whose
points are very different from my own.
But, here are some for which I think a reply is needed:
To "sailesh" the IBM DB/2 engineer:
- Your statement that you can't do a range query is completely wrong.
ObjectStore has supported B-tree indexes and range queries since its
first release.
- You say "Relational databases are great at recovery of your data
using log information". So is ObjectStore. It uses write-ahead
logging and recovers just fine, and has done so since its first
release.
- "Actually, academia has well and truly come to the conclusion that
RDBMS technology is the way to go." There isn't any one single way to
go, any more than there is one single best programming language. I am
still waiting to see an interactive CAD system that models its basic
items using relations. If IBM's CAD sytems are now using DB/2 at this
level, please let me know. As for academia, they have largely lost
interest in the whole issue and moved on to other things.
To "micromoog", who said "Given the dramatically higher costs
associated with designing and maintaining an OO system, most
applications just don't need the incremental performance gains
associated with it." I completely deny the claim that designing and
maintaining OO systems has a "dramatically higher cost". Taking a
bunch of guys who know COBOL and SQL, and try to dump C++ on them,
sure, that won't work very well. But the whole point of OO systems is
to make design and maintenance easier and cheaper, and I certainly
belive that they succeed that way. Why do you think nearly every
serious programming langauge successfully introduced in the last ten
years has included OO programming? And the performance gains for
using ObjectStore as opposed to an RDBMS, for those applications for
which ObjectStore is suitable, are not "incremental"; they can easily
be 100x or 1000x.
To the small number of people who claim to have used ObjectStore and
had all kinds of trouble with it: well, I'm really sorry that was your
experience. We have certainly had many customers who've been quite
successful with ObjectStore. We've had a lot of repeat business.
I've been involved with our support organization to know that the
great majority of user experiences were not nearly as bad as the ones
you report.
To the people who complain that OODBMS's give you a problem with
"vendor lock-in": well, there's a lot of truth in that. There aren't
good standards among commercial OODBMS's, largely due to fundamental
and irreconcilable philosophical differences, and different choices
made on certain major engineering tradeoffs. On the other hand, it's
pretty late in the day to claim that's it's easy to switch among
commercial relational database systems. They are each full of handy
features that no other vendor supports. They all have different SQL
dialects with different extensions to the query language. And, of
course, they all have entirely different data storage formats. How
many times have you heard of an IT organization that decided one day
to switch from Oracle to Sybase or from DM2 to MS SQL Server? Yes, it
happens sometimes, but it's hardly something one guy does in an hour.
To "Pedrito", who is in the business of selling applications designed
to work with the customer's existing database system, and for which
relational database systems are approriate: yes, certainly you should
be designing your applications to work with relational database
systems.
To the people who say that the problem with OODBMS's is that "there
isn't one single object model" or "objects lack the theoretical
underpinnings of Relational Algebra": gee, then why don't we use
relations as the fundamental data model of all our programming
languages? Really, the significance of the "mathematical
underpinnings" business often seems to take on spiritual or mystical
dimensions. From the point of view of our CAD engineer, there's a
tradeoff between (1) having a sophisticated ad-hoc query language that
lets you express complex queries that you never anticipated, and (2)
getting that screen to redisplay nice and fast. It would be great if
you could have both. I have yet to see how to do that. Maybe someone
else will figure it out. As an engineer, you learn about the concept
of "engineering tradeoffs". A design approach that's best at one
thing often isn't best at another thing.
As for me, I decided that nine years of working on ObjectStore was
enough; I liked doing it but one needs variety in life. I am now
working on our XML-based business-to-business integration server
product. It's based on our XML information management product, which
is based on ObjectStore internally (but invisibly to its users). So
I've been rather far from the actual ObjectStore business of the last
several years. So I don't feel comfortable trying to summarize "how
it all turned out". I know that we have lots of customers using
ObjectStore together with relational databases, letting each do what
it does best, and maybe that's one place that object-oriented
databases are headed.
Slashdot Mirror
In response to the discussion of the paper "Lisp as an Alternative To
Java" by Erran Gat, 1999:
I have a lot of experience with both Common Lisp and Java. I like
both langauges. Although I spent many years of my life as an
enthusiastic Lisp booster, if I were to start a new project now, under
most circumstances that I can imagine, I would select Java. But the
reasons don't have that much to do with deep programming language
concepts. (More about my credentials later.)
In the following, when I say "Lisp" I mean "contemporary Common Lisp
including CLOS" if I don't say otherwise. I talk about Lisp as it is
now. Lisp's history is an interesting subject but quite irrelevant to
the questions brought up by the paper.
>> The Erran Gat paper of 1999
The experiment and results in this paper don't persuade me of
anything. The programmers were self-selected; the Java programmers
were apparently quite inexperienced; the sample size is just too
small; judging a whole langauge based on a single programming problem
is too narrow. A small programming exercise like this tests only a
small fraction of the interesting aspects of programming.
The paper is valuable in that it makes you think harder about the
"which langauge is faster" question. Remember, langauages don't have
speeds; langauge implementations have speeds. And what you have heard
may not be true. Be skeptical of "common wisdom" about what's slow
and what's fast; such "common wisdom" is often outdated, limited to
particular contexts, or just plain wrong.
>> Why isn't Lisp more popular?
A language's popularity is strongly influenced by "network effects":
that is, as more people use a language, it becomes more desirable to
use. It is hard for a new language to "break through", and once it
does there is a strong positive-feedback effect. Breaking through is
very difficult and depends a lot on timing, luck, and often on
forceful publicity and marketing.
Java managed to do it, due to a confluence of many factors. It was in
the right place at the right time. Java's early success had a lot to
do with the rise of the World Wide Web, the decision of Netscape to
incorporate Java into their browser, the politics of Microsoft's entry
into the Internet area, the use of Sun's marketing resources, Sun's
decision to give out the implementation for free, and many other
factors on top of the technical merits (and demerits) of the language
and its then-available implementation.
Lisp's attempts to break through didn't succeed because the right
confluence didn't happen, partly due to luck and partly due to
ineptness of all of us who were hoping to promote it. It did not help
that Lisp was marketed primarily on the coat-tails of the "AI
industry" of the 1980's, which did not succeed as an industry
(although many parts of the AI technology are alive, well, and making
money today). It was also hurt by claims that Lisp could only run
well on special-purpose hardware, by its unusual syntax that puts off
so many people initially, by the lack of good free implementations (at
the time); I could go on and on.
It has helped Java's cause that there is one organization promoting
and defining the langauge and establishing standard API's in so many
areas. Many Lisp enthusiasts put their energy into refining and
improving the language (resulting in excellent technology such as
Scheme and Dylan) rather than all concentrating on stabilizing and
developing one standard.
The extensive set of standard API's created by Sun and the Java
Community Process is very valuable. There just isn't any Lisp
equivalent standard API for JMS, JDBC, Enterprise JavaBeans, and so on.
The positive-feedback "network" effects is extremely valuable to
Java's cause. It's much easier to find trained Java programmers than
trained Lisp programmers. All kinds of tools and libraries are
available (many free), many more than for Lisp. There are lots of
books available about Java, so many that even the subset that are
*good* books is pretty large, and not just the core Java language but
facilities such as RMI, Enterprise JavaBeans, JDBC, JMS, and on and
on.
For example, there are many commercial producers of messaging
subsystems that implement the JMS specification, and competition
between them is driving higher functionality and lower prices.
There's nothing like that going on for messaging subsystems and Lisp.
This has nothing to do with any technical features of Java and Lisp as
languages.
>> Lisp as a General-Purpose Language -- Lisp is not "exotic"
"Lisp" means different things to different people. If you read "The
Structure and Interpretation of Programming Langauages", you're going
to see Scheme code that is fundamentally different from the way code
looks in most langauges. I think this is all extremely interesting
and valuable, but it's not what I have spent my time on.
To me, Lisp is general-purpose programming language in which I have
done all kinds of system programming. In my experience, Lisp is a lot
less "exotic" than some people might expect. For example, the whole
"Lisp is functional, not procedural" business is largely irrelevant.
The control structure and overall organization of a program in Common
Lisp is extremely similar to that of a Java or C++ program. All these
languages have subroutine calling, object-oriented programming,
iteration, recursion, data structures, structured programming,
exceptions, multithreading, etc.
Any claims that groups can't program in Lisp, or that Lisp programs
are inherently unmaintainable, are nonsense. There is nothing about
writing a program in Lisp that makes it harder to maintain than a
program in C++ or Java.
>> Static Typing
Static typing is one of the biggest differences. I agree that Common
Lisp basically does not have static typing (yes, I know about
"declare"). Some languages have static typing that is so restrictive
that it seriously gets in the way of getting useful work done, and in
the Old Days we of the Lisp world were very much reacting to
constricting type systems. I am pretty happy with the Java concept of
types, which I feel provides useful expression of intent in the code,
and useful compile-time error checking, and doesn't get in your way
very much. The parameterized typing coming to Java (in JDK 1.5, last
I heard) should improve the situation further. Still in all, it's not
that big of a deal, and if I went back to using Lisp without static
typing I don't thihk my life would be all that different.
>> Parentheses and Macros
Lisp's unusual syntax, with the parentheses and what used to be called
"Polish notation" (i.e. no infix operators), bothers some people. It
looks unusual and unwieldy to those who are unaccustomed to it.
However, once you get used to it, and if you use a decent programming
environment (particularly a text editor that can "indent for Lisp" and
otherwise knows Lisp syntax, such as but not necessarily Emacs), you
quickly get used to it and it seems perfectly normal. (It is also
possible to make a Lisp-family langauge with a more conventional
syntax, as the Dylan designers decided to do.)
Lisp's ayntax allow programmers to see programs in the form of a
simple data structure, which is the basis upon which is built the Lisp
"macro" facility, one of the truly different and powerful features of
Lisp. Lisp macros make Lisp an extensible language, in profound sense
of the word "extensible". They are a form of programming abstraction
that, properly used, can help make complex systems more simple and
understandable. Now that I used Java, I do miss Lisp macros. On the
other hand, it turns out it's not such a big deal, and if I had the
power of Lisp macros in Java it would not really change my life all
that much.
>> My Experience
Lisp: I wrote the first Emacs-written-in-Lisp (known variously as
EINE, ZWEI, and Zmacs). It was in fact the second Emacs ever,
developed concurrently with the original Emacs (written in TECO). I
have also written in Lisp an interactive debugger, a local area
network control program, a compiler, and an object-oriented database
management system. I worked with small and medium-sized groups. I
maintained lots of code written by people other than myself. I was
one of the five co-authors of "Common Lisp: The Language". I used
Lisp, in an implementation that eventually evolved into Common Lisp,
between 1976 and 1988.
Java: I was one of the reviewers of "The Java Language Specification".
I edited early drafts for Bill Joy and Guy Steele Jr. I co-wrote the
Java part of the ObjectStore database management system, and a
transactional data manager called "PSE Pro for Java", both from Object
Design. Currently I am developing business-to-business integration
software in Java at the same company, now known as eXcelon. I have
been using Java since early 1996. (In between I did C++.)
I'd like to go into the question of development environments but this
is long enough as it is. In a nutshell, I really wish I had my good
old Lisp Machine development environment again; what I'm using for
Java nowadays is stone knives and bearskins.
-- Daniel Weinreb
dlw@exceloncorp.com
In fact, for the last twenty years at least, Marvin has been supportive of a very wide range of approaches to AI research. He wants to see people trying all kinds of methods and philosophies. In fact he wants nothing more than for a younger generation of AI researchers to come in and vigorously try different approaches. He's been talking that way ever since the 1980's. So your portrayal of him is really inaccurate.
And your implication that he is somehow holding back the field of AI is not too plausible, either. It's hardly as if he is controlling AI research all over the world. He's not even controlling AI research at MIT. If neural networks haven't yet taken over the world, you can hardly lay the blame at Marvin's doorstep.
This book is absolutely top-notch. I've been working with SSL lately and have found the book invaluable. The writing is crystal clear and the author really knows what he's talking about. It's worth reading even if you don't need to know SSL, just for general education and edification (although if you're not using SSL you can skip over the sections that go into detail). It doesn't get any better than this.
If your company doesn't ever plan to be publicly traded, how do you expect to be able to convert your stock options into money?
Nevertheless, that's not the same thing as the answer to the question "How do you value your options?". The question really means "How much would I have to pay you for you to give up your options?" If you really valued your options at "nothing", you'd be willing to give them up for $1, which you are presumably too wise to do. On the other hand, if someone offered you $1,000,000, you'd probably be wise to accept. Somewhere in between is where you value your options.
Luckily for you, nobody's offering, so you don't have to decide.
Indeed, ObjectStore (and maybe other OODBMS's, I genuinely don't know) is being used as a middle-tier storage platform quite a lot these days. This has turned out to be one of the best fits for its technology. We have a product called Javlin which provides various enhancements for ObjectStore that are particularly relevant to this kind of use.
The term "object-oriented database" means many different things to different people, so there is no one way to reply to the wide range of things being said here about them. The best I can do is present my own personal view, which should not be taken as an official statement of my employer.
The most important thing I want to say is that I don't agree with the original article's assertion that, in general, people who are using relational database systems ought to be using object-oriented database systems.
Way back in 1988 when the original engineering team first started to design ObjectStore, we had no intentions (or illusions) of replacing relational database systems. Rather, our purpose was to create a data management system suitable for certain applications that, so far, had not found existing data management systems to be appropriate. There were many examples of these, but just to keep things simple I'll focus on the one that was most representative: computed-aided design (CAD).
In 1988, no leading CAD system was using relational database systems to model their most fundamental data, such as transistors, connections between electronic parts, traces on PC boards, electronic schematic diagrams, and so on. As far as I know, production CAD systems still aren't using relational databases for this purpose.
Our goal was to design a data management system suitable for CAD (and other applications) for which RDBMS's didn't seem to be doing the job. Why? Because they had a very different set of requirements, and they needed a different tool to do a different job. The only way to understand what we were up to and where we were coming from is to think carefully about our target market, and how different it was from the traditional RDBMS target market.
ObjectStore was originally designed with goals in mind such as:
(1) Be familiar to the programmers who develop things like CAD systems. At the time, C-based object-oriented programming langauges such as C++ and Objective C were starting to emerge, as the programming world started to see the benefits of "object-oriented programming", or, at least, the useful data modelling capabilities provided by OOP. We bet that C++ would probably catch on in our target markets, and I think we turned out to be right. So our goal was to make a system that would be easy to use for programmers trained in C++. We wanted to say that you could program very much as you are accustomted to program, and with a little extra effort, it can all be stored away automatically. (We didn't care about people trained in SQL: they were not our target market.) Also, make it as easy as possible to use your existing base of C++ code with the new data management system (i.e. we cared a lot about pre-existing code; the issue of pre-existing relationsl databases wasn't an issue for our target market). Integration with the programming language was way up high in our list of goals.
(2) Optimize performance for cases in which there was excellent spatial locality of reference and excellent temporal locality of reference. This is true of the typical CAD application. An example of the kind of thing we wanted to benchmark well on was: "Redisplay the screen, with lots and lots of transistors and resistors and capacitors and wires connecting them all" (or whatever, depending on the electronics technology in question). (We didn't care about TPC benchmarks: that was not our target market.) Imagine what the code looks like to redisplay the screen. Now imagine if every time you wanted to know "what wires are connected to this transistor", you had to put together an SQL string, the SQL string had to be sent over a network to a database server, the server had to parse the SQL string, the server had to run the query optimizer over the SQL string, the server had to find the appropriate pages in its data cache, etc, etc, etc. In ObjectStore the whole thing happens in a few CPU instructions. This one example of where we get huge performance gains.
(3) Provide real ACID transactions. (Yes, of course programmers had to think about transactions, and where the transaction boundaries should be, and what transactional consistency meant in their system: that's necessary for dealing with concurrency and recovery consideration. We didn't promise that you'd program exactly as you did pre-database-system; but we made it as close as possible.)
(4) Provide simple associative queries, with automatic optimization, subject to goal (1), i.e. make the query language natural for C++ programmers. It was not a goal to support some of the fancier queries that SQL can do; that didn't seem to be too important to our target market.
(5) Be able to provide fast navigation through database objects, just as any modern programming langauge lets you navigate quickly through programming language objects. Many of the most basic, inner-loop computations that programers in our target market wanted to do would, if expressed in SQL, use five-way or seven-way or ten-way joins, which existing commercial RDBMS's handled very poorly. (Ten-way joins look just find in Codd's mathematical structures, and you could conceive of ways to implement an RDBMS that would do much better at them, but it would be hard, and as far as I know it hasn't been done; Oracle's traditional target market didn't much care about them.)
(6) Distribute the use of CPU resources. The CAD people were moving in the direction of using what were then called "workstation"-class computers, connected by local-area networks, in which the "client" workstations had a great deal of processing power and memory (by the standards of the time), and it made sense to take advantage of that.
We were not the only people thinking this way. There were many, many papers published by university and corporate researchers, saying the same things. Our principal competitors, as far as I've ever been able to tell, were using the same general principals. The benchmarks that our customers used to evaluate us were obviously designed around these same assumptions.
Let me point out again that "object-oriented database system" means many things to many people: the list of goals that I gave above is certainly not what a lot of people mean by OODBMS!
Now, as it happens, most of the leading CAD vendors did not end up buying ObjectStore and building CAD systems based on it. If you look, you'll find that they are using pretty much the same technology now that they were using twelve years ago. At the time, many of them were talking about upgrading to use some kind of data mangement system, very much like what we were talking about: we were in touch with the CAD company engineers and spent a lot of time talking to them about what they wanted. Unfortunately, for various business reasons, most of them did not move ahead with major advances to their technical underpinnings. (That said, there were some new CAD companies that did created CAD systems based on ObjectStore, some of which are still being sold today.)
In particular, the locality-related optimizations (point 2 above) were fundamental in our overall design. In our classic use case, one user would access a hunk of information, and pretty much all of it would get cached on his powerful workstation, and would be operated on at the powerful local workstation for a considerable period of time. So it made a lot of sense to move a good deal of processing to the client side, and try to minimize the burden on the (shared) server.
There are plenty of tradeoffs involved in these optimizations. For applications without that kind of locality, they aren't optimizations at all. Also, basic pure ObjectStore does a lot less runtime error-checking than it might, because the CAD engineers specifically did not want to pay for that kind of runtime error-checking, at least from what we learned. (Note that there are other OODBMS's that make very different choices here; much of what I'm saying, especially about performance issues, should not be taken to apply to anything besides ObjectStore.)
We built a different tool to solve a different problem. So all the arguments of the form "relational database experts are easier to find and hire than OO database experts" are true, and probably good answers to the original article, but aren't relevant to our business, because we were not trying to replace relational databases for what relational databases are good at.
Similarly, if I may indulge in sarcasm for a moment, not every collection of human beings on the planet Earth have (a) already learned about expressing data in normal form (which is really quite un-obvious until you study it) and already learned SQL, and (b) have no idea how to program in any modern programming language. Some people are skilled C++ programmers who have never used a relational database system in their life.
I still think we did a pretty good job of what we originally set out to do. As things developed, though, we ended up with a far more diverse and heterogeneous set of customers than we had originally expected: lots of telecommunications, lots of financial applications, and all sorts of exotic stuff. Our customers have tried to use ObjectStore as part of all kinds of software systems, many of them really cool and innovative and not falling into any traditional category at all. Some of them had requirements that were pretty much like the ones above; some had very different requirements. We did a good job for some of them and a not-so-good job for others. In my personal opinion, we had trouble developing ObjectStore because our requirement set became so diffuse; that is, everybody was asking for something different.
There turned out to be some applications in a middle ground between what we designed for, and what relational database have traditionally been used for. So this complicated things a lot, and meant that sometimes ObjectStore was being asked to do things too far away from what its original design was intended for. One thing I'll 'fess up to is that I think in the past our company sometimes tried to sell ObjectStore for applications that were too far away from the original design goals, and so sometimes it would turn out that ObjectStore just wasn't the right tool for the job. I think we're a lot better about that these days then we used to be. And we've found more areas in which our kind of system turns out to be a good and appropriate tool.
I can't reply to everything everyone has said. And a lot of what has been said, rightly, has been directed at the original article, whose points are very different from my own.
But, here are some for which I think a reply is needed:
To "sailesh" the IBM DB/2 engineer:
- Your statement that you can't do a range query is completely wrong. ObjectStore has supported B-tree indexes and range queries since its first release.
- You say "Relational databases are great at recovery of your data using log information". So is ObjectStore. It uses write-ahead logging and recovers just fine, and has done so since its first release.
- "Actually, academia has well and truly come to the conclusion that RDBMS technology is the way to go." There isn't any one single way to go, any more than there is one single best programming language. I am still waiting to see an interactive CAD system that models its basic items using relations. If IBM's CAD sytems are now using DB/2 at this level, please let me know. As for academia, they have largely lost interest in the whole issue and moved on to other things.
To "micromoog", who said "Given the dramatically higher costs associated with designing and maintaining an OO system, most applications just don't need the incremental performance gains associated with it." I completely deny the claim that designing and maintaining OO systems has a "dramatically higher cost". Taking a bunch of guys who know COBOL and SQL, and try to dump C++ on them, sure, that won't work very well. But the whole point of OO systems is to make design and maintenance easier and cheaper, and I certainly belive that they succeed that way. Why do you think nearly every serious programming langauge successfully introduced in the last ten years has included OO programming? And the performance gains for using ObjectStore as opposed to an RDBMS, for those applications for which ObjectStore is suitable, are not "incremental"; they can easily be 100x or 1000x.
To the small number of people who claim to have used ObjectStore and had all kinds of trouble with it: well, I'm really sorry that was your experience. We have certainly had many customers who've been quite successful with ObjectStore. We've had a lot of repeat business. I've been involved with our support organization to know that the great majority of user experiences were not nearly as bad as the ones you report.
To the people who complain that OODBMS's give you a problem with "vendor lock-in": well, there's a lot of truth in that. There aren't good standards among commercial OODBMS's, largely due to fundamental and irreconcilable philosophical differences, and different choices made on certain major engineering tradeoffs. On the other hand, it's pretty late in the day to claim that's it's easy to switch among commercial relational database systems. They are each full of handy features that no other vendor supports. They all have different SQL dialects with different extensions to the query language. And, of course, they all have entirely different data storage formats. How many times have you heard of an IT organization that decided one day to switch from Oracle to Sybase or from DM2 to MS SQL Server? Yes, it happens sometimes, but it's hardly something one guy does in an hour.
To "Pedrito", who is in the business of selling applications designed to work with the customer's existing database system, and for which relational database systems are approriate: yes, certainly you should be designing your applications to work with relational database systems.
To the people who say that the problem with OODBMS's is that "there isn't one single object model" or "objects lack the theoretical underpinnings of Relational Algebra": gee, then why don't we use relations as the fundamental data model of all our programming languages? Really, the significance of the "mathematical underpinnings" business often seems to take on spiritual or mystical dimensions. From the point of view of our CAD engineer, there's a tradeoff between (1) having a sophisticated ad-hoc query language that lets you express complex queries that you never anticipated, and (2) getting that screen to redisplay nice and fast. It would be great if you could have both. I have yet to see how to do that. Maybe someone else will figure it out. As an engineer, you learn about the concept of "engineering tradeoffs". A design approach that's best at one thing often isn't best at another thing.
As for me, I decided that nine years of working on ObjectStore was enough; I liked doing it but one needs variety in life. I am now working on our XML-based business-to-business integration server product. It's based on our XML information management product, which is based on ObjectStore internally (but invisibly to its users). So I've been rather far from the actual ObjectStore business of the last several years. So I don't feel comfortable trying to summarize "how it all turned out". I know that we have lots of customers using ObjectStore together with relational databases, letting each do what it does best, and maybe that's one place that object-oriented databases are headed.