Kent M. Pitman's Second Wind
12) Good texts for learning Scheme?
by drenehtsral
I have recently been working on learning Scheme in my spare time, with the eventual goal of writing a scheme based scripting system to run the guts of a massive adventure game/graphical mud sort of system, everything from environment simulation (predator/prey cycles, etc...) to 3d models (i.e. models will be geometry glued together by scripts so you could have trees that by a random seed and a growth level variable have grown over time and are unique to provide interresting landscape features). Scheme is appealing because it's simple, powerful, and adapts well to the idea of a threaded interpreter.
To further my goal of learning Scheme inside and out, I've been reading "The Little Schemer," as well as "Structure and Interpretation of Computer Programs." Do you have any other recommendations for good Scheme programming texts?
Kent M. Pitman: You can get a list of textbooks from Schemers website. If you can articulate a particular need or preference that you think should help narrow down the many available choices, I'd suggest posting a more specific inquiry to the comp.lang.scheme newsgroup.
13) Overlooked practical aspects of Lisp
by hding
Why do you think that people so often overlook many of the wonderful things in Common Lisp such as unwind-protect, the whole condition system (which you are of course closely associated with), and so on - things that make it very useful for day-to-day programming, and are there any such things that you'd particularly highlight, or conversely that you wish had become part of the standard but did not.
Incidentally, thank you for all of the insight so generously and regularly poured forth in comp.lang.lisp.
KMP: Well, people program with tools that are familiar to them. Unless Common Lisp is someone's first language, it'd be easy for them to overlook the things it contains that are not like the things they're familiar with. There's a certain irony here because often the reason people will leave a language for another language is that they've reached the limits of what they can do with the first language and they need more power. So you'd expect that they'd aggressively look for features of the new language that were different than the things they've used before. And probably some do. But you're right that others cling to the safety and familiarity of the operators they could just as well have found in the old language they left behind, and so in the process they miss out on what the language can offer them.
Fortunately, unwind-protect is finally (pardon the pun) present in Java. And some hints of the Common Lisp condition system made it into Java as well. So probably people who come to Common Lisp from Java will be inclined to seek out those capabilities. But there's a lot of other stuff there and I hope new users will indulge their curiosity and take the time to explore.
As to what we should have in the language, the main omission of note is some sort of system definition tool (the in-Lisp analog of make). It was a shame that we did a feature-freeze on ANSI Common Lisp in 1988 but didn't get the standard out until 1994, and the suggestion of including such a tool didn't come until the after-freeze period. All vendors offer such a facility, but programs would be more portable if there were a uniform solution.
There are also quite a number of things about Common Lisp that are available in the same or similar forms in nearly all implementations. Multi-tasking, sockets, database access, external function call, windowing, and so on. It wouldn't be bad to have included any of these, but the fact is that they weren't ready for standarization in 1988. At this point, though, I think other mechanisms than standards are the right way to proceed.
The Lisp community used to expect the delivery mechanism for new functionality to be a new language spec. But that requires working through consensus standards bodies. The problem is that, by their nature, standards bodies are synchronization mechanisms. The problem with synchronization mechanisms in a massively parallel world is that they slow things down. The world is not going to wait for us to slow down, so I think we need to evolve mechanisms that will keep up better with a degree of pace that is externally dictated.
I think this is an area where Lisp as a community has been slow to respond. There need to be community mechanisms for sharing the many great commercial and private packages people have been creating in Lisp, so that we can properly reap the cross-product benefits of our community's productivity. I see evidence that this is changing. The Common Lisp Open Code Collection (CLOCC) is one such mechanism that addresses open source code. I'd like to see similar mechanisms arise for the exchange of proprietary products as well.
As to my posts on the comp.lang.lisp newsgroup, thank you. I'm glad you enjoy them. Frankly, I always consider it a victory to hear I haven't bored everyone to death. In background I've been working on putting together several books on Lisp, but one never quite knows if one will finish such things. I regard comp.lang.lisp as a kind of insurance policy, assuring that at least some piece of what I have seen and done in my career gets transferred from individual memory to global group memory.
I think preserving individual experiences for history is quite important. In the future, this will happen naturally due to logs kept by online collaboration tools. But I'm especially worried about the records of what happened between about 1960 (the birth of programming languages) and 1994 (the birth of the web). Most of everything in that time range is recorded on paper and will eventually be lost. Looking back from the future, I expect it to be as confusing to figure out how the information society was born as it is to look back in a telescope to see the birth of the Universe. You'll get very close, but then you'll get to a point where you can see nothing. The informational big bang. I've been working on webbing all of my old hardcopy papers, and I hope others of that era will commit to doing the same.
14) Lisp - Scheme - ML
by Tom7
I know a lot of big academic (erstwhile) lisp shops, such as CMU, have transitioned away from lisp to ML [standardml.org] and relatives [inria.fr]. Some of the reasons we might give are:
- Sophisticated type systems, catching most bugs before your program is run, ensuring safety, etc.
- Much more efficient (http://www.bagley.org/~doug/shootout/craps.shtml), partly due to compilation strategies using types
- Increased modularity and abstraction
- Pattern matching, (subjectively) more natural syntax
In fact, I'm one of those people. I've been scoffed at by lisp fans, but most had never used ML. But I have an open mind, so, in the face of more "modern" languages, what advantages do lisp and scheme offer? Do you think that these advantages are fundamentally impossible to achieve in a typed setting?
KMP: First, I assume by "typed" you mean "statically typed." I think of Lisp as "dynamically typed." I think of most machine languages as "untyped." I've heard statically typed languages sometimes called strongly typed, and I sometimes use this terminology myself out of habit, but I've grown to dislike it because it seems to me that the issue of strength ought to refer to whether you check types, not when you do. The terms "static" and "dynamic" seem to me to better get to the heart of the matter.
To quote Abraham Lincoln, admittedly somewhat out of context, "People who like this sort of thing will find this the sort of thing they like." So to somewhat flippantly re-interpret Lincoln's remarks in a modern context, applying perhaps just a bit of obligatory political spin to the result: The fact that functional languages appeal to people who like functional languages is not a proof that functional languages are of general purpose appeal.
I think the real reason that CMU (or any university with a grant-based funding model) changed its direction is good sources of funding in research depend on saying you're doing something "new and different." Such a shift doesn't imply that the thing left behind wasn't "tried and true," but only that "tried and true" is not what gets research dollars. Research must constantly stir the mix, but that doesn't imply obsolescence to what came before. So don't read too much into that.
Answering each of your points in detail might require a whole article, but I'll touch on each in brief:
- Sophisticated type systems, catching most bugs before your
program is run, ensuring safety, etc. Much more efficient partly due to
compilation strategies using types.
Actually, it's funny that you both mention the CMU project and then make this comment. Before moving away from Common Lisp, the CMU crowd was successful in demonstrating to the Lisp community's satisfaction that there were enormous opportunities offered by the Common Lisp language design in terms of type inferencing that still today go untapped by implementations. This is really a market issue, not a language design issue. The fact is that although other languages do a lot more type inferencing, vendors are not getting huge numbers of bug reports saying that better type inferencing is what stands between programmers and the commercial success of their product. Over time, I think you'll see more and more interesting type analysis done, but such work is always balanced against other needs of users, such as CORBA, COM, RMI, and web interfaces, for example, such as UI toolkits and debugging options. When I observe, as I often do, that languages are political parties, this is what I mean. They are each responsive not to the needs of the world, but to the needs of their constituencies. And the Lisp constituency, while it is not oblivious to the value of type inferencing, does not see that issue as its number one priority.
- Increased modularity and abstraction.
This is quite a multidimensional space. I think Lisp provides great opportunities for modularity and abstraction that other languages do not. And yet, there are sometimes things I can't abstract as well as I wish. An example of a minor omission: Common Lisp's CLOS doesn't do protocol abstraction as well as Zetalisp's New Flavors; among other things, one can't declare that certain unimplemented methods are required. But with the use of the macro system and the Meta-Object Protocol (MOP), one can add this kind of thing. Further, the package system is missing certain kinds of inheritance capabilities I've often wished for, but I recently sat down and did the work of writing my own versions of defpackage for my own use, adding the capabilities I wanted in a way that my own tools can use, and I had no difficulty. For the most part, I've found the limitations of Common Lisp's abstraction capabilites to be incidental, and not deep, and I've found its syntactic reorganization capabilities more than capable of making up for it.
-
Pattern matching.
I think you're right that Lisp doesn't do pattern matching. Whether or not that's a good or bad thing is subjective. I think there are people who like pattern matching and people who don't. In fairness to Lisp, though, on the few occasions in my career where I've felt a strong need for pattern matching, I've been able to implement it easily. And, importantly, Lisp's syntactic adaptability has allowed me to make my personal implementation look as natural in the programs I write as if it were natively provided by the language; most other languages don't give me the syntactic control to be able to add new functionality in a way that feels appropriate to the language. So personally, I don't find this a strong negative; rather, I see it as an opportunity for you to create a layered library that supports the needs of yourself and others like you.
-
(Subjectively) more natural syntax.
I don't think you can make the case that much of any language has "natural" syntax. COBOL and HyperTalk gave this the fairest shot and there's a big difference even between them and any natural language. I personally find Lisp syntax remarkably natural in that it focuses on symbols that you could say out loud, marking them minimally to indicate grouping. Other languages contain lots of special-purpose markers like commas, semicolons, asterisks, and braces/brackets/parens that are used in quite nitpicky ways. All this to say that you're right on this one: it's subjective. And as such, I hope I can fairly dismiss this as an even draw.
15) Lisp in Mathematics Programming
by An Anonymous Coward
Gregory Chaitin has a book called "The Limits of Mathematics." In it he claims that mathematicians should love Lisp because Lisp is basically set theory, and all mathematicians love set theory. I wholeheartedly agree with this, one only needs to look at Chaitin's Lisp programs to realize how quickly and succinctly one can arrive at astonishing incompleteness results in mathematics. So we know Lisp is great for stuff like this, really researching a mathematical subject. Do you see Lisp continuing in this direction, showing and discovering theorems, or will it move into industry? Or has it moved into industry, and we just don't know it? Do the likes of NASA and JPL use Lisp and Scheme religiously? I would bet so.
KMP: Lisp may have started out as a way of addressing abstract topics like math (logic, calculus, prime numbers, etc.) and artificial intelligence, but it long ago made the transition to commercial applications. Both Scheme and Common Lisp have been and continue to be used in real-world applications that might surprise you. These include (but are certainly not limited to) applications in airline scheduling, commercial database manipulation, computer operating systems, bioengineering, web services, document format translation, and, yes, even space exploration. Franz, Inc. has created quite a nice page of Lisp success stories that I think expand on this much better than I could in the space I've allowed myself to answer this question. And speaking of NASA/JPL, they did a comparative study of Lisp vs Java and C++ that some might find interesting.
16) Scheme in CS
by An Anonymous Coward
It seems many of the more popular CS programs in the world use Scheme as a teaching language. A lot of times, students complain about this, saying they'd prefer to learn about C or another language that is considered "apt for industry." I used to be like this too, but have now discovered the error of my thinking. How have you convinced others that while the latest programs might not be written in Scheme, that it is worth a student's time to learn Scheme. Many seem stuck to the point that if they won't use it outside of school, they shouldn't learn it. How can we convince them otherwise, to become scholarly citizens instead of drones?
KMP: I think the thing to explain to a student is that the world is ever changing and that one cannot put ones eggs all in one basket. Furthermore, modern environments are often quite heterogeneous, with different languages and systems being used together cooperatively. Especially for a CS student, who often has the luxury of time that a person in the job world does not, I think it's worth taking time to learn as many different languages as possible. This not only exposes the students to alternate ways of thinking, but it also prepares the student to quickly change modes of thought or languages of expression later. Once on the job, one often can't afford the ramp-up time to learn a new language at the point it becomes necessary to use. Better to already know it and just have to "brush up".
One is much more likely to consider alternative approaches if one has a sense of what is involved in them; it's very easy to fear the unknown, even when the unknown might be of great help. So get to know as many things as you can while you can. Common Lisp and Scheme, which I regard as two very different languages, by the way, should definitely be among the things every student studies. But they should not be the only things the person studies. Like it or not, there is a lot the professional programmer needs to know to be really successful not just tomorrow, but for a lifetime.
As Oliver Wendell Holmes is often quoted as saying, "A mind stretched to a new idea never returns to its original dimensions." In order to stretch a student's mind, I recommend they make a list of "kinds of languages" and then learn as many different kinds as they can. Here are some that come to mind, though I'm sure others with different experience than me might reasonably contribute still others.
- A block-structured language, such as Algol or Pascal or Scheme.
- A line-oriented language, such as Fortran or BASIC.
- A rule-based logic language, such as Prolog or EMYCIN.
- An Englishy language such as HyperTalk or Cobol.
- A "stack" language such as PostScript.
- A "line noise" language (heavy emphasis on one-letter operators) like APL or Teco.
- A dynamic object-oriented language, such as Common Lisp or Smalltalk .
- A strongly-typed, statically analyzed language such as ML or Haskell.
- A dynamically-typed functional programming language, such as Scheme.
- A string or macro processing language, such as Perl or m4 or TeX or Teco.
- A database access language, such as SQL.
- An abstract high-level, assembly language, such as Java.
- A concrete high-level, assembly language, such as C.
- A low-level, traditional assembly language.
- A scripting language, such as Javascript.
- An interface-definition language such as Visual Basic, HyperTalk, or Javascript.
- A document-structuring language such as XSL or TeX.
- A language with a modern error system, such as Common Lisp or Dylan.
- A reflective/introspective language such as Common Lisp or Java.
- A symbolic programming language, such as Common Lisp or Scheme.
- A security-oriented language, such as Java or MOO.
- A language where both programs and data are fully persistent, such as MOO or HyperTalk.
17) A question for Kent
by MarkusQ
Do you have a maclisp manual I could borrow?
KMP:For those not familiar with Maclisp, it's a defunct dialect of Lisp that predated and strongly influenced the design of Common Lisp.
I've been working on webbing The Revised Maclisp Manual, which I had published on paper back in the early 1980's. It's not quite ready to go out yet, but should finally be ready sometime in the not terribly distant future. Probably a month or two. Watch the site maclisp.info for more information.
18) Open Implementationsby Martin Pomije
What is your opinion of the idea of Open Implementations from Gregor Kiczales? Do you think that his idea could help Lisp be more widely used?
You can see him giving a lecture about this idea here. [microsoft.com] The video is only available in Windows Media format on this site.
KMP: I hadn't seen Gregor Kiczales's talk on Open Implementations, so I enjoyed watching it. Thanks for the pointer!
The talk made me think back to various related ideas I've seen batted around for a long time, the earliest of which that I can recall is a short paper on something called "Capsules" (an object system where classes were allowed to have multiple implementations) by Richard Zippel back in the late 1970s or early 1980s at MIT. Often, especially in a university environment, people will make up such a concept, bat it around for a bit, and then go on to something else. There are some very interesting ideas there and I'm glad to see that they're being pursued seriously, especially by someone as thoughtful and talented as I know Gregor to be.
As a formalized area of study, this topic of "aspect-oriented programming" is new to me. It reuses some old ideas in new ways, and introduces some new ones along with it. I'm only just barely becoming conversant in the terminology, so I can't really speak to it from a theoretical point of view. But it looks promising. And from a practical point of view, I can note that I'm getting daily on-the-job training in it through my consulting relationship with The Software Smith. They're using Lisp as a vehicle to apply the principles of aspect-oriented programming, and the results they get are quite spectacular.
19) What was up with CLisp's "loop" form?
by Jayson
Did you can have anything to do or know who had anything to do with the "loop" form in Common Lisp? Why does it look and feel just like a FOR loop on C (from the Graham book):
(loop for x = 8 then (/x 2)
until (< x 1)
do (princ x))
This is one of by biggest minor nags about CLisp and I am very curious what was going through the committee's collective head. Didn't anybody balk at this enough to at least get the syntax cleaned up?
KMP: The example you cite is quite simplistic and if this were the only reason for using LOOP, we wouldn't have it. Lisp has a number of other iteration operators for doing simple loops like this. However, the reason for using LOOP is that it can represent much more complicated arrangements of iteration paths and collection techniques. I used to grumble a lot myself about how "un-Lispy" LOOP seemed, but over time I come to the belief that the benefits outweigh the costs. A loop like this:
(loop for x from 0
for y in some-list
when
(good-result? y)
collect (list x y))
is easy to write and maintain, and much easier to explain than the equivalent, but more Lispy:
(do ((x 0 (+ x 1))(y-list some-list (cdr y-list))
(result '()))
((null y)
(nreverse result))
(let ((y (first y-list)))
(when (good-result? y)
(push (list x y) result))))
The Common Lisp community likes to offer the traditional Lispy notations for places where they enhance readability, but we also offer alternative notations for situations where we've learned there's a call for it. We leave the choice of which style to use up to the individual taste of the programmer. Common Lisp is not a minimalist language offering only one way to do things or rigidly attempting to force people into a single programming paradigm.
By the way, this is a fundamental difference between the ANSI Common Lisp design philosophy and the Scheme design philosophy. The introduction to the Scheme specification states:
Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary. Scheme demonstrates that a very small number of rules for forming expressions, with no restrictions on how they are composed, suffice to form a practical and efficient programming language that is flexible enough to support most of the major programming paradigms in use today.By contrast, the charter for X3J13, the group that designed ANSI Common Lisp, stated the following in the X3J13 charter:
It will codify existing practice, provide additional features to facilitate portability of code among diverse implementations, and establish normative Common Lisp programming practice. The committee will begin with the language described in Common Lisp: The Language by Guy L. Steele Jr. (Digital Press, 1984), which is the current de facto standard for Common Lisp. Whenever there is a proposal for the standard to differ from Common Lisp: The Language, the committee shall weigh both future costs of adopting (or not adopting) a change and costs of conversion of existing code. Aesthetic criteria shall be a subordinate consideration.
In other words, the Scheme community is a very conservative community that is highly focused on keeping its language specification as highly aesthetic and minimal in size as possible. By contrast, the Common Lisp community is an industrial standard that is concerned with messier issues of compatibility, portability, and commercial need; while the Common Lisp community cares about aesthetics, it does not allow aesthetics to dominate practicality as a design criterion.
The relevance of this here is that the Lisp family of languages is made up of a number of smaller communities who share a few core ideas, but really have some very divergent points of view. Each is worthy of study in its own right. One should not, having looked at Scheme, assume they have good intuitions about Common Lisp, nor vice versa.
You use a programming language for one primary reason: you believe most other people in your domain space are using it. This is why C will be around for at least another fifty years, Java will unfortunately be around for at least twenty, and perl will probably last another fifteen.
No one cares how useful a programming language is - they care how many competent programmers they can hire who can code with it. This is why Lisp, ML, Haskell, Prolog and the rest have been consigned to the dustbin.
The only reason you feel comfortable reading C or C++ is because you actually spent some time learning these languages. If you spent just a little time learning Scheme or LISP you'd realize that the language is really not hard to read or use at all.
I'm not that impressed with the NASA study linked. Anyone who has done any Lisp programming in school will know that Lisp is a great language for algorithmic problem solving, and the problem used in the study really plays to that strength in Lisp. Perl would also likely kick C++ and Java's ass in this task. That doesn't mean that Perl or Lisp would necessarily be as good a choice as C++ or Java in a wide range of problem domains.
Lisp is a great language, but it traditionally has not interfaced as well with the Real World as a number of other languages that were more practically oriented.
I know that people absolutely rave about the pleasures of programming on all-Lisp machines, but most of us don't have those, and without either having a Java-style complete set of API's for doing GUI and system interfacing, or having C's ability to directly call system API's, one is left without the tools to directly influence the environment. And, yes, I'm sure there are implementations that have those things there, but without them being absolutely standardized, you can wind up locked in a ghetto of your own making.
As a side note, Kent, don't you think it a little churlish to denigrate Java as merely 'an abstract, high level assembly language'? ;-)
- jon
Ganymede, a GPL'ed metadirectory for UNIX
Hiring programmers not willing to learn a new language is probably not a good idea, certainly not if you are a small(ish) company that needs every advantage you can get.
If you're a contract programmer looking to move from job to job at Fortune 500 companies every 6 months, then by all means stick with C++/Java.
Much of choosing a language (or a project, or even an industry) comes down to being willing to bet the farm on your success. Big companies rarely bet the farm on anything. Small companies have already done it just by starting up. It's not surprising that more and more start ups are choosing Python or Lisp. See Paul Graham's article on "beating the averages" for his take on this issue. Google will find it for you. There were even some slashdot articles about him.
((lambda (x) (x x)) (lambda (x) (x x))) http://www.endpointcomputing.com a scientific approach to custom computing.
Actually, I'd argue that the genericity of lisp's syntax is as much a hindrance as it is a help, emacs parenthesis-matching aside.
The above quoted lisp sexp could mean almost anything or nothing, depending on the context in which that sexp occurs. (Is it data? Is it code? In what evaluation context will it be processed?). Java, C, and most other languages at least give you more distinct contextual tokens to guide you in your understanding.
- jon
Ganymede, a GPL'ed metadirectory for UNIX
The argument for efficiency via type inferencing is made for many languages, including BASIC. Seldom is it implemented. Smalltalk implementations probably got further in that direction than LISP did.
The object system in Common LISP was an afterthought, and it shows. It's really a macro package kind of thing. LISP has a strong history of overly elaborate macros; ever see the MIT LOOP macro? These have the usual problem; what you're looking at in the source isn't what's running, and debugging macros (especially those written by others) is hell. Most other high-level languages have banished macros, or discourage their use.
If you really want dynamism at run time, Smalltalk has most of what you want, and it's cleaner. But Smalltalk is dying, if not dead. Even the ParcPlace people gave up. Java ate their market.
But in the usability sense explored by the Psychology of Programming field, there are important differences between programming languages, in terms of ease of use, learnability, maintainability, readability, syntactic surface area, cognitive load, etc.
The trivial fact that a language is Turing complete, does not mean that it's well designed or easy to use. It's not all just a function of the programmer's skill. Certainly there are objective qualitative differences between programming languages that make some substantially better than others.
-Don
Take a look and feel free: http://www.PieMenu.com
While I agree with your sentiment in principle, some customers agree with the usage of Lisp, even though they know that maintainance will be much more expensive. It depends on the returned value of that usage.
For example, in one of our projects we substituted a yacc/flex/C++ programm with a Lisp program, the latter running 15x [sic!] faster: The C++ program needed around 17 hours for a job, the Lisp program does it in roughly 1 hour. The job was about data analysis and statistics - and we got statistics just an hour away from the data collection, and not one day away. This advantage is more than worth the raised maintainance costs.
OK, for our customer (a large European bank) it will be hard to maintain the system himself. So what? They're used to this; they can't maintain SAP or MS Office either, but rely on it nevertheless. In addition, I doubt that they would have been able to maintain the yacc/C++ program - they're a Cobol shop, after all. The Real World(tm) runs on Cobol, don't forget this... :-)
Joachim
People don't write Manifestos any more -- what's going on in this world? [Frank Zappa]
Just as another data point, my customers are low-end customers who can't afford technology at all. If replacing me meant paying a programmer a commercial wage, they'd be sunk. It doesn't matter if it was C or Lisp. What will make them win is getting a cheap enough price that they can get any solution at all in their price range. And that requires very smart software, of the kind I am confident I can write in Lisp and not confident I can write in any other language, because I can afford to write that software but I cannot afford to both employ consultants for them and charge them the almost-pennies a month they're willing to pay.
I happen to also do some consulting on the side for people who want to use Lisp for their own reasons, and that keeps the bills paid while I develop the software for the market area I've picked off (that I'm being deliberately vague about because I have not formally entered it yet, except for some limited testing). But my reasons for using Lisp are simply that it enables me to do what I need to do, and it enables my customers to get something they can't get at all in any form because no one cares about them.
Kent M Pitman
Philosopher, Technologist, Writer