Slashdot Mirror

HughPickens.com writes Micah Lee writes at The Intercept that coming up with a good passphrase by just thinking of one is incredibly hard, and if your adversary really is capable of one trillion guesses per second, you'll probably do a bad job of it. It turns out humans are a species of patterns, and they are incapable of doing anything in a truly random fashion. But there is a method for generating passphrases that are both impossible for even the most powerful attackers to guess, yet very possible for humans to memorize. First, grab a copy of the Diceware word list, which contains 7,776 English words — 37 pages for those of you printing at home. You'll notice that next to each word is a five-digit number, with each digit being between 1 and 6. Now grab some six-sided dice (yes, actual real physical dice), and roll them several times, writing down the numbers that you get. You'll need a total of five dice rolls to come up with each word in your passphrase. Using Diceware, you end up with passphrases that look like "cap liz donna demon self", "bang vivo thread duct knob train", and "brig alert rope welsh foss rang orb". If you want a stronger passphrase you can use more words; if a weaker passphrase is ok for your purpose you can use less words. If you choose two words for your passphrase, there are 60,466,176 different potential passphrases. A five-word passphrase would be cracked in just under six months and a six-word passphrase would take 3,505 years, on average, at a trillion guesses a second.

After you've generated your passphrase, the next step is to commit it to memory.You should write your new passphrase down on a piece of paper and carry it with you for as long as you need. Each time you need to type it, try typing it from memory first, but look at the paper if you need to. Assuming you type it a couple times a day, it shouldn't take more than two or three days before you no longer need the paper, at which point you should destroy it. "Simple, random passphrases, in other words, are just as good at protecting the next whistleblowing spy as they are at securing your laptop," concludes Lee. "It's a shame that we live in a world where ordinary citizens need that level of protection, but as long as we do, the Diceware system makes it possible to get CIA-level protection without going through black ops training."

Back in the dawn of prehistory, only universities, government agencies, and a few big corporations could get on the Internet. The rest of us either had computers connected to nothing (except maybe an electric outlet), Compuserve, Prodigy, AOL or another service or possibly to an online bulletin board service (BBS). And then, one day in 1989, Barry Shein hooked a server and some modems to an Internet node he managed for a corporate/academic wholesale Internet provider -- and started selling dialup accounts for $20 per month to individuals, small companies, and just about anyone else who came along. Barry called his ISP The World, which is still out there with a retro home page ("Page last modified April 27, 2006"), still selling shell accounts. We may run a second interview with Barry next week, so please stay tuned. (Alternate Video Link)

Arnold Reinhold writes "This month ends with the 125th anniversary of one of the most remarkable achievements in technology history. Over two days beginning Monday, May 31, 1886, the railroad network in the southern United States was converted from a five-foot gauge to one compatible with the slightly narrower gauge used in the US North, now know as standard gauge. The shift was meticulously planned and executed. It required one side of every track to be moved three inches closer to the other. All wheel sets had to be adjusted as well. Some minor track and rolling stock was sensibly deferred until later, but by Wednesday the South's 11,500 mile rail network was back in business and able to exchange rail cars with the North. Other countries are still struggling with incompatible rail gauges. Australia still has three. Most of Europe runs on standard gauge, but Russia uses essentially the same five foot gauge as the old South and Spain and Portugal use an even broader gauge. India has a multi-year Project Unigauge, aimed at converting its narrow gauge lines to the subcontinent's five foot six inch standard."

richard_za writes "Until now the rival compression software vendors PKWare and Winzip have had different (incompatible) ways of password protecting the ZIP format. In a bid to prevent fragmentation of the standard they have agreed to have their software support opening of the other's files. They have however not agreed to support a single standard. PKZip's encryption is RSA-based while Winzip use an AES approach which is fully documented here. The Register is running this story. PKWare has this press release."

c1ay writes "In an article at the ScienceDaily News it is reported that two researchers at the University of Rochester have discovered a new property of carbon nanotubes, ideal photon emission. "The emission bandwidth is as narrow as you can get at room temperature," says Lukas Novotny, professor of optics at Rochester and co-author of the study. Such a narrow and steady emission can make such fields as quantum cryptography and single-molecule sensors a practical reality. RSA and Elliptic Curve wouldn't stand a chance against this unbreakable encryption."

RT Alec writes "The 2003 Spam Conference has concluded, reports InfoWorld. (related read: abstracts of the conference discussions). I was unable to attend the conference, but it appears all that was discussed was filters (client and server). I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25 (you do have permission to use that SMTP server, don't you?). It is not too tough to set up an SMTP server to require authentication, or at a minimum to run off a different port. I am suprised that this is never mentioned as a cure for spam. If just AOL blocked port 25, this could reduce spam by 50% (I base this figure on close examination of the headers of the spam I receive). I was pleased to see that Barry Shein, president of The World (a Boston based ISP) was included in the talks. I am not sure by the abstract (see link above) posted if he mentioned blocking port 25. In a recent interview he did not mention it."

swm asks: "My father-in-law runs Windows/XP on a low-end machine. He gets internet access through MSN over a 56K dialup. This worked OK for many months. Two or 3 weeks ago, MSN presented him with an auto-upgrade, and he clicked OK, and the system has been virtually unusable ever since. I booted the machine to see what it does. First, it thinks he is on a LAN (he isn't) and presents a window telling him it can't connect to the internet and he should disable his firewall. I dismiss that window. A few seconds later another window pops up and tries to dial out. I can cancel and close the dial-out window, but it just comes back in about 15 seconds and starts dialing out again. No matter how many times I cancel and close the dial-out window, it just keeps coming back. I reboot the machine and let it dial out. It connects to MSN. I click the 'Offline' button, but it doesn't drop carrier. I shut down the machine and it still doesn't drop carrier. Finally I pull the power cord out of the wall socket and it drops carrier. I've checked msn.com and Googled around a bit, and I can't find any mention of problems like this. Does anyone have any idea what is going on?" Have any MSN users experienced this problem? What have you done in your attempts to solve it?

dooling writes: "Perl & XML is a well-written book that accomplishes what it sets out to do. It states in the preface that it is written for Perl programmers who want to learn about XML and what is available in Perl for XML processing. It achieves this goal, but little else. When you are done reading this book you will have been given an overview of Perl and XML, know where to begin to attack an XML document, and know where to look to find more information." For dooling's more complete review, read on below. Perl & XML author Erik T. Ray & Jason McIntosh pages 202 publisher O'Reilly and Associates rating 6 reviewer dooling ISBN 059600205X summary Good introduction to XML for Perl programmers.

The book starts out with a brief explanation of why XML and Perl are well-suited for each other. It then provides a teaser of things to come: an explanation of how to use the XML::Simple module. The first chapter concludes with some warnings and gotchas that seem a little premature since they have not really explained XML. Fortunately, most of these gotchas are covered in context later in the book.

The second chapter provides a whirlwind overview of XML -- covering its structure, DTDs, schemas, and XSLT (transformation). The discussion of XML in general, its history, and parts of an XML document are well done. They give someone who is familiar with static HTML the needed background to understand the structure of an XML document and the vocabulary used to describe it. Unfortunately, the discussion of where XML begins to distinguish itself from HTML, namely with DTDs, the new replacement for DTDs called schemas, and the transformation language XSLT, is too brief. They gloss over these topics with little explanation and few examples. That said, there are other books that do provide more in-depth coverage of XML (this book only promises an introduction).

The next five chapters cover Perl modules designed to process XML, starting with simple parsers and writers. Only methods and syntax relating to XML processing are explained. Therefore, if you are considering reading this book, you should be fairly comfortable with Perl and object-oriented (OO) interfaces to CPAN modules (nearly all the modules discussed provide OO APIs). Again, there are other books and perldoc documentation that cover Perl and it's OO features; so read them first if you are not familiar with OO Perl. If you are familiar with OO Perl, these chapters provide a good overview of the different ways XML can be processed (stream- and tree-based approaches), the advantages and disadvantages of each, and the Perl modules best suited for each approach. These chapters are the biggest strength of this book. The modules discussed in these chapters are by no means an exhaustive list of XML-related modules available from CPAN nor do the explanations of each module cover everything the module does. These chapters do, however, provide the reader with enough information that she can begin to process XML documents intelligently and know where to turn when she needs more information.

The next chapter, Chapter 8, covers XML tree iterators, XPath, XSLT, and XML::Twig. All of these topics are covered in a span of 16 pages (with only slightly over two pages dedicated to XSLT). Indeed, after reading the chapter, you may get the feeling that it was only included so the authors could cram more trite colloquialisms into the book. The short shrift given to these topics creates the impression, which is strengthened in the chapters that follow, that this book was rushed a bit to press.

Chapter 9 discusses applications of XML, including RSS and SOAP, and Chapter 10 is mostly example code. These chapters are intended to give you a feeling for what is possible without really giving you enough information to make it happen. The main problem with these chapters are the examples: the examples are long and the explanations are short. Thus, they are more useful as templates or a quick reference than for learning these topics in detail. Of course, the authors never promised you would be programming SOAP applications when you were done reading this book. And again, there are other books out there which discuss these topics in more detail. So the authors stay true to their promise throughout the book: they will introduce you to XML and tell you how to interact with XML using Perl, no more.

Personally, I found this book did, in general, give me enough information to get started using XML and pointed me where I needed to go to get more information. I am an experienced Perl programmer who is new to XML and comfortable with on-line documentation. This book seems to be written for people who fit this profile and who want to learn by doing (finding the answers to the "hard" questions as they arise). It does introduce a wide variety of XML-related topics and the Perl modules used to interact with them, which is what the authors promised to do in the preface. While it is by no means an authoritative text on Perl and XML, there is something to be said for keeping promises ...

Index As with most first-edition books, the index was adequate but not complete. For example, XML::Twig, which has an entire section covering it, does not appear in the index at all.

Contents
Preface

1. Perl and XML
2. • Why Use Perl with XML?
   • XML Is Simple with XML::Simple
   • XML Processors
   • A Myriad of Modules
   • Keep in Mind ...
   • XML Gotchas
3. An XML Recap
4. • A Brief History of XML
   • Markup, Elements, and Structure
   • Namespaces
   • Spacing
   • Entities
   • Unicode, Character Sets, and Encodings
   • The XML Declaration
   • Processing Instructions and Other Markup
   • Free-Form XML and Well-Formed Documents
   • Declaring Elements and Attributes
   • Schemas
   • Transformations
5. XML Basics: Reading and Writing
6. • XML Parsers
   • XML::Parser
   • Stream-Based Versus Tree-Based Processing
   • Putting Parsers to Work
   • XML::LibXML
   • XML::XPath
   • Document Validation
   • XML::Writer
   • Character Sets and Encodings
7. Event Streams
8. • Working with Streams
   • Events and Handlers
   • The Parser as Commodity
   • Stream Applications
   • XML::PYX
   • XML::Parser
9. SAX
10. • SAX Event Handlers
    • DTD Handlers
    • External Entity Resolution
    • Drivers for Non-XML Sources
    • A Handler Base Class
    • XML::Handler::YAWriter as a Base Handler Class
    • XML::SAX: The Second Generation
11. Tree Processing
12. • XML Trees
    • XML::Simple
    • XML::Parser's Tree Mode
    • XML::SimpleObject
    • XML::TreeBuilder
    • XML::Grove
13. DOM
14. • DOM and Perl
    • DOM Class Interface Reference
    • XML::DOM
    • XML::LibXML
15. Beyond Trees: XPath, XSLT, and More
16. • Tree Climbers
    • XPath
    • XSLT
    • Optimized Tree Processing
17. RSS, SOAP, and Other XML Applications
18. • XML Modules
    • XML::RSS
    • XML Programming Tools
    • SOAP::Lite
19. Coding Strategies
20. • Perl and XML Namespaces
    • Subclassing
    • Converting XML to HTML with XSLT
    • A Comics Index

Index

You may also want to check out Erik T. Ray's home page, Jason McIntosh's home page, or O'Reilly's page for the book. You can purchase Perl &amp XML from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Kent M. Pitman has already given you his first 11 answers to the questions you asked him about Lisp, Scheme, the creation of programming standards, and much more -- below are his answers to another eight (starting with answer #12). Thanks again, Kent.

12) Good texts for learning Scheme?
by drenehtsral

I have recently been working on learning Scheme in my spare time, with the eventual goal of writing a scheme based scripting system to run the guts of a massive adventure game/graphical mud sort of system, everything from environment simulation (predator/prey cycles, etc...) to 3d models (i.e. models will be geometry glued together by scripts so you could have trees that by a random seed and a growth level variable have grown over time and are unique to provide interresting landscape features). Scheme is appealing because it's simple, powerful, and adapts well to the idea of a threaded interpreter.

To further my goal of learning Scheme inside and out, I've been reading "The Little Schemer," as well as "Structure and Interpretation of Computer Programs." Do you have any other recommendations for good Scheme programming texts?

Kent M. Pitman: You can get a list of textbooks from Schemers website. If you can articulate a particular need or preference that you think should help narrow down the many available choices, I'd suggest posting a more specific inquiry to the comp.lang.scheme newsgroup.

13) Overlooked practical aspects of Lisp
by hding

Why do you think that people so often overlook many of the wonderful things in Common Lisp such as unwind-protect, the whole condition system (which you are of course closely associated with), and so on - things that make it very useful for day-to-day programming, and are there any such things that you'd particularly highlight, or conversely that you wish had become part of the standard but did not.

Incidentally, thank you for all of the insight so generously and regularly poured forth in comp.lang.lisp.

KMP: Well, people program with tools that are familiar to them. Unless Common Lisp is someone's first language, it'd be easy for them to overlook the things it contains that are not like the things they're familiar with. There's a certain irony here because often the reason people will leave a language for another language is that they've reached the limits of what they can do with the first language and they need more power. So you'd expect that they'd aggressively look for features of the new language that were different than the things they've used before. And probably some do. But you're right that others cling to the safety and familiarity of the operators they could just as well have found in the old language they left behind, and so in the process they miss out on what the language can offer them.

Fortunately, unwind-protect is finally (pardon the pun) present in Java. And some hints of the Common Lisp condition system made it into Java as well. So probably people who come to Common Lisp from Java will be inclined to seek out those capabilities. But there's a lot of other stuff there and I hope new users will indulge their curiosity and take the time to explore.

As to what we should have in the language, the main omission of note is some sort of system definition tool (the in-Lisp analog of make). It was a shame that we did a feature-freeze on ANSI Common Lisp in 1988 but didn't get the standard out until 1994, and the suggestion of including such a tool didn't come until the after-freeze period. All vendors offer such a facility, but programs would be more portable if there were a uniform solution.

There are also quite a number of things about Common Lisp that are available in the same or similar forms in nearly all implementations. Multi-tasking, sockets, database access, external function call, windowing, and so on. It wouldn't be bad to have included any of these, but the fact is that they weren't ready for standarization in 1988. At this point, though, I think other mechanisms than standards are the right way to proceed.

The Lisp community used to expect the delivery mechanism for new functionality to be a new language spec. But that requires working through consensus standards bodies. The problem is that, by their nature, standards bodies are synchronization mechanisms. The problem with synchronization mechanisms in a massively parallel world is that they slow things down. The world is not going to wait for us to slow down, so I think we need to evolve mechanisms that will keep up better with a degree of pace that is externally dictated.

I think this is an area where Lisp as a community has been slow to respond. There need to be community mechanisms for sharing the many great commercial and private packages people have been creating in Lisp, so that we can properly reap the cross-product benefits of our community's productivity. I see evidence that this is changing. The Common Lisp Open Code Collection (CLOCC) is one such mechanism that addresses open source code. I'd like to see similar mechanisms arise for the exchange of proprietary products as well.

As to my posts on the comp.lang.lisp newsgroup, thank you. I'm glad you enjoy them. Frankly, I always consider it a victory to hear I haven't bored everyone to death. In background I've been working on putting together several books on Lisp, but one never quite knows if one will finish such things. I regard comp.lang.lisp as a kind of insurance policy, assuring that at least some piece of what I have seen and done in my career gets transferred from individual memory to global group memory.

I think preserving individual experiences for history is quite important. In the future, this will happen naturally due to logs kept by online collaboration tools. But I'm especially worried about the records of what happened between about 1960 (the birth of programming languages) and 1994 (the birth of the web). Most of everything in that time range is recorded on paper and will eventually be lost. Looking back from the future, I expect it to be as confusing to figure out how the information society was born as it is to look back in a telescope to see the birth of the Universe. You'll get very close, but then you'll get to a point where you can see nothing. The informational big bang. I've been working on webbing all of my old hardcopy papers, and I hope others of that era will commit to doing the same.

14) Lisp - Scheme - ML
by Tom7

I know a lot of big academic (erstwhile) lisp shops, such as CMU, have transitioned away from lisp to ML [standardml.org] and relatives [inria.fr]. Some of the reasons we might give are:

• Sophisticated type systems, catching most bugs before your program is run, ensuring safety, etc.
• Much more efficient (http://www.bagley.org/~doug/shootout/craps.shtml), partly due to compilation strategies using types
• Increased modularity and abstraction
• Pattern matching, (subjectively) more natural syntax

In fact, I'm one of those people. I've been scoffed at by lisp fans, but most had never used ML. But I have an open mind, so, in the face of more "modern" languages, what advantages do lisp and scheme offer? Do you think that these advantages are fundamentally impossible to achieve in a typed setting?

KMP: First, I assume by "typed" you mean "statically typed." I think of Lisp as "dynamically typed." I think of most machine languages as "untyped." I've heard statically typed languages sometimes called strongly typed, and I sometimes use this terminology myself out of habit, but I've grown to dislike it because it seems to me that the issue of strength ought to refer to whether you check types, not when you do. The terms "static" and "dynamic" seem to me to better get to the heart of the matter.

To quote Abraham Lincoln, admittedly somewhat out of context, "People who like this sort of thing will find this the sort of thing they like." So to somewhat flippantly re-interpret Lincoln's remarks in a modern context, applying perhaps just a bit of obligatory political spin to the result: The fact that functional languages appeal to people who like functional languages is not a proof that functional languages are of general purpose appeal.

I think the real reason that CMU (or any university with a grant-based funding model) changed its direction is good sources of funding in research depend on saying you're doing something "new and different." Such a shift doesn't imply that the thing left behind wasn't "tried and true," but only that "tried and true" is not what gets research dollars. Research must constantly stir the mix, but that doesn't imply obsolescence to what came before. So don't read too much into that.

Answering each of your points in detail might require a whole article, but I'll touch on each in brief:

• Sophisticated type systems, catching most bugs before your program is run, ensuring safety, etc. Much more efficient partly due to compilation strategies using types.

  Actually, it's funny that you both mention the CMU project and then make this comment. Before moving away from Common Lisp, the CMU crowd was successful in demonstrating to the Lisp community's satisfaction that there were enormous opportunities offered by the Common Lisp language design in terms of type inferencing that still today go untapped by implementations. This is really a market issue, not a language design issue. The fact is that although other languages do a lot more type inferencing, vendors are not getting huge numbers of bug reports saying that better type inferencing is what stands between programmers and the commercial success of their product. Over time, I think you'll see more and more interesting type analysis done, but such work is always balanced against other needs of users, such as CORBA, COM, RMI, and web interfaces, for example, such as UI toolkits and debugging options. When I observe, as I often do, that languages are political parties, this is what I mean. They are each responsive not to the needs of the world, but to the needs of their constituencies. And the Lisp constituency, while it is not oblivious to the value of type inferencing, does not see that issue as its number one priority.

• Increased modularity and abstraction.

  This is quite a multidimensional space. I think Lisp provides great opportunities for modularity and abstraction that other languages do not. And yet, there are sometimes things I can't abstract as well as I wish. An example of a minor omission: Common Lisp's CLOS doesn't do protocol abstraction as well as Zetalisp's New Flavors; among other things, one can't declare that certain unimplemented methods are required. But with the use of the macro system and the Meta-Object Protocol (MOP), one can add this kind of thing. Further, the package system is missing certain kinds of inheritance capabilities I've often wished for, but I recently sat down and did the work of writing my own versions of defpackage for my own use, adding the capabilities I wanted in a way that my own tools can use, and I had no difficulty. For the most part, I've found the limitations of Common Lisp's abstraction capabilites to be incidental, and not deep, and I've found its syntactic reorganization capabilities more than capable of making up for it.

• Pattern matching.

  I think you're right that Lisp doesn't do pattern matching. Whether or not that's a good or bad thing is subjective. I think there are people who like pattern matching and people who don't. In fairness to Lisp, though, on the few occasions in my career where I've felt a strong need for pattern matching, I've been able to implement it easily. And, importantly, Lisp's syntactic adaptability has allowed me to make my personal implementation look as natural in the programs I write as if it were natively provided by the language; most other languages don't give me the syntactic control to be able to add new functionality in a way that feels appropriate to the language. So personally, I don't find this a strong negative; rather, I see it as an opportunity for you to create a layered library that supports the needs of yourself and others like you.

• (Subjectively) more natural syntax.

  I don't think you can make the case that much of any language has "natural" syntax. COBOL and HyperTalk gave this the fairest shot and there's a big difference even between them and any natural language. I personally find Lisp syntax remarkably natural in that it focuses on symbols that you could say out loud, marking them minimally to indicate grouping. Other languages contain lots of special-purpose markers like commas, semicolons, asterisks, and braces/brackets/parens that are used in quite nitpicky ways. All this to say that you're right on this one: it's subjective. And as such, I hope I can fairly dismiss this as an even draw.

15) Lisp in Mathematics Programming
by An Anonymous Coward

Gregory Chaitin has a book called "The Limits of Mathematics." In it he claims that mathematicians should love Lisp because Lisp is basically set theory, and all mathematicians love set theory. I wholeheartedly agree with this, one only needs to look at Chaitin's Lisp programs to realize how quickly and succinctly one can arrive at astonishing incompleteness results in mathematics. So we know Lisp is great for stuff like this, really researching a mathematical subject. Do you see Lisp continuing in this direction, showing and discovering theorems, or will it move into industry? Or has it moved into industry, and we just don't know it? Do the likes of NASA and JPL use Lisp and Scheme religiously? I would bet so.

KMP: Lisp may have started out as a way of addressing abstract topics like math (logic, calculus, prime numbers, etc.) and artificial intelligence, but it long ago made the transition to commercial applications. Both Scheme and Common Lisp have been and continue to be used in real-world applications that might surprise you. These include (but are certainly not limited to) applications in airline scheduling, commercial database manipulation, computer operating systems, bioengineering, web services, document format translation, and, yes, even space exploration. Franz, Inc. has created quite a nice page of Lisp success stories that I think expand on this much better than I could in the space I've allowed myself to answer this question. And speaking of NASA/JPL, they did a comparative study of Lisp vs Java and C++ that some might find interesting.

16) Scheme in CS
by An Anonymous Coward

It seems many of the more popular CS programs in the world use Scheme as a teaching language. A lot of times, students complain about this, saying they'd prefer to learn about C or another language that is considered "apt for industry." I used to be like this too, but have now discovered the error of my thinking. How have you convinced others that while the latest programs might not be written in Scheme, that it is worth a student's time to learn Scheme. Many seem stuck to the point that if they won't use it outside of school, they shouldn't learn it. How can we convince them otherwise, to become scholarly citizens instead of drones?

KMP: I think the thing to explain to a student is that the world is ever changing and that one cannot put ones eggs all in one basket. Furthermore, modern environments are often quite heterogeneous, with different languages and systems being used together cooperatively. Especially for a CS student, who often has the luxury of time that a person in the job world does not, I think it's worth taking time to learn as many different languages as possible. This not only exposes the students to alternate ways of thinking, but it also prepares the student to quickly change modes of thought or languages of expression later. Once on the job, one often can't afford the ramp-up time to learn a new language at the point it becomes necessary to use. Better to already know it and just have to "brush up".

One is much more likely to consider alternative approaches if one has a sense of what is involved in them; it's very easy to fear the unknown, even when the unknown might be of great help. So get to know as many things as you can while you can. Common Lisp and Scheme, which I regard as two very different languages, by the way, should definitely be among the things every student studies. But they should not be the only things the person studies. Like it or not, there is a lot the professional programmer needs to know to be really successful not just tomorrow, but for a lifetime.

As Oliver Wendell Holmes is often quoted as saying, "A mind stretched to a new idea never returns to its original dimensions." In order to stretch a student's mind, I recommend they make a list of "kinds of languages" and then learn as many different kinds as they can. Here are some that come to mind, though I'm sure others with different experience than me might reasonably contribute still others.

• A block-structured language, such as Algol or Pascal or Scheme.
• A line-oriented language, such as Fortran or BASIC.
• A rule-based logic language, such as Prolog or EMYCIN.
• An Englishy language such as HyperTalk or Cobol.
• A "stack" language such as PostScript.
• A "line noise" language (heavy emphasis on one-letter operators) like APL or Teco.
• A dynamic object-oriented language, such as Common Lisp or Smalltalk .
• A strongly-typed, statically analyzed language such as ML or Haskell.
• A dynamically-typed functional programming language, such as Scheme.
• A string or macro processing language, such as Perl or m4 or TeX or Teco.
• A database access language, such as SQL.
• An abstract high-level, assembly language, such as Java.
• A concrete high-level, assembly language, such as C.
• A low-level, traditional assembly language.
• A scripting language, such as Javascript.
• An interface-definition language such as Visual Basic, HyperTalk, or Javascript.
• A document-structuring language such as XSL or TeX.
• A language with a modern error system, such as Common Lisp or Dylan.
• A reflective/introspective language such as Common Lisp or Java.
• A symbolic programming language, such as Common Lisp or Scheme.
• A security-oriented language, such as Java or MOO.
• A language where both programs and data are fully persistent, such as MOO or HyperTalk.

I'm confident that if people make a sincere effort to learn each of these kinds of languages on their own terms, the Lisp community will get its share of converts. But further, I would hope that the people who come to the Lisp community bring with them the ideas of other communities so that we can incorporate, rather than hide from, the good ideas that we need in order to continually be moving forward.

17) A question for Kent
by MarkusQ

Do you have a maclisp manual I could borrow?

KMP:For those not familiar with Maclisp, it's a defunct dialect of Lisp that predated and strongly influenced the design of Common Lisp.

I've been working on webbing The Revised Maclisp Manual, which I had published on paper back in the early 1980's. It's not quite ready to go out yet, but should finally be ready sometime in the not terribly distant future. Probably a month or two. Watch the site maclisp.info for more information.

18) Open Implementations
by Martin Pomije

What is your opinion of the idea of Open Implementations from Gregor Kiczales? Do you think that his idea could help Lisp be more widely used?

You can see him giving a lecture about this idea here. [microsoft.com] The video is only available in Windows Media format on this site.

KMP: I hadn't seen Gregor Kiczales's talk on Open Implementations, so I enjoyed watching it. Thanks for the pointer!

The talk made me think back to various related ideas I've seen batted around for a long time, the earliest of which that I can recall is a short paper on something called "Capsules" (an object system where classes were allowed to have multiple implementations) by Richard Zippel back in the late 1970s or early 1980s at MIT. Often, especially in a university environment, people will make up such a concept, bat it around for a bit, and then go on to something else. There are some very interesting ideas there and I'm glad to see that they're being pursued seriously, especially by someone as thoughtful and talented as I know Gregor to be.

As a formalized area of study, this topic of "aspect-oriented programming" is new to me. It reuses some old ideas in new ways, and introduces some new ones along with it. I'm only just barely becoming conversant in the terminology, so I can't really speak to it from a theoretical point of view. But it looks promising. And from a practical point of view, I can note that I'm getting daily on-the-job training in it through my consulting relationship with The Software Smith. They're using Lisp as a vehicle to apply the principles of aspect-oriented programming, and the results they get are quite spectacular.

19) What was up with CLisp's "loop" form?
by Jayson

Did you can have anything to do or know who had anything to do with the "loop" form in Common Lisp? Why does it look and feel just like a FOR loop on C (from the Graham book):

(loop for x = 8 then (/x 2)
until (< x 1)
do (princ x))

This is one of by biggest minor nags about CLisp and I am very curious what was going through the committee's collective head. Didn't anybody balk at this enough to at least get the syntax cleaned up?

KMP: The example you cite is quite simplistic and if this were the only reason for using LOOP, we wouldn't have it. Lisp has a number of other iteration operators for doing simple loops like this. However, the reason for using LOOP is that it can represent much more complicated arrangements of iteration paths and collection techniques. I used to grumble a lot myself about how "un-Lispy" LOOP seemed, but over time I come to the belief that the benefits outweigh the costs. A loop like this:

(loop for x from 0
for y in some-list
when (good-result? y)
collect (list x y))

is easy to write and maintain, and much easier to explain than the equivalent, but more Lispy:

(do ((x 0 (+ x 1))
(y-list some-list (cdr y-list))
(result '()))
((null y)
(nreverse result))
(let ((y (first y-list)))
(when (good-result? y)
(push (list x y) result))))

The Common Lisp community likes to offer the traditional Lispy notations for places where they enhance readability, but we also offer alternative notations for situations where we've learned there's a call for it. We leave the choice of which style to use up to the individual taste of the programmer. Common Lisp is not a minimalist language offering only one way to do things or rigidly attempting to force people into a single programming paradigm.

By the way, this is a fundamental difference between the ANSI Common Lisp design philosophy and the Scheme design philosophy. The introduction to the Scheme specification states:

Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary. Scheme demonstrates that a very small number of rules for forming expressions, with no restrictions on how they are composed, suffice to form a practical and efficient programming language that is flexible enough to support most of the major programming paradigms in use today.

By contrast, the charter for X3J13, the group that designed ANSI Common Lisp, stated the following in the X3J13 charter:

It will codify existing practice, provide additional features to facilitate portability of code among diverse implementations, and establish normative Common Lisp programming practice. The committee will begin with the language described in Common Lisp: The Language by Guy L. Steele Jr. (Digital Press, 1984), which is the current de facto standard for Common Lisp. Whenever there is a proposal for the standard to differ from Common Lisp: The Language, the committee shall weigh both future costs of adopting (or not adopting) a change and costs of conversion of existing code. Aesthetic criteria shall be a subordinate consideration.

In other words, the Scheme community is a very conservative community that is highly focused on keeping its language specification as highly aesthetic and minimal in size as possible. By contrast, the Common Lisp community is an industrial standard that is concerned with messier issues of compatibility, portability, and commercial need; while the Common Lisp community cares about aesthetics, it does not allow aesthetics to dominate practicality as a design criterion.

The relevance of this here is that the Lisp family of languages is made up of a number of smaller communities who share a few core ideas, but really have some very divergent points of view. Each is worthy of study in its own right. One should not, having looked at Scheme, assume they have good intuitions about Common Lisp, nor vice versa.

A few weeks ago, you asked Kent M. Pitman about Lisp, Scheme, standards, and other things -- He's answered your questions below, at length. At such length, in fact, that only the first eleven of his answers are shown below -- expect more shortly! Thanks, Kent.

1) (just one thing (I) want to (know))?
by An Anonymous Coward

((
What (
(is) with (all)
) of (the) ()s?
)
Hmmm?
)

Kent M. Pitman: This question actually got scored down to -1 and marked as a troll question, but I fished it out of the barrel and restored it because everyone asks and I might as well confront the issue head-on.

Ironically it's non-Lisp languages that allow and encourage you to put ()'s in any place you want, as if there were no meaning to the introduction of gratuitous paren groups.

3+(2*5)+7 means the same thing in an algebraic language as does 3+2*5+7. In Lisp, we write:

(+ 3 (* 2 5) 7)

This shows you the structure and means you never have to learn obscure precedence rules that make expressions like -3! confusing in algebraic languages, where you must learn whether it means (-3)! or -(3!). In Lisp, the parens would show you immediately that (factorial -3) or (- (factorial 3)) was intended.

The thing I personally like about (+ (* 2 y) x) rather than 2*y+x is that it simplifies my editing. I'm a touch-typist and I use the emacs commands to go forward and backward over expressions, to swap expressions, and to delete expressions very heavily. And I don't have to reach for the mouse to manipulate large, complex expressions because they are paren-bounded. If I put the cursor at the head of 2*y+x and say "go forward an expression", ought this go forward over 2, 2*y, or 2*y+x? Having different editor commands to move across a sum, a product, etc. would be unwieldy. Yet without that, I don't see how the editor would know. In Lisp, there can't be any ambiguity because every sub-expression has its own start character, so a single notion of "the expression in front of the cursor" or "the expression after the cursor" suffices.

This, by the way, also answers the question of why we don't write foo(x) and instead write (foo x). In Lisp notation, foo is an expression. In the expression (foo x), it's a subexpression, so it's enclosed within it. Were it outside, a text editor would not be sure if foo(x) were one expression (a function call) or two expressions (the symbol foo followed by the list (x)). That would make going forward over 'one expression' ambiguous when at the start of foo(x). Should the cursor end up after the foo or after the (x)? In other words, The natural purpose of parentheses is to enclose things, so that's what Lisp uses them for. Avoiding ambiguity is critical to the writing of correct "keyboard macros" in Emacs, where I might interactively write a program to do a lot of code transformations quickly. In an algebraic language, such keyboard macros can be much harder to write robustly.

2) It's not just me is it?
by demo9orgon

After trying to "self-learn" lisp in the 80's I get this physical reaction to the word "lambda"...a cold sweat combined with the involuntary retraction of my testicles to a protected location in my abdomen (damn unpleasant shit)...I usually avoid that second one by mentally going through the mechanics of "hello world" in C, or any half-a-dozen other programming languages.

Lisp is one of those meta-languages you either learn or avoid. I write practical stuff all the time, daily in fact, and I've never had something that required the arcane stuff in LISP.

KMP: Actually, "hello world" in Lisp looks like this:

(defun hello-world ()
(write-line "Hello, World!"))

I don't know about you, but I find that pretty soothing.

And as to LAMBDA, one only needs use it when they find it useful. For example, after a while, one sometimes gets tired of writing a separate function where that function will only be used once, as in:

(defun sort-by-name (list)
(sort list #'name<))

(defun name<(name1 name2)
(or(string<(last-name name1) (last-name name2))
&nbsp(and (string= (last-name name1) (last-name name2))
(string< (first-name name1) (first-name name2)))))

so Lisp allows one to instead say:

(defun sort-by-name (list)
(sort list #'(lambda (name1 name2)
(or (string< (last-name name1) (last-name name2))
(and (string= (last-name name1) (last-name name2))
(string< (first-name name1) (first-name name2)))))))

Whether one actually does this is purely a personal preference. Some people like having separate named functions, some don't. Sometimes the separately named function might have a nonsensical name, though, and it's nicer not to have to invent a stupid name for a one-shot use.

Now, as to why it's called LAMBDA and not FUNCTION, that's just a piece of history. You get used to it. Toward that end, I'll offer a story that will perhaps help you put it in perspective:

Early in my not-yet career as a computer scientist, which is to say, while I was in high school, I lived in the Panama Canal Zone. Computers were not at all common there at the time. In fact, the place being entirely run by the US Government, there was some weird edict that said no one was allowed to own one so that they would all be centralized in the Comptroller's Office and not wasted in individual offices around the Zone. Our school had to bend the rules in order to get us a computer to study. So one thing I did while trying to learn about computers was to go downtown (out of the Canal Zone into Panama City, in the Republic of Panama) and visit a company there who did computer work. Of course, people there spoke Spanish, but fortunately I did, too. They showed me some of their code, and I was immediately struck by the fact that all the language keywords were in English.

"Doesn't that bother you?" I asked. But the person I was talking to was quite a thoughtful person and he immediately responded this way: "Do you know how to read music?" "A little," I said. "Have you seen the notations on music like forte, sotto voce, and so on?" I nodded. "Does it bother you that they are in Italian?" "No," I had to admit. His point was to make me see that it could be viewed as part of the charm and history of the notation. He was, perhaps, unusually forgiving. But this was in the late 1970s, when everyone who had access to computers was far too excited about just plain having them to care about subtle issues of whose culture got too much say in the design of a world-wide phenomenon.

So when today I look at the very few mysterious-looking terms like LAMBDA, CAR, and CDR that still linger untouched in modern Lisp's design, I think of them as I do those musical notations, conceptual links to a little piece of history that I'm just as happy not to see crushed by an overeager rush to regularize and homogenize the world--something the computer culture has done altogether too much of.

3) Interactively programmable applications
by divbyzero (divbyzero@hotmail.com)

One of the primary reasons why Scheme and Lisp interest me is that they are well suited for making applications interactively programmable at runtime (Scheme especially, due to its small size). This is far more flexible and useful than applications which are only extensible through heavyweight, precompiled plugins. Since the Slashdot readership tends to be made up of people who are comfortable with programatic interfaces (unlike the general computer-using public), why do we not see more such applications?

KMP: I think it's just an issue of education, formal and otherwise. Without being explicitly guided, some people will try out all kinds of ways to do things, or invent them where they're not present. But many others will simply do what they have been taught to do without exploring the alternatives.

In the past, everything was about speed. Every instruction was precious. The focus was entirely on "micro" efficiency. People would examine the cost of being able to redefine something (which sometimes involves as much as following pointer indirection), and if there was a cycle lost, the game was over. Today, hardware cache and prefetch architectures can often hide such costs anyway, but even if they couldn't, processors run so fast that one has time to worry not only about micro efficiency but also macro efficiency--that is, "running smart", not just "running fast", as a way of assuring total efficiency.

A lot of people identify Lisp as a language that is "just good for Artificial Intelligence (AI)". Certainly Lisp is good for AI. But saying it is just good for AI misses the point. Lisp doesn't do AI. Lisp is a programming language. AI researchers program AI, and often their language of choice has been and continues to be Lisp. But the important thing is that AI researchers have been banging on the door of Lisp implementors for years, demanding the introduction and tuning of the features and constructs they need in order to get their work done. Lisp hasn't become a mere AI toolbox as a result of that. Rather, it has become a robust tool for addressing the world's most complex and vexing problems. The Lisp community has a long experience with supporting "intelligent programming", and with doing so efficiently.

Lisp's biggest problem in the past is probably that it hit its commercial peak too early, in the mid 1980s, before most computational problems the world was confronting were big enough to need the power Lisp had to offer. Those were the days of MacWrite and MacPaint and Lotus 1-2-3, and it just didn't make any difference whether one used Lisp or C for those. But for better or worse, the world has grown up around us, and the important problems of the day are a lot more complex. I think Lisp has a lot more to offer to the world of today than it ever did in the past.

4) The standard process
by VP

As participant in the standardization process for Lisp, what are your thoughts on standards for programming languages? What would you like to see different in this process? And speaking of standards, what do you think about the RAND licensing issue and the W3C?

KMP: I think standards have served their time to provide a stable base for people to build on, but for the modern environment, they move way too slowly to keep pace with the speed of change in business. It took a long time to put the Common Lisp standard together. We began in 1986, finished work in 1994, and got the actual document to press just before the end of 1995. Getting community consensus on something that big really does take that long, and I think it was an exercise worth doing to create the stable base that we created, but for future evolution of the language, I think there needs to be another way with far less overhead.

I see standards as having two components: The first is to simply cast a name into concrete so that reference to that name will always have a clear meaning. The definition of ANSI Common Lisp, at least for 1994, is now permanently registered. Anyone who wants to can now conform to that definition and others will know exactly what they mean by that. The second component is to assert an informal consensus in the community that there is a single right way of doing things. This latter component may be useful for the foundation (to define the initial market space), but I'm not sure it's appropriate for the library level of the language.

For the base language, if 60% of the community wanted to do things one way and 40% another way, the 60% got to roll over the 40%, and 100% of the community was expected to do things in the way that won. But at the library level, if 60% want one library and 40% want another, I'd rather 100% of the community get what they want by having some people just do it one way and the rest of the people do it the other way. The Lisp community has not traditionally done things that way; they've sought consensus. The Scheme community has been even more conservative about this than the Common Lisp community, and as a result has even fewer standardized facilities than the Common Lisp community.

The Scheme community has moved to a more loose-knit approach to break the design deadlock brought on by the core language committee's consensus process through its Scheme Requests for Implementation (SRFI) process. The Common Lisp community hasn't got anything quite so organized yet, but I suspect will eventually evolve something similar.

As to the question of the W3C, I'm not a huge fan at the moment. At a prior employer, we had the opportunity to join, but the contract we'd have had to sign made it clear that votes among members were advisory only, and W3C itself could decide to override what people voted on. This, to me, is not a consensus body. Furthermore, although I think standards bodies like ANSI move in near glacial time, I don't think you can fix things by just shortening the times. True national and global consensus just takes time, and shortening timelines doesn't just make things move faster, it also disenfranchises people. While I use the existing HTML, CSS, XML, XSL, and other W3C guidelines, I don't feel they were created in a manner that I respect as proper consensus process. I think the process was insular and rushed.

Neither am I happy with the notion of processes involving Reasonable and Non-Discriminatory (RAND) fees being part of a standard; I think consensus standards should only involve royalty-free (RF) technologies. I think adherence to standards should not induce a baseline cost beyond the cost of creating the code so that the cost of compliance with standards can closely approach zero. If there is a profit to be made on the implementation of a standard, it should go to the implementor, not to a patent holder. Then again, while I'm a strong proponent of software copyright, I'm not at all a fan of software patents. Rather than seeing independent creation as infringement, I think independent creation should be contributory proof that an idea was more obvious than perhaps the patent office thought. I don't mind copyright because there are ways that one can demonstrate that one did not merely copy another's work, and independent creation is a defense.

5) Advice to Aspirants
by An Anonymous Coward

Kent, I am one of the lucky ones who programs professionally in Common Lisp. I certainly appreciate your hard work and the hard work of everyone else who helped to bring us the ANSI standard - which serves to reify much of the esoteric knowledge the Lisp community has developed in the many years since the language was born.

While I do not need to be sold on Lisp, I know many people who do not fully appreciate the power of the language. To a large degree, this is due to misconceptions about the language. Specifically, there seem to be a number of what I would call 'cultural misconceptions'. Because many people have never worked in a tightly interactive development environment with incremental compilation, language-level introspection, and real code/data equivalence (not to mention the differences between CLOS and what the rest of the world seems to have come to believe is the God-given definition of 'object-oriented' programming) - they don't really 'get' what makes Lisp so special and so powerful. More to the point, because the logistics of developing and deploying applications in Lisp is different than what the typical c/c++/perl/java developer knows, the hurdle to even investigating or considering Lisp as a real possibility seems unnecessarily high.

Could you talk a bit about how those who have a feeling that Lisp might help them with their hard problems could go about bootstrapping their way into finding out? How would you suggest getting started? What is a reasonable set of tools for experimentation, and where should a beginner start with the language? (The standard is a big document!) Also, could you give an example of the type of problem space and style of application delivery that demonstrates that Lisp is more practical than many seem to believe?

KMP: Well, one thing to note is that there's very little overhead to just downloading an implementation and diving in. Not only do the major commercial vendors like Xanalys and Franz offer high quality, no-cost trial versions of their proprietary software, but there are quite a number of free (non-proprietary) versions of Lisp as well. Information about these, as well as much other useful information about Lisp, can be found at the Association of Lisp Users (ALU) web site. I've also recently purchased common-lisp.info, which I plan to maintain as a repository for information about Common Lisp; the site doesn't have a large base of information yet, but it does have a list of the problem spaces in which you might consider using Lisp.

The ANSI Common Lisp standard, effectively available in webbed form as the Common Lisp HyperSpec, is indeed a big document (about 16MB and having about 108 kilohyperlinks downloadable). I think it's fairly readable as standards go. But you're right that it takes some work to get through and it wasn't really intended as a tutorial.

The ALU web site will also have pointers to books and online tutorials about Lisp. Books by Paul Graham and Peter Norvig on the subject are very highly regarded. I think there is always room for more, and I'm working on several, at least one of which I hope to complete in the not too distant future; feedback from you and others is useful to me in understanding what areas most urgently require treatment.

One resource that some people might find useful is an article I wrote called Accelerating Hindsight: Lisp as a Vehicle for Rapid Prototyping. This article is intended primarily for a Lisp programmer audience, to help them articulate some of the ideas you've asked about to others. It was not intended to be read by the audience you'd like to convince mainly because it appeals periodically to Lispy notation that might not be familiar to them, but it may still be of interest to the adventurous non-Lisp reader.

As your project becomes more sophisticated, and evolves from a personal toy to a real commercial product, it also doesn't hurt to ask an expert for help. My company offers consulting services that include helping companies manage the transition into Lisp. One of my major clients, The Software Smith approached me on just such a basis and the result has been very exciting both for me (getting to help them improve their system) and, I think, for them (getting to see more of how Lisp is supposed to be used). I don't want to turn this interview into a huge advertisement, but people can contact me for more information. If I'm either not competent to help you or am too busy to help you, there's a very good chance I can refer you to someone else who can help you.

6) Language feature trickle-down
by WillWare

I was a big Scheme/Lisp fan five or six years ago, but now I see most of my favorite Lisp-like language features available in Python, which is getting a huge amount of high-quality development mindshare these days. Some of the Lisp-ish features in Python that spring right to mind are functions as objects, closures, garbage collection, and dynamic-yet-strong typing, and convenient rapid-app development.

One needn't look far to find arguments that there is still something unique to Lisp that differentiates it even from very recent languages which have had ample opportunity to borrow from Lisp. But one rarely finds a really clear articulation of that uniqueness. Do you think concur with the view that Lisp is still unique, and if so, do you think that Lisp's putative advantage really is ineffable?

If there is an advantage but it's ineffable and therefore opaque to managers with purchasing power, that would explain why Franz, Harlequin, et al have had such a rocky road. Does the Lisp/Scheme community regard this as a worrisome issue? (Some folks on c.l.lisp clearly don't think so, but I don't know if they are just a noisy minority.)

KMP: I guess I think Lisp is unique, but whether it is or not doesn't affect its usefulness as a tool. I'll enumerate some things I like about Lisp, but Slashdot readers shouldn't assume that I'm asserting for each of these features that Lisp has a lock on these. Various other languages surely have some of these. But I am often heard to say: languages are ecologies. Language features are not a priori good or bad. Rather, language features are good or bad in context, based on how well they interact with other language features. Some of what makes Lisp what it is has to do with the features it offers, but some of what makes Lisp what it is has to do with how the features work together to make a coherent whole. Lifting some of these features out of context might sometimes work, but in other cases, it might not. To get a real feel for Lisp, or any language, I think you have to really use it.

Also, in my 1994 article Lambda, the Ultimate Political Party, I advance the hypothesis that languages are defined as much by their community as by their semantics. That is, languages are forever in flux, and the semantics you read about in a language spec is a point in a multi-dimensional space telling you the current location, but it does not tell you the velocity vector in that space. For that, you must look to the community. Even if two languages happened to occupy precisely the same point in design space, that is, if they had the same semantics, would they continue to over time? I think not.

For what it's worth, here are just some of the things I personally like about ANSI Common Lisp:

• Lisp is dynamic. The world is ever changing and it's useful to allow programs to change dynamically with it. I can load new or changed functions, classes, and method definitions into a running image that I'm debugging, or even in a deployed production application. When I do, the code that was running will immediately start using the new definitions. Classes can be redefined even if the new class has different slots, and, if I care to, I can control how the update is done from old to new slot arrangements for already-created instances. This kind of thing supports programs that must be continually running yet must be responsive to changes or even just bug fixes.

• Lisp is introspective. Not only can functions, packages, classes, methods be dynamically added, redefined, or removed, but programs can also inquire about whether aspects of the programming environment (functions, packages, classes, and so on) are defined, can manipulate those objects as data, can save them away, can transform or encapsulate them, etc. Also, the Lisp compiler is a standard part of the language and can be invoked even at runtime by applications that need to augment themselves. New programs can be created on the fly, then compiled and loaded and executed in the same running image as they were created, without ever exiting (and even without doing file I/O). This facilitates automatic programming and the development of layered languages.

• Lisp's syntax is malleable. There's nothing worse than being stuck in a syntax that you don't like in a language you're going to use for a long time. Lisp allows programmers to reconfigure the syntax rules for parsing characters into data and programs, as well as allowing macro technology that transforms one parsed program expression into another. And it allows control of how data is displayed during program execution and debugging. Moreover, this can generally be done in such a way that one programmer's customizations don't adversely impact another's. This makes interactions with Lisp more pleasant and debugging sessions more productive.

• Lisp doesn't force users to use variable type declarations in order to just get a program to run. The initial focus in Lisp is on getting programs working. You can add type declarations when you're done if you want to, in order to enable additional compiler optimizations. This facilitates rapid prototyping by first getting an application running quickly with low overhead, and then allowing an application to be tuned as a second pass operation.

• Lisp has a powerful class system, and a flexible meta-class system. The class system allows powerful slot and method definition, method combination, and a great many other detailed features. The meta-class system allows users to treat the object system as data that can be programmed, creating new kinds of classes.

• Lisp gives the user powerful tools for both signaling and handling errors. This means that when an error occurs, there are often a variety of ways to continue programs other than simply aborting or dumping core. Moreover, object-oriented error handling allows programs to represent errant situations, evaluate the options for how to proceed, and select an appropriate option under program control.

• Lisp uses automatic memory management. This means that when a programmer is done with an object, they just let go of it and the garbage collector reliably frees its storage. This means Lisp programs do not suffer from the memory leaks that commonly plague programmers in many other languages.

7) What will it take to make Lisp fashionable again?
by kfogel

For myself and a number of friends, Lisp/Scheme programming has for too long been a kind of mystical Eden, fading in our memories, from which we have been mostly banished in our professional lives. But we can still recall how it felt to work in a language able to shape itself to any pattern our minds might ask: coding was more interesting and more expressive, and the rate of increasing returns over time was tremendous, because fine-grained -- almost continuous -- abstraction was in the nature of the language. Life was just more fun, frankly.

Alas! In our jobs and even in our personal projects, we are often forced to use C, C++, Java, Perl, or Python -- not because we prefer to write in those languages, but for two much less satisfying reasons: first, everyone else knows those languages, so we'll get more developers with them. And second, you can't count on users and testers having the right environment to run programs written in Lisp/Scheme, so right away you take a portability hit if you choose to develop in them.

Do you think there is a chance of Lisp/Scheme becoming "mainstream" again? That is, when someone contemplates starting a project, it would be as realistic for them to consider Lisp or Scheme as, say, Perl, without worrying about losing developers or initial testers? What will it take?

KMP: First, let me say that I really appreciate the poetic description you offer in the first paragraph above. I very much think that captures how I and others think about the experience of using Lisp.

And as to the future of Lisp, I think the situation for Lisp is looking pretty upbeat these days. Enough so that my own infant business is building its tools in Lisp, both for sale and for our own internal use on products we produce.

There are a lot of implementations, both commercially maintained and "free", with a wide range of delivery options, from conventional executables to "remote" solutions: Some implementations support CORBA and/or COM interfaces, for example. Also, most implement some kind of sockets interface, and there are several Lisp-based web servers available that build on this. Lisp programs can dynamically load DLLs, or can be delivered as DLLs themselves. They can do "foreign function call" to functions in other languages. It can also communicate with databases, and so with other programs via databases.

As the world moves increasingly to high-bandwidth global connectivity, I think the issue of the delivery environment will become less important. People have been waiting for an e-Service based society to take off, and it hasn't quite done that yet, but I think it's coming. I can't see how it won't. The overall savings in quality assurance and support of not having to re-deploy an application in a hostile customer-premise environment will be a lot, just as your question implies. One will just bring an application up on the right kind of hardware, connect it to the net, and then forget about where the program is actually being used. That may be an oversimplification today, but I wouldn't waste my money betting against it for tomorrow.

8) Questions I've Come Across Learning Lisp
by Jon Howard

I was recently (April) hired-on as webmaster at Franz [franz.com], a commercial lisp company (we make Allegro Common Lisp [franz.com]) which has introduced me to lisp in a very loud way. Since joining these guys (and gals), I've been thoroughly indoctrinated - with my full consent - because of my belief that as computing hardware progresses programming in more abstract languages will allow for more creative and effective use of the platform. Sure, coding assembler on a new super-duper petaflop chip will still be possible and less wasteful, but who would want to code a million lines of asm to save a few (or even a few thousand) operations out of a few billion, or trillion when it will only net a difference of nanoseconds in the end? I'm less interested in making super-fast programs than I am in making artistic and super-functional programs.

I'm not expressing the views of Franz, every member of the company has their own beliefs on what makes for great programming - which is one of the major reasons I find this place so fulfilling, everyone has complex reasons for their design considerations, and everyone communicates them (something I've grown to appreciate from working in too many places where this was definitely not the case), and consequently I've been exposed to quite a few different techniques of Lisp coding since my introduction half a year ago. I'm constantly amazed that so many different styles of programming can be expressed in the same language, it's capable of accommodating any logical thought process that can be converted to code - and I doubt many of you often use recursion in a logical way on a daily basis, but even that can be done efficiently in lisp.

I'm still very new to lisp, and I was never a serious programmer in the past, but I've always been accustomed to asking questions, and here are a few that I'd like some input on:

• If you learned any other programming language, did you initially find the formalities of its structure to be a significant stumbling block to understanding the language as a whole? Was the same true of learning lisp?
• How much time do you spend debugging non-lisp code? How much on lisp?
• What language took you the most time to learn - was it your first?
• What feature do you consider to be the most important for an abstract language to support efficiently - and which features have you found to be most poorly implemented in lisp distributions?

I'd love to hear about what people think sucks about lisp and needs improvement - or can't be improved, so far I haven't found anything that I could complain about, the most difficult thing for me has been managing all the documentation on a half-century old language in the process of learning it. I've begun to love working in lisp, but I suppose being surrounded by a group so full of passion for it has helped contribute to my bias - if I'm wrong, help snap me out of it with a good argument against using lisp. ;)

KMP: I knew FORTRAN and Basic before I learned Lisp. And I've dealt with numerous languages of all kinds since learning Lisp. With most, the syntax itself is generally not a burden. Some languages have more pleasant syntaxes than others, but the human brain has an amazing ability to cope. Of all the many languages and syntaxes I've seen, about the only thing I've never been able to cope with is the "*" used to notate indirection in C. I understand thoroughly the notion of pointer indirection, and the difference between "pointer to array" and "array of pointers", but I find it forever hard to read and write that particular awful notation for some reason. Give me Teco or Perl any day.

Mostly, though, I think the issue of how hard a syntax makes it to learn a language is overblown. Humans have brains that are adapted to processing myriad special cases and can mostly cope with obscure syntaxes. The real issue is how hard it is for humans to pass on their knowledge to programs. People are good at judgment, and programs are good at repetition. Over time, though, judgment tasks become repetitive and it's time for programs to take them over. I like to write macros to package up things I do a lot, and the key to that is having a reliable mapping between program syntax and program structure. The last thing one wants is a macro language based on character syntax, since such syntax is too unpredictable. Lisp offers macros based on program structure, and that greatly reduces the number of programmer errors one makes in macro writing.

As to debugging, I try to use non-lisp code as little as possible because of how hard it is to debug. Most other languages don't have good visual representations of their data, so when I get in the debugger, the manner in which I am presented with errant data is usually low-level and hard to read. A great deal of my valuable time is spent painstakingly piecing structure back together. But in Lisp data objects have familiar visual representations and I find it's usually easier to see what has gone wrong.

What language took me the most time to learn? Probably Teco. There was a lot of trivia to learn there. What language took the least time? Probably FORTRAN, BASIC, Lisp, HyperTalk, and MOO. Fortran just because it was small. The others because they are highly interactive, which is a huge boon to learning.

Actually, I learned PostScript very fast, too. There are some excellent cookbooks on this. But I never learned to debug PostScript. When my programs erred, I mostly just wrote them anew and hoped they'd work then because debugging was too painful.

What do I consider it most important for an abstract language to support efficiently? My time. Time is the only true, non-renewable commodity. I eschew languages like C because they often waste enormous amounts of my time trying to develop and debug programs, and justify it on the basis of micro-differences in speed that have just never ended up mattering to me. I regard C as appropriate for use as an assembly language, but it doesn't provide enough high-level services for me. When I'm old and grey and look back on my life, I want to have done a lot of interesting things, not just have done a few interesting things but "boy were they fast".

I think it's important to pick a language not on the basis of how fast its implementations are today, but on the basis of how much they do what you want. Lisp has an undeserved reputation for being slow, which I think results from deciding to make it do things that there are not always known optimizations for at the outset. Like garbage collection. But as Lisp is used, people complain about the things that are slow, and fixes get found. So Lisp moves ahead. If Lisp had started instead only with the things it knew how to implement efficiently, it would be holding things back. I want my ideas to lead my technology and my tools, not to have my technology and tools leading my ideas.

9) Basis set for programming languages?
by PseudonymousCoward

As a Scheme and Common Lisp programmer, I got excited when I heard that the Java Virtual Machine would have automatic memory allocation and garbage collection. I thought it would be possible to build Lispish languages to run on the JVM. The rate at which Kawa has been developed, to implement a near-Scheme on the JVM has been frustrating to me. I attribute this at least in part to the absence in the JVM of a construct equivalent to Scheme's continuations. Do you think it is feasible to establish a "basis set" of programming language concepts on which all programming languages could be built, so that the distinctions between C, Scheme, etc would be "merely" syntactic? If yes, please enumerate your candidate set.

KMP: Well, continuations are just functions. What's really lacking to make this easier is good tail call support so that continuations can be called correctly without pushing stack.

I don't really have personal experience with using the JVM directly, but my experience with the MOO programming language led me to believe that there might be a problem with integrating tail calling and security, since sometimes security is implemented by asking "who called me?" and tail calls can mean that the apparent caller is not the real caller. So I asked my spies at Sun about this.

I'm told that the original security model for Java worked the way I expected (by examining the call chain), and that concern over consequent security matters contributed to the absence of tail calling support in early releases. But apparently it was conceded a long time ago that such support should be added some day, and that day simply hasn't come yet. So perhaps there is hope.

Even so, I'm not so sure no matter how hard you try that you can just paper over the many differences between languages and say that the only remaining issues are ones of syntax. I do think you can probably get to a point where all languages can compile to this machine, but that may not always mean that programs in one language are as efficient as those in another, or that data structures in one language are as naturally represented as those in another. For example, both Lisp and Scheme assume that small integers (that would fit in a machine number) are still integers; they don't have the int/Integer disjointness that Java has. A Lisp-to-JVM compiler could presumably hide this distinction, but it would be wrong to say that the only difference between Java and Lisp was syntax--there are really some material philosophical disagreements between the two languages.

10) Scheme as an XML Translation Language
by Evangelion

I've become fairly interested lately in using Scheme (probably mzscheme) and the SXML package as a way to do arbitrary XML translations in my free time (if I had any).

From the looks of it, the ability to create a reflexive mapping between an arbitrary XML document and an interpretable programming language is too powerful to be ignored.

Do you think that in the future one of the primary roles of Scheme/Lisp is going to be in manipulation of XML documents, or is this going to be relegated as an academic curiosity while the world struggles through parsing XML in Java?

KMP: Are those my only two choices? The second one sounds awfully bleak. I'd better choose the former.

I don't know whether you'll see XML as a formal part of either Lisp or Scheme any time in the near future, but a lot of that is because the standards bodies administering these are not extraordinarily active at this time. That doesn't mean the languages are dead, just stable. Ongoing work is mostly happening at the level of libraries, and such libraries can generally be written by anyone using existing primitives, without modifications to the core language.

Lisp manipulation of XML and HTML is something people have been working on for a long time. For example, the Document Style Semantics and Specification Language (DSSSL) was a purely functional, side-effect free variant of Scheme. Even XSL, the apparent replacement to DSSSL, offers the same kind of functionality. It just uses a more CSS-like page model and XML syntax. But, conceptually, it's Scheme inside.

In my recent professional life, I have personally written several XML parsers, all in Lisp, for various employers and most recently for myself and my fledgling company. My company's implementation is not available on the market yet, but when it is, I'm quite sure the chief competition will not be around the availability of mere "availability". Already there are a variety of libraries related to XML, XSL, and SAX floating around. And I'm quite sure there will be more to come. Competition will be over things like efficiency, robustness, representation, and optional additional features.

11) Lisp vs. the world
by hjs

What do you see as the unique strengths and weaknesses of Lisp?

What strengths does it specifically have over other functional languages (such as ML), over structured languages (such as C, Algol, etc), over object oriented languages (such as C++, smalltalk, simula, etc), and over scripting languages (such as TCL, perl, etc)? Can these other languages or classes of languages be enhanced to include these strengths? If so, how, and if not, why?

What about weaknesses? What do you see as the weaknesses of Lisp, both in general and in comparison to the above classes of languages? Can these weaknesses be eliminated? If so, how and if not, why?

I mean strengths and weaknesses not only in the formal sense of the language itself, but also in terms of its usability in today's world. For example, difficulty in delivering binaries or lack of accessibility of system libraries from within common implementations of a language would be considered weaknesses.

KMP: There are so many things I like about Lisp, but most of them come under the heading of "doing things in the right order."

For example, type declarations in many languages are required but in Lisp they're optional. I prefer to first get my program working, and only then to tune it to be more efficient by adding type declarations. What's the point of doing a lot of make-work declarations if you're not even sure you're going to keep the result? I do a lot of exploratory programming just to answer "what if" questions. I also write lots of little throwaway programs just to compute a simple result. I don't need such programs to run in 5 microseconds instead of 10.

I also view the process of programming as a series of "times" at which decisions can be made: "coding time," "parsing time" (Lisp calls this "read time"), "macro expansion time," "compilation time," "load time," and "execution time." Lisp gives me a great deal more control for each piece of code as to when it runs, so that it can run at the appropriate time when the data it depends on is known. Other languages, especially statically typed ones, often make me specify information too soon, before it is really known, which usually means "making up" answers instead of really knowing the answers. Sometimes that makes programs run faster. Sometimes it just makes them run wrong.

And I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not important that the representation be the Right® choice. It's just important that it be a common, agreed-upon choice so that there can be a rich community of program-manipulating programs that "do trade" in this common representation.

I write a lot of macros because there are a lot of interesting things one can do with macros in Lisp. In other languages, macro-writing is a process of manipulating strings containing input syntax. That feels very unreliable and I've never liked that. Lisp's willingness to represent its code in known data structures makes macro writing feel a lot more reliable. And the presence of macros in Lisp generally means that the boring parts of coding get removed, because repetitive patterns usually get captured by a macro and hidden away, keeping the developer's attention on the "interesting parts", and making the activity of programming itself both more fun and more efficient.

Could other languages borrow some of Lisp's strengths? Sure. And they do. Java, Dylan, and I suspect even C++ have all borrowed ideas from Lisp. But that's ok. We'll make more. And anyway, it's not a zero sum game. Everyone benefits when there's this kind of cross-pollination, whether it's Lisp influencing other languages or vice versa.

Weaknesses of the language? Well, that's harder to say. I think the basic design is quite strong. Sometimes you see an implementation that has put more energy into some parts of the language than others, but usually that has created a market opportunity for another, so overall we have our bases covered.

For example, you might find some implementations that have big "hello world" footprint sizes compared to "hello world" in other languages. Some in the Lisp community, don't think this matters much, because disk and RAM are getting ever cheaper. "Real" applications (i.e., not "hello world," but something meaty) of 5-10 megabytes are pretty commonplace these days. Years ago, Lisp used to be seen as large, but due to such criticism, Lisp has held its size constant in the last decade while other languages and systems have bloated rapidly. So nowadays, Lisp is comparatively quite small. And even still, if you don't like the size you get from one vendor, it seems there's always another trying to squeeze into the niche of addressing your need. Corman Common Lisp (an up-and-coming commercial implementation) and CLISP (a GPL-style "free" implementation) have given special attention to this issue. So there's a vendor for everyone on the size issue. And, though I deal more often in Common Lisp in my day-to-day work these days, I would be remiss if I didn't mention that image size is also a key concern of the Scheme language community, so that's yet another way the size issue is addressed for those who see it as critical.

Some might have heard that Lisp, being dynamic, doesn't make use of static type information. This isn't quite right. In fact, the language doesn't require static type analysis, it merely permits it. This gives a lot of leeway to each implementation to address the specific needs of its own customer base. The CMU Common Lisp implementation has, for example, addressed the issue of type analysis in great detail and offered a clear demonstration that there are many exciting things that implementations of Common Lisp can do with type declarations if they choose to.

Why don't all implementations optimize all of these aspects--footprint size, static type analysis, etc.? The Common Lisp language is admittedly conceptually large and correct, efficient compilation requires considerable time and cleverness to implement. "Why not make the language smaller so it requires less work to implement?" is a query you hear a lot from the outside, and even from members of the Scheme community. The answer from the Common Lisp community amounts to this: Programs are written all the time, but implementations are written much more rarely. What the implementation does not do is left for the user. The more hard work the language does, the less hard work programs do. In effect, the thesis of Common Lisp is that bigger languages make for smaller sentences in the language. (To see that there is at least some intuitive basis for this, think about how long a novel like Gone With the Wind is in English, then try to imagine whether the same novel re-expressed in Esperanto would be longer or shorter.)

If a language offers only what a programmer could implement overnight, it gives its programmers not much of a leg up on their final application. Many members of the Scheme community boast that they have written a Scheme implementation, while many Common Lisp programmers have not. Common Lisp is surely harder to implement, but the Common Lisp community does not see as its primary purpose to put out legions of implementors, each with their own easily-created implementation. The Common Lisp community has chosen to be about commercial applications, and its designers have provided a "meaty chunk" of useful power for programmers to use, with the promise that if programmers write their programs to that standard, not only will those programs work well today, but as implementations get better, those same programs will work even better tomorrow.

[to be continued...]

A few weeks ago, you asked Kent M. Pitman about Lisp, Scheme, standards, and other things -- He's answered your questions below, at length. At such length, in fact, that only the first eleven of his answers are shown below -- expect more shortly! Thanks, Kent.

1) (just one thing (I) want to (know))?
by An Anonymous Coward

((
What (
(is) with (all)
) of (the) ()s?
)
Hmmm?
)

Kent M. Pitman: This question actually got scored down to -1 and marked as a troll question, but I fished it out of the barrel and restored it because everyone asks and I might as well confront the issue head-on.

Ironically it's non-Lisp languages that allow and encourage you to put ()'s in any place you want, as if there were no meaning to the introduction of gratuitous paren groups.

3+(2*5)+7 means the same thing in an algebraic language as does 3+2*5+7. In Lisp, we write:

(+ 3 (* 2 5) 7)

This shows you the structure and means you never have to learn obscure precedence rules that make expressions like -3! confusing in algebraic languages, where you must learn whether it means (-3)! or -(3!). In Lisp, the parens would show you immediately that (factorial -3) or (- (factorial 3)) was intended.

The thing I personally like about (+ (* 2 y) x) rather than 2*y+x is that it simplifies my editing. I'm a touch-typist and I use the emacs commands to go forward and backward over expressions, to swap expressions, and to delete expressions very heavily. And I don't have to reach for the mouse to manipulate large, complex expressions because they are paren-bounded. If I put the cursor at the head of 2*y+x and say "go forward an expression", ought this go forward over 2, 2*y, or 2*y+x? Having different editor commands to move across a sum, a product, etc. would be unwieldy. Yet without that, I don't see how the editor would know. In Lisp, there can't be any ambiguity because every sub-expression has its own start character, so a single notion of "the expression in front of the cursor" or "the expression after the cursor" suffices.

This, by the way, also answers the question of why we don't write foo(x) and instead write (foo x). In Lisp notation, foo is an expression. In the expression (foo x), it's a subexpression, so it's enclosed within it. Were it outside, a text editor would not be sure if foo(x) were one expression (a function call) or two expressions (the symbol foo followed by the list (x)). That would make going forward over 'one expression' ambiguous when at the start of foo(x). Should the cursor end up after the foo or after the (x)? In other words, The natural purpose of parentheses is to enclose things, so that's what Lisp uses them for. Avoiding ambiguity is critical to the writing of correct "keyboard macros" in Emacs, where I might interactively write a program to do a lot of code transformations quickly. In an algebraic language, such keyboard macros can be much harder to write robustly.

2) It's not just me is it?
by demo9orgon

After trying to "self-learn" lisp in the 80's I get this physical reaction to the word "lambda"...a cold sweat combined with the involuntary retraction of my testicles to a protected location in my abdomen (damn unpleasant shit)...I usually avoid that second one by mentally going through the mechanics of "hello world" in C, or any half-a-dozen other programming languages.

Lisp is one of those meta-languages you either learn or avoid. I write practical stuff all the time, daily in fact, and I've never had something that required the arcane stuff in LISP.

KMP: Actually, "hello world" in Lisp looks like this:

(defun hello-world ()
(write-line "Hello, World!"))

I don't know about you, but I find that pretty soothing.

And as to LAMBDA, one only needs use it when they find it useful. For example, after a while, one sometimes gets tired of writing a separate function where that function will only be used once, as in:

(defun sort-by-name (list)
(sort list #'name<))

(defun name<(name1 name2)
(or(string<(last-name name1) (last-name name2))
&nbsp(and (string= (last-name name1) (last-name name2))
(string< (first-name name1) (first-name name2)))))

so Lisp allows one to instead say:

(defun sort-by-name (list)
(sort list #'(lambda (name1 name2)
(or (string< (last-name name1) (last-name name2))
(and (string= (last-name name1) (last-name name2))
(string< (first-name name1) (first-name name2)))))))

Whether one actually does this is purely a personal preference. Some people like having separate named functions, some don't. Sometimes the separately named function might have a nonsensical name, though, and it's nicer not to have to invent a stupid name for a one-shot use.

Now, as to why it's called LAMBDA and not FUNCTION, that's just a piece of history. You get used to it. Toward that end, I'll offer a story that will perhaps help you put it in perspective:

Early in my not-yet career as a computer scientist, which is to say, while I was in high school, I lived in the Panama Canal Zone. Computers were not at all common there at the time. In fact, the place being entirely run by the US Government, there was some weird edict that said no one was allowed to own one so that they would all be centralized in the Comptroller's Office and not wasted in individual offices around the Zone. Our school had to bend the rules in order to get us a computer to study. So one thing I did while trying to learn about computers was to go downtown (out of the Canal Zone into Panama City, in the Republic of Panama) and visit a company there who did computer work. Of course, people there spoke Spanish, but fortunately I did, too. They showed me some of their code, and I was immediately struck by the fact that all the language keywords were in English.

"Doesn't that bother you?" I asked. But the person I was talking to was quite a thoughtful person and he immediately responded this way: "Do you know how to read music?" "A little," I said. "Have you seen the notations on music like forte, sotto voce, and so on?" I nodded. "Does it bother you that they are in Italian?" "No," I had to admit. His point was to make me see that it could be viewed as part of the charm and history of the notation. He was, perhaps, unusually forgiving. But this was in the late 1970s, when everyone who had access to computers was far too excited about just plain having them to care about subtle issues of whose culture got too much say in the design of a world-wide phenomenon.

So when today I look at the very few mysterious-looking terms like LAMBDA, CAR, and CDR that still linger untouched in modern Lisp's design, I think of them as I do those musical notations, conceptual links to a little piece of history that I'm just as happy not to see crushed by an overeager rush to regularize and homogenize the world--something the computer culture has done altogether too much of.

3) Interactively programmable applications
by divbyzero (divbyzero@hotmail.com)

One of the primary reasons why Scheme and Lisp interest me is that they are well suited for making applications interactively programmable at runtime (Scheme especially, due to its small size). This is far more flexible and useful than applications which are only extensible through heavyweight, precompiled plugins. Since the Slashdot readership tends to be made up of people who are comfortable with programatic interfaces (unlike the general computer-using public), why do we not see more such applications?

KMP: I think it's just an issue of education, formal and otherwise. Without being explicitly guided, some people will try out all kinds of ways to do things, or invent them where they're not present. But many others will simply do what they have been taught to do without exploring the alternatives.

In the past, everything was about speed. Every instruction was precious. The focus was entirely on "micro" efficiency. People would examine the cost of being able to redefine something (which sometimes involves as much as following pointer indirection), and if there was a cycle lost, the game was over. Today, hardware cache and prefetch architectures can often hide such costs anyway, but even if they couldn't, processors run so fast that one has time to worry not only about micro efficiency but also macro efficiency--that is, "running smart", not just "running fast", as a way of assuring total efficiency.

A lot of people identify Lisp as a language that is "just good for Artificial Intelligence (AI)". Certainly Lisp is good for AI. But saying it is just good for AI misses the point. Lisp doesn't do AI. Lisp is a programming language. AI researchers program AI, and often their language of choice has been and continues to be Lisp. But the important thing is that AI researchers have been banging on the door of Lisp implementors for years, demanding the introduction and tuning of the features and constructs they need in order to get their work done. Lisp hasn't become a mere AI toolbox as a result of that. Rather, it has become a robust tool for addressing the world's most complex and vexing problems. The Lisp community has a long experience with supporting "intelligent programming", and with doing so efficiently.

Lisp's biggest problem in the past is probably that it hit its commercial peak too early, in the mid 1980s, before most computational problems the world was confronting were big enough to need the power Lisp had to offer. Those were the days of MacWrite and MacPaint and Lotus 1-2-3, and it just didn't make any difference whether one used Lisp or C for those. But for better or worse, the world has grown up around us, and the important problems of the day are a lot more complex. I think Lisp has a lot more to offer to the world of today than it ever did in the past.

4) The standard process
by VP

As participant in the standardization process for Lisp, what are your thoughts on standards for programming languages? What would you like to see different in this process? And speaking of standards, what do you think about the RAND licensing issue and the W3C?

KMP: I think standards have served their time to provide a stable base for people to build on, but for the modern environment, they move way too slowly to keep pace with the speed of change in business. It took a long time to put the Common Lisp standard together. We began in 1986, finished work in 1994, and got the actual document to press just before the end of 1995. Getting community consensus on something that big really does take that long, and I think it was an exercise worth doing to create the stable base that we created, but for future evolution of the language, I think there needs to be another way with far less overhead.

I see standards as having two components: The first is to simply cast a name into concrete so that reference to that name will always have a clear meaning. The definition of ANSI Common Lisp, at least for 1994, is now permanently registered. Anyone who wants to can now conform to that definition and others will know exactly what they mean by that. The second component is to assert an informal consensus in the community that there is a single right way of doing things. This latter component may be useful for the foundation (to define the initial market space), but I'm not sure it's appropriate for the library level of the language.

For the base language, if 60% of the community wanted to do things one way and 40% another way, the 60% got to roll over the 40%, and 100% of the community was expected to do things in the way that won. But at the library level, if 60% want one library and 40% want another, I'd rather 100% of the community get what they want by having some people just do it one way and the rest of the people do it the other way. The Lisp community has not traditionally done things that way; they've sought consensus. The Scheme community has been even more conservative about this than the Common Lisp community, and as a result has even fewer standardized facilities than the Common Lisp community.

The Scheme community has moved to a more loose-knit approach to break the design deadlock brought on by the core language committee's consensus process through its Scheme Requests for Implementation (SRFI) process. The Common Lisp community hasn't got anything quite so organized yet, but I suspect will eventually evolve something similar.

As to the question of the W3C, I'm not a huge fan at the moment. At a prior employer, we had the opportunity to join, but the contract we'd have had to sign made it clear that votes among members were advisory only, and W3C itself could decide to override what people voted on. This, to me, is not a consensus body. Furthermore, although I think standards bodies like ANSI move in near glacial time, I don't think you can fix things by just shortening the times. True national and global consensus just takes time, and shortening timelines doesn't just make things move faster, it also disenfranchises people. While I use the existing HTML, CSS, XML, XSL, and other W3C guidelines, I don't feel they were created in a manner that I respect as proper consensus process. I think the process was insular and rushed.

Neither am I happy with the notion of processes involving Reasonable and Non-Discriminatory (RAND) fees being part of a standard; I think consensus standards should only involve royalty-free (RF) technologies. I think adherence to standards should not induce a baseline cost beyond the cost of creating the code so that the cost of compliance with standards can closely approach zero. If there is a profit to be made on the implementation of a standard, it should go to the implementor, not to a patent holder. Then again, while I'm a strong proponent of software copyright, I'm not at all a fan of software patents. Rather than seeing independent creation as infringement, I think independent creation should be contributory proof that an idea was more obvious than perhaps the patent office thought. I don't mind copyright because there are ways that one can demonstrate that one did not merely copy another's work, and independent creation is a defense.

5) Advice to Aspirants
by An Anonymous Coward

Kent, I am one of the lucky ones who programs professionally in Common Lisp. I certainly appreciate your hard work and the hard work of everyone else who helped to bring us the ANSI standard - which serves to reify much of the esoteric knowledge the Lisp community has developed in the many years since the language was born.

While I do not need to be sold on Lisp, I know many people who do not fully appreciate the power of the language. To a large degree, this is due to misconceptions about the language. Specifically, there seem to be a number of what I would call 'cultural misconceptions'. Because many people have never worked in a tightly interactive development environment with incremental compilation, language-level introspection, and real code/data equivalence (not to mention the differences between CLOS and what the rest of the world seems to have come to believe is the God-given definition of 'object-oriented' programming) - they don't really 'get' what makes Lisp so special and so powerful. More to the point, because the logistics of developing and deploying applications in Lisp is different than what the typical c/c++/perl/java developer knows, the hurdle to even investigating or considering Lisp as a real possibility seems unnecessarily high.

Could you talk a bit about how those who have a feeling that Lisp might help them with their hard problems could go about bootstrapping their way into finding out? How would you suggest getting started? What is a reasonable set of tools for experimentation, and where should a beginner start with the language? (The standard is a big document!) Also, could you give an example of the type of problem space and style of application delivery that demonstrates that Lisp is more practical than many seem to believe?

KMP: Well, one thing to note is that there's very little overhead to just downloading an implementation and diving in. Not only do the major commercial vendors like Xanalys and Franz offer high quality, no-cost trial versions of their proprietary software, but there are quite a number of free (non-proprietary) versions of Lisp as well. Information about these, as well as much other useful information about Lisp, can be found at the Association of Lisp Users (ALU) web site. I've also recently purchased common-lisp.info, which I plan to maintain as a repository for information about Common Lisp; the site doesn't have a large base of information yet, but it does have a list of the problem spaces in which you might consider using Lisp.

The ANSI Common Lisp standard, effectively available in webbed form as the Common Lisp HyperSpec, is indeed a big document (about 16MB and having about 108 kilohyperlinks downloadable). I think it's fairly readable as standards go. But you're right that it takes some work to get through and it wasn't really intended as a tutorial.

The ALU web site will also have pointers to books and online tutorials about Lisp. Books by Paul Graham and Peter Norvig on the subject are very highly regarded. I think there is always room for more, and I'm working on several, at least one of which I hope to complete in the not too distant future; feedback from you and others is useful to me in understanding what areas most urgently require treatment.

One resource that some people might find useful is an article I wrote called Accelerating Hindsight: Lisp as a Vehicle for Rapid Prototyping. This article is intended primarily for a Lisp programmer audience, to help them articulate some of the ideas you've asked about to others. It was not intended to be read by the audience you'd like to convince mainly because it appeals periodically to Lispy notation that might not be familiar to them, but it may still be of interest to the adventurous non-Lisp reader.

As your project becomes more sophisticated, and evolves from a personal toy to a real commercial product, it also doesn't hurt to ask an expert for help. My company offers consulting services that include helping companies manage the transition into Lisp. One of my major clients, The Software Smith approached me on just such a basis and the result has been very exciting both for me (getting to help them improve their system) and, I think, for them (getting to see more of how Lisp is supposed to be used). I don't want to turn this interview into a huge advertisement, but people can contact me for more information. If I'm either not competent to help you or am too busy to help you, there's a very good chance I can refer you to someone else who can help you.

6) Language feature trickle-down
by WillWare

I was a big Scheme/Lisp fan five or six years ago, but now I see most of my favorite Lisp-like language features available in Python, which is getting a huge amount of high-quality development mindshare these days. Some of the Lisp-ish features in Python that spring right to mind are functions as objects, closures, garbage collection, and dynamic-yet-strong typing, and convenient rapid-app development.

One needn't look far to find arguments that there is still something unique to Lisp that differentiates it even from very recent languages which have had ample opportunity to borrow from Lisp. But one rarely finds a really clear articulation of that uniqueness. Do you think concur with the view that Lisp is still unique, and if so, do you think that Lisp's putative advantage really is ineffable?

If there is an advantage but it's ineffable and therefore opaque to managers with purchasing power, that would explain why Franz, Harlequin, et al have had such a rocky road. Does the Lisp/Scheme community regard this as a worrisome issue? (Some folks on c.l.lisp clearly don't think so, but I don't know if they are just a noisy minority.)

KMP: I guess I think Lisp is unique, but whether it is or not doesn't affect its usefulness as a tool. I'll enumerate some things I like about Lisp, but Slashdot readers shouldn't assume that I'm asserting for each of these features that Lisp has a lock on these. Various other languages surely have some of these. But I am often heard to say: languages are ecologies. Language features are not a priori good or bad. Rather, language features are good or bad in context, based on how well they interact with other language features. Some of what makes Lisp what it is has to do with the features it offers, but some of what makes Lisp what it is has to do with how the features work together to make a coherent whole. Lifting some of these features out of context might sometimes work, but in other cases, it might not. To get a real feel for Lisp, or any language, I think you have to really use it.

Also, in my 1994 article Lambda, the Ultimate Political Party, I advance the hypothesis that languages are defined as much by their community as by their semantics. That is, languages are forever in flux, and the semantics you read about in a language spec is a point in a multi-dimensional space telling you the current location, but it does not tell you the velocity vector in that space. For that, you must look to the community. Even if two languages happened to occupy precisely the same point in design space, that is, if they had the same semantics, would they continue to over time? I think not.

For what it's worth, here are just some of the things I personally like about ANSI Common Lisp:

• Lisp is dynamic. The world is ever changing and it's useful to allow programs to change dynamically with it. I can load new or changed functions, classes, and method definitions into a running image that I'm debugging, or even in a deployed production application. When I do, the code that was running will immediately start using the new definitions. Classes can be redefined even if the new class has different slots, and, if I care to, I can control how the update is done from old to new slot arrangements for already-created instances. This kind of thing supports programs that must be continually running yet must be responsive to changes or even just bug fixes.

• Lisp is introspective. Not only can functions, packages, classes, methods be dynamically added, redefined, or removed, but programs can also inquire about whether aspects of the programming environment (functions, packages, classes, and so on) are defined, can manipulate those objects as data, can save them away, can transform or encapsulate them, etc. Also, the Lisp compiler is a standard part of the language and can be invoked even at runtime by applications that need to augment themselves. New programs can be created on the fly, then compiled and loaded and executed in the same running image as they were created, without ever exiting (and even without doing file I/O). This facilitates automatic programming and the development of layered languages.

• Lisp's syntax is malleable. There's nothing worse than being stuck in a syntax that you don't like in a language you're going to use for a long time. Lisp allows programmers to reconfigure the syntax rules for parsing characters into data and programs, as well as allowing macro technology that transforms one parsed program expression into another. And it allows control of how data is displayed during program execution and debugging. Moreover, this can generally be done in such a way that one programmer's customizations don't adversely impact another's. This makes interactions with Lisp more pleasant and debugging sessions more productive.

• Lisp doesn't force users to use variable type declarations in order to just get a program to run. The initial focus in Lisp is on getting programs working. You can add type declarations when you're done if you want to, in order to enable additional compiler optimizations. This facilitates rapid prototyping by first getting an application running quickly with low overhead, and then allowing an application to be tuned as a second pass operation.

• Lisp has a powerful class system, and a flexible meta-class system. The class system allows powerful slot and method definition, method combination, and a great many other detailed features. The meta-class system allows users to treat the object system as data that can be programmed, creating new kinds of classes.

• Lisp gives the user powerful tools for both signaling and handling errors. This means that when an error occurs, there are often a variety of ways to continue programs other than simply aborting or dumping core. Moreover, object-oriented error handling allows programs to represent errant situations, evaluate the options for how to proceed, and select an appropriate option under program control.

• Lisp uses automatic memory management. This means that when a programmer is done with an object, they just let go of it and the garbage collector reliably frees its storage. This means Lisp programs do not suffer from the memory leaks that commonly plague programmers in many other languages.

7) What will it take to make Lisp fashionable again?
by kfogel

For myself and a number of friends, Lisp/Scheme programming has for too long been a kind of mystical Eden, fading in our memories, from which we have been mostly banished in our professional lives. But we can still recall how it felt to work in a language able to shape itself to any pattern our minds might ask: coding was more interesting and more expressive, and the rate of increasing returns over time was tremendous, because fine-grained -- almost continuous -- abstraction was in the nature of the language. Life was just more fun, frankly.

Alas! In our jobs and even in our personal projects, we are often forced to use C, C++, Java, Perl, or Python -- not because we prefer to write in those languages, but for two much less satisfying reasons: first, everyone else knows those languages, so we'll get more developers with them. And second, you can't count on users and testers having the right environment to run programs written in Lisp/Scheme, so right away you take a portability hit if you choose to develop in them.

Do you think there is a chance of Lisp/Scheme becoming "mainstream" again? That is, when someone contemplates starting a project, it would be as realistic for them to consider Lisp or Scheme as, say, Perl, without worrying about losing developers or initial testers? What will it take?

KMP: First, let me say that I really appreciate the poetic description you offer in the first paragraph above. I very much think that captures how I and others think about the experience of using Lisp.

And as to the future of Lisp, I think the situation for Lisp is looking pretty upbeat these days. Enough so that my own infant business is building its tools in Lisp, both for sale and for our own internal use on products we produce.

There are a lot of implementations, both commercially maintained and "free", with a wide range of delivery options, from conventional executables to "remote" solutions: Some implementations support CORBA and/or COM interfaces, for example. Also, most implement some kind of sockets interface, and there are several Lisp-based web servers available that build on this. Lisp programs can dynamically load DLLs, or can be delivered as DLLs themselves. They can do "foreign function call" to functions in other languages. It can also communicate with databases, and so with other programs via databases.

As the world moves increasingly to high-bandwidth global connectivity, I think the issue of the delivery environment will become less important. People have been waiting for an e-Service based society to take off, and it hasn't quite done that yet, but I think it's coming. I can't see how it won't. The overall savings in quality assurance and support of not having to re-deploy an application in a hostile customer-premise environment will be a lot, just as your question implies. One will just bring an application up on the right kind of hardware, connect it to the net, and then forget about where the program is actually being used. That may be an oversimplification today, but I wouldn't waste my money betting against it for tomorrow.

8) Questions I've Come Across Learning Lisp
by Jon Howard

I was recently (April) hired-on as webmaster at Franz [franz.com], a commercial lisp company (we make Allegro Common Lisp [franz.com]) which has introduced me to lisp in a very loud way. Since joining these guys (and gals), I've been thoroughly indoctrinated - with my full consent - because of my belief that as computing hardware progresses programming in more abstract languages will allow for more creative and effective use of the platform. Sure, coding assembler on a new super-duper petaflop chip will still be possible and less wasteful, but who would want to code a million lines of asm to save a few (or even a few thousand) operations out of a few billion, or trillion when it will only net a difference of nanoseconds in the end? I'm less interested in making super-fast programs than I am in making artistic and super-functional programs.

I'm not expressing the views of Franz, every member of the company has their own beliefs on what makes for great programming - which is one of the major reasons I find this place so fulfilling, everyone has complex reasons for their design considerations, and everyone communicates them (something I've grown to appreciate from working in too many places where this was definitely not the case), and consequently I've been exposed to quite a few different techniques of Lisp coding since my introduction half a year ago. I'm constantly amazed that so many different styles of programming can be expressed in the same language, it's capable of accommodating any logical thought process that can be converted to code - and I doubt many of you often use recursion in a logical way on a daily basis, but even that can be done efficiently in lisp.

I'm still very new to lisp, and I was never a serious programmer in the past, but I've always been accustomed to asking questions, and here are a few that I'd like some input on:

• If you learned any other programming language, did you initially find the formalities of its structure to be a significant stumbling block to understanding the language as a whole? Was the same true of learning lisp?
• How much time do you spend debugging non-lisp code? How much on lisp?
• What language took you the most time to learn - was it your first?
• What feature do you consider to be the most important for an abstract language to support efficiently - and which features have you found to be most poorly implemented in lisp distributions?

I'd love to hear about what people think sucks about lisp and needs improvement - or can't be improved, so far I haven't found anything that I could complain about, the most difficult thing for me has been managing all the documentation on a half-century old language in the process of learning it. I've begun to love working in lisp, but I suppose being surrounded by a group so full of passion for it has helped contribute to my bias - if I'm wrong, help snap me out of it with a good argument against using lisp. ;)

KMP: I knew FORTRAN and Basic before I learned Lisp. And I've dealt with numerous languages of all kinds since learning Lisp. With most, the syntax itself is generally not a burden. Some languages have more pleasant syntaxes than others, but the human brain has an amazing ability to cope. Of all the many languages and syntaxes I've seen, about the only thing I've never been able to cope with is the "*" used to notate indirection in C. I understand thoroughly the notion of pointer indirection, and the difference between "pointer to array" and "array of pointers", but I find it forever hard to read and write that particular awful notation for some reason. Give me Teco or Perl any day.

Mostly, though, I think the issue of how hard a syntax makes it to learn a language is overblown. Humans have brains that are adapted to processing myriad special cases and can mostly cope with obscure syntaxes. The real issue is how hard it is for humans to pass on their knowledge to programs. People are good at judgment, and programs are good at repetition. Over time, though, judgment tasks become repetitive and it's time for programs to take them over. I like to write macros to package up things I do a lot, and the key to that is having a reliable mapping between program syntax and program structure. The last thing one wants is a macro language based on character syntax, since such syntax is too unpredictable. Lisp offers macros based on program structure, and that greatly reduces the number of programmer errors one makes in macro writing.

As to debugging, I try to use non-lisp code as little as possible because of how hard it is to debug. Most other languages don't have good visual representations of their data, so when I get in the debugger, the manner in which I am presented with errant data is usually low-level and hard to read. A great deal of my valuable time is spent painstakingly piecing structure back together. But in Lisp data objects have familiar visual representations and I find it's usually easier to see what has gone wrong.

What language took me the most time to learn? Probably Teco. There was a lot of trivia to learn there. What language took the least time? Probably FORTRAN, BASIC, Lisp, HyperTalk, and MOO. Fortran just because it was small. The others because they are highly interactive, which is a huge boon to learning.

Actually, I learned PostScript very fast, too. There are some excellent cookbooks on this. But I never learned to debug PostScript. When my programs erred, I mostly just wrote them anew and hoped they'd work then because debugging was too painful.

What do I consider it most important for an abstract language to support efficiently? My time. Time is the only true, non-renewable commodity. I eschew languages like C because they often waste enormous amounts of my time trying to develop and debug programs, and justify it on the basis of micro-differences in speed that have just never ended up mattering to me. I regard C as appropriate for use as an assembly language, but it doesn't provide enough high-level services for me. When I'm old and grey and look back on my life, I want to have done a lot of interesting things, not just have done a few interesting things but "boy were they fast".

I think it's important to pick a language not on the basis of how fast its implementations are today, but on the basis of how much they do what you want. Lisp has an undeserved reputation for being slow, which I think results from deciding to make it do things that there are not always known optimizations for at the outset. Like garbage collection. But as Lisp is used, people complain about the things that are slow, and fixes get found. So Lisp moves ahead. If Lisp had started instead only with the things it knew how to implement efficiently, it would be holding things back. I want my ideas to lead my technology and my tools, not to have my technology and tools leading my ideas.

9) Basis set for programming languages?
by PseudonymousCoward

As a Scheme and Common Lisp programmer, I got excited when I heard that the Java Virtual Machine would have automatic memory allocation and garbage collection. I thought it would be possible to build Lispish languages to run on the JVM. The rate at which Kawa has been developed, to implement a near-Scheme on the JVM has been frustrating to me. I attribute this at least in part to the absence in the JVM of a construct equivalent to Scheme's continuations. Do you think it is feasible to establish a "basis set" of programming language concepts on which all programming languages could be built, so that the distinctions between C, Scheme, etc would be "merely" syntactic? If yes, please enumerate your candidate set.

KMP: Well, continuations are just functions. What's really lacking to make this easier is good tail call support so that continuations can be called correctly without pushing stack.

I don't really have personal experience with using the JVM directly, but my experience with the MOO programming language led me to believe that there might be a problem with integrating tail calling and security, since sometimes security is implemented by asking "who called me?" and tail calls can mean that the apparent caller is not the real caller. So I asked my spies at Sun about this.

I'm told that the original security model for Java worked the way I expected (by examining the call chain), and that concern over consequent security matters contributed to the absence of tail calling support in early releases. But apparently it was conceded a long time ago that such support should be added some day, and that day simply hasn't come yet. So perhaps there is hope.

Even so, I'm not so sure no matter how hard you try that you can just paper over the many differences between languages and say that the only remaining issues are ones of syntax. I do think you can probably get to a point where all languages can compile to this machine, but that may not always mean that programs in one language are as efficient as those in another, or that data structures in one language are as naturally represented as those in another. For example, both Lisp and Scheme assume that small integers (that would fit in a machine number) are still integers; they don't have the int/Integer disjointness that Java has. A Lisp-to-JVM compiler could presumably hide this distinction, but it would be wrong to say that the only difference between Java and Lisp was syntax--there are really some material philosophical disagreements between the two languages.

10) Scheme as an XML Translation Language
by Evangelion

I've become fairly interested lately in using Scheme (probably mzscheme) and the SXML package as a way to do arbitrary XML translations in my free time (if I had any).

From the looks of it, the ability to create a reflexive mapping between an arbitrary XML document and an interpretable programming language is too powerful to be ignored.

Do you think that in the future one of the primary roles of Scheme/Lisp is going to be in manipulation of XML documents, or is this going to be relegated as an academic curiosity while the world struggles through parsing XML in Java?

KMP: Are those my only two choices? The second one sounds awfully bleak. I'd better choose the former.

I don't know whether you'll see XML as a formal part of either Lisp or Scheme any time in the near future, but a lot of that is because the standards bodies administering these are not extraordinarily active at this time. That doesn't mean the languages are dead, just stable. Ongoing work is mostly happening at the level of libraries, and such libraries can generally be written by anyone using existing primitives, without modifications to the core language.

Lisp manipulation of XML and HTML is something people have been working on for a long time. For example, the Document Style Semantics and Specification Language (DSSSL) was a purely functional, side-effect free variant of Scheme. Even XSL, the apparent replacement to DSSSL, offers the same kind of functionality. It just uses a more CSS-like page model and XML syntax. But, conceptually, it's Scheme inside.

In my recent professional life, I have personally written several XML parsers, all in Lisp, for various employers and most recently for myself and my fledgling company. My company's implementation is not available on the market yet, but when it is, I'm quite sure the chief competition will not be around the availability of mere "availability". Already there are a variety of libraries related to XML, XSL, and SAX floating around. And I'm quite sure there will be more to come. Competition will be over things like efficiency, robustness, representation, and optional additional features.

11) Lisp vs. the world
by hjs

What do you see as the unique strengths and weaknesses of Lisp?

What strengths does it specifically have over other functional languages (such as ML), over structured languages (such as C, Algol, etc), over object oriented languages (such as C++, smalltalk, simula, etc), and over scripting languages (such as TCL, perl, etc)? Can these other languages or classes of languages be enhanced to include these strengths? If so, how, and if not, why?

What about weaknesses? What do you see as the weaknesses of Lisp, both in general and in comparison to the above classes of languages? Can these weaknesses be eliminated? If so, how and if not, why?

I mean strengths and weaknesses not only in the formal sense of the language itself, but also in terms of its usability in today's world. For example, difficulty in delivering binaries or lack of accessibility of system libraries from within common implementations of a language would be considered weaknesses.

KMP: There are so many things I like about Lisp, but most of them come under the heading of "doing things in the right order."

For example, type declarations in many languages are required but in Lisp they're optional. I prefer to first get my program working, and only then to tune it to be more efficient by adding type declarations. What's the point of doing a lot of make-work declarations if you're not even sure you're going to keep the result? I do a lot of exploratory programming just to answer "what if" questions. I also write lots of little throwaway programs just to compute a simple result. I don't need such programs to run in 5 microseconds instead of 10.

I also view the process of programming as a series of "times" at which decisions can be made: "coding time," "parsing time" (Lisp calls this "read time"), "macro expansion time," "compilation time," "load time," and "execution time." Lisp gives me a great deal more control for each piece of code as to when it runs, so that it can run at the appropriate time when the data it depends on is known. Other languages, especially statically typed ones, often make me specify information too soon, before it is really known, which usually means "making up" answers instead of really knowing the answers. Sometimes that makes programs run faster. Sometimes it just makes them run wrong.

And I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not important that the representation be the Right® choice. It's just important that it be a common, agreed-upon choice so that there can be a rich community of program-manipulating programs that "do trade" in this common representation.

I write a lot of macros because there are a lot of interesting things one can do with macros in Lisp. In other languages, macro-writing is a process of manipulating strings containing input syntax. That feels very unreliable and I've never liked that. Lisp's willingness to represent its code in known data structures makes macro writing feel a lot more reliable. And the presence of macros in Lisp generally means that the boring parts of coding get removed, because repetitive patterns usually get captured by a macro and hidden away, keeping the developer's attention on the "interesting parts", and making the activity of programming itself both more fun and more efficient.

Could other languages borrow some of Lisp's strengths? Sure. And they do. Java, Dylan, and I suspect even C++ have all borrowed ideas from Lisp. But that's ok. We'll make more. And anyway, it's not a zero sum game. Everyone benefits when there's this kind of cross-pollination, whether it's Lisp influencing other languages or vice versa.

Weaknesses of the language? Well, that's harder to say. I think the basic design is quite strong. Sometimes you see an implementation that has put more energy into some parts of the language than others, but usually that has created a market opportunity for another, so overall we have our bases covered.

For example, you might find some implementations that have big "hello world" footprint sizes compared to "hello world" in other languages. Some in the Lisp community, don't think this matters much, because disk and RAM are getting ever cheaper. "Real" applications (i.e., not "hello world," but something meaty) of 5-10 megabytes are pretty commonplace these days. Years ago, Lisp used to be seen as large, but due to such criticism, Lisp has held its size constant in the last decade while other languages and systems have bloated rapidly. So nowadays, Lisp is comparatively quite small. And even still, if you don't like the size you get from one vendor, it seems there's always another trying to squeeze into the niche of addressing your need. Corman Common Lisp (an up-and-coming commercial implementation) and CLISP (a GPL-style "free" implementation) have given special attention to this issue. So there's a vendor for everyone on the size issue. And, though I deal more often in Common Lisp in my day-to-day work these days, I would be remiss if I didn't mention that image size is also a key concern of the Scheme language community, so that's yet another way the size issue is addressed for those who see it as critical.

Some might have heard that Lisp, being dynamic, doesn't make use of static type information. This isn't quite right. In fact, the language doesn't require static type analysis, it merely permits it. This gives a lot of leeway to each implementation to address the specific needs of its own customer base. The CMU Common Lisp implementation has, for example, addressed the issue of type analysis in great detail and offered a clear demonstration that there are many exciting things that implementations of Common Lisp can do with type declarations if they choose to.

Why don't all implementations optimize all of these aspects--footprint size, static type analysis, etc.? The Common Lisp language is admittedly conceptually large and correct, efficient compilation requires considerable time and cleverness to implement. "Why not make the language smaller so it requires less work to implement?" is a query you hear a lot from the outside, and even from members of the Scheme community. The answer from the Common Lisp community amounts to this: Programs are written all the time, but implementations are written much more rarely. What the implementation does not do is left for the user. The more hard work the language does, the less hard work programs do. In effect, the thesis of Common Lisp is that bigger languages make for smaller sentences in the language. (To see that there is at least some intuitive basis for this, think about how long a novel like Gone With the Wind is in English, then try to imagine whether the same novel re-expressed in Esperanto would be longer or shorter.)

If a language offers only what a programmer could implement overnight, it gives its programmers not much of a leg up on their final application. Many members of the Scheme community boast that they have written a Scheme implementation, while many Common Lisp programmers have not. Common Lisp is surely harder to implement, but the Common Lisp community does not see as its primary purpose to put out legions of implementors, each with their own easily-created implementation. The Common Lisp community has chosen to be about commercial applications, and its designers have provided a "meaty chunk" of useful power for programmers to use, with the promise that if programmers write their programs to that standard, not only will those programs work well today, but as implementations get better, those same programs will work even better tomorrow.

[to be continued...]

A few weeks ago, you asked Kent M. Pitman about Lisp, Scheme, standards, and other things -- He's answered your questions below, at length. At such length, in fact, that only the first eleven of his answers are shown below -- expect more shortly! Thanks, Kent.

1) (just one thing (I) want to (know))?
by An Anonymous Coward

((
What (
(is) with (all)
) of (the) ()s?
)
Hmmm?
)

Kent M. Pitman: This question actually got scored down to -1 and marked as a troll question, but I fished it out of the barrel and restored it because everyone asks and I might as well confront the issue head-on.

Ironically it's non-Lisp languages that allow and encourage you to put ()'s in any place you want, as if there were no meaning to the introduction of gratuitous paren groups.

3+(2*5)+7 means the same thing in an algebraic language as does 3+2*5+7. In Lisp, we write:

(+ 3 (* 2 5) 7)

This shows you the structure and means you never have to learn obscure precedence rules that make expressions like -3! confusing in algebraic languages, where you must learn whether it means (-3)! or -(3!). In Lisp, the parens would show you immediately that (factorial -3) or (- (factorial 3)) was intended.

The thing I personally like about (+ (* 2 y) x) rather than 2*y+x is that it simplifies my editing. I'm a touch-typist and I use the emacs commands to go forward and backward over expressions, to swap expressions, and to delete expressions very heavily. And I don't have to reach for the mouse to manipulate large, complex expressions because they are paren-bounded. If I put the cursor at the head of 2*y+x and say "go forward an expression", ought this go forward over 2, 2*y, or 2*y+x? Having different editor commands to move across a sum, a product, etc. would be unwieldy. Yet without that, I don't see how the editor would know. In Lisp, there can't be any ambiguity because every sub-expression has its own start character, so a single notion of "the expression in front of the cursor" or "the expression after the cursor" suffices.

This, by the way, also answers the question of why we don't write foo(x) and instead write (foo x). In Lisp notation, foo is an expression. In the expression (foo x), it's a subexpression, so it's enclosed within it. Were it outside, a text editor would not be sure if foo(x) were one expression (a function call) or two expressions (the symbol foo followed by the list (x)). That would make going forward over 'one expression' ambiguous when at the start of foo(x). Should the cursor end up after the foo or after the (x)? In other words, The natural purpose of parentheses is to enclose things, so that's what Lisp uses them for. Avoiding ambiguity is critical to the writing of correct "keyboard macros" in Emacs, where I might interactively write a program to do a lot of code transformations quickly. In an algebraic language, such keyboard macros can be much harder to write robustly.

2) It's not just me is it?
by demo9orgon

After trying to "self-learn" lisp in the 80's I get this physical reaction to the word "lambda"...a cold sweat combined with the involuntary retraction of my testicles to a protected location in my abdomen (damn unpleasant shit)...I usually avoid that second one by mentally going through the mechanics of "hello world" in C, or any half-a-dozen other programming languages.

Lisp is one of those meta-languages you either learn or avoid. I write practical stuff all the time, daily in fact, and I've never had something that required the arcane stuff in LISP.

KMP: Actually, "hello world" in Lisp looks like this:

(defun hello-world ()
(write-line "Hello, World!"))

I don't know about you, but I find that pretty soothing.

And as to LAMBDA, one only needs use it when they find it useful. For example, after a while, one sometimes gets tired of writing a separate function where that function will only be used once, as in:

(defun sort-by-name (list)
(sort list #'name<))

(defun name<(name1 name2)
(or(string<(last-name name1) (last-name name2))
&nbsp(and (string= (last-name name1) (last-name name2))
(string< (first-name name1) (first-name name2)))))

so Lisp allows one to instead say:

(defun sort-by-name (list)
(sort list #'(lambda (name1 name2)
(or (string< (last-name name1) (last-name name2))
(and (string= (last-name name1) (last-name name2))
(string< (first-name name1) (first-name name2)))))))

Whether one actually does this is purely a personal preference. Some people like having separate named functions, some don't. Sometimes the separately named function might have a nonsensical name, though, and it's nicer not to have to invent a stupid name for a one-shot use.

Now, as to why it's called LAMBDA and not FUNCTION, that's just a piece of history. You get used to it. Toward that end, I'll offer a story that will perhaps help you put it in perspective:

Early in my not-yet career as a computer scientist, which is to say, while I was in high school, I lived in the Panama Canal Zone. Computers were not at all common there at the time. In fact, the place being entirely run by the US Government, there was some weird edict that said no one was allowed to own one so that they would all be centralized in the Comptroller's Office and not wasted in individual offices around the Zone. Our school had to bend the rules in order to get us a computer to study. So one thing I did while trying to learn about computers was to go downtown (out of the Canal Zone into Panama City, in the Republic of Panama) and visit a company there who did computer work. Of course, people there spoke Spanish, but fortunately I did, too. They showed me some of their code, and I was immediately struck by the fact that all the language keywords were in English.

"Doesn't that bother you?" I asked. But the person I was talking to was quite a thoughtful person and he immediately responded this way: "Do you know how to read music?" "A little," I said. "Have you seen the notations on music like forte, sotto voce, and so on?" I nodded. "Does it bother you that they are in Italian?" "No," I had to admit. His point was to make me see that it could be viewed as part of the charm and history of the notation. He was, perhaps, unusually forgiving. But this was in the late 1970s, when everyone who had access to computers was far too excited about just plain having them to care about subtle issues of whose culture got too much say in the design of a world-wide phenomenon.

So when today I look at the very few mysterious-looking terms like LAMBDA, CAR, and CDR that still linger untouched in modern Lisp's design, I think of them as I do those musical notations, conceptual links to a little piece of history that I'm just as happy not to see crushed by an overeager rush to regularize and homogenize the world--something the computer culture has done altogether too much of.

3) Interactively programmable applications
by divbyzero (divbyzero@hotmail.com)

One of the primary reasons why Scheme and Lisp interest me is that they are well suited for making applications interactively programmable at runtime (Scheme especially, due to its small size). This is far more flexible and useful than applications which are only extensible through heavyweight, precompiled plugins. Since the Slashdot readership tends to be made up of people who are comfortable with programatic interfaces (unlike the general computer-using public), why do we not see more such applications?

KMP: I think it's just an issue of education, formal and otherwise. Without being explicitly guided, some people will try out all kinds of ways to do things, or invent them where they're not present. But many others will simply do what they have been taught to do without exploring the alternatives.

In the past, everything was about speed. Every instruction was precious. The focus was entirely on "micro" efficiency. People would examine the cost of being able to redefine something (which sometimes involves as much as following pointer indirection), and if there was a cycle lost, the game was over. Today, hardware cache and prefetch architectures can often hide such costs anyway, but even if they couldn't, processors run so fast that one has time to worry not only about micro efficiency but also macro efficiency--that is, "running smart", not just "running fast", as a way of assuring total efficiency.

A lot of people identify Lisp as a language that is "just good for Artificial Intelligence (AI)". Certainly Lisp is good for AI. But saying it is just good for AI misses the point. Lisp doesn't do AI. Lisp is a programming language. AI researchers program AI, and often their language of choice has been and continues to be Lisp. But the important thing is that AI researchers have been banging on the door of Lisp implementors for years, demanding the introduction and tuning of the features and constructs they need in order to get their work done. Lisp hasn't become a mere AI toolbox as a result of that. Rather, it has become a robust tool for addressing the world's most complex and vexing problems. The Lisp community has a long experience with supporting "intelligent programming", and with doing so efficiently.

Lisp's biggest problem in the past is probably that it hit its commercial peak too early, in the mid 1980s, before most computational problems the world was confronting were big enough to need the power Lisp had to offer. Those were the days of MacWrite and MacPaint and Lotus 1-2-3, and it just didn't make any difference whether one used Lisp or C for those. But for better or worse, the world has grown up around us, and the important problems of the day are a lot more complex. I think Lisp has a lot more to offer to the world of today than it ever did in the past.

4) The standard process
by VP

As participant in the standardization process for Lisp, what are your thoughts on standards for programming languages? What would you like to see different in this process? And speaking of standards, what do you think about the RAND licensing issue and the W3C?

KMP: I think standards have served their time to provide a stable base for people to build on, but for the modern environment, they move way too slowly to keep pace with the speed of change in business. It took a long time to put the Common Lisp standard together. We began in 1986, finished work in 1994, and got the actual document to press just before the end of 1995. Getting community consensus on something that big really does take that long, and I think it was an exercise worth doing to create the stable base that we created, but for future evolution of the language, I think there needs to be another way with far less overhead.

I see standards as having two components: The first is to simply cast a name into concrete so that reference to that name will always have a clear meaning. The definition of ANSI Common Lisp, at least for 1994, is now permanently registered. Anyone who wants to can now conform to that definition and others will know exactly what they mean by that. The second component is to assert an informal consensus in the community that there is a single right way of doing things. This latter component may be useful for the foundation (to define the initial market space), but I'm not sure it's appropriate for the library level of the language.

For the base language, if 60% of the community wanted to do things one way and 40% another way, the 60% got to roll over the 40%, and 100% of the community was expected to do things in the way that won. But at the library level, if 60% want one library and 40% want another, I'd rather 100% of the community get what they want by having some people just do it one way and the rest of the people do it the other way. The Lisp community has not traditionally done things that way; they've sought consensus. The Scheme community has been even more conservative about this than the Common Lisp community, and as a result has even fewer standardized facilities than the Common Lisp community.

The Scheme community has moved to a more loose-knit approach to break the design deadlock brought on by the core language committee's consensus process through its Scheme Requests for Implementation (SRFI) process. The Common Lisp community hasn't got anything quite so organized yet, but I suspect will eventually evolve something similar.

As to the question of the W3C, I'm not a huge fan at the moment. At a prior employer, we had the opportunity to join, but the contract we'd have had to sign made it clear that votes among members were advisory only, and W3C itself could decide to override what people voted on. This, to me, is not a consensus body. Furthermore, although I think standards bodies like ANSI move in near glacial time, I don't think you can fix things by just shortening the times. True national and global consensus just takes time, and shortening timelines doesn't just make things move faster, it also disenfranchises people. While I use the existing HTML, CSS, XML, XSL, and other W3C guidelines, I don't feel they were created in a manner that I respect as proper consensus process. I think the process was insular and rushed.

Neither am I happy with the notion of processes involving Reasonable and Non-Discriminatory (RAND) fees being part of a standard; I think consensus standards should only involve royalty-free (RF) technologies. I think adherence to standards should not induce a baseline cost beyond the cost of creating the code so that the cost of compliance with standards can closely approach zero. If there is a profit to be made on the implementation of a standard, it should go to the implementor, not to a patent holder. Then again, while I'm a strong proponent of software copyright, I'm not at all a fan of software patents. Rather than seeing independent creation as infringement, I think independent creation should be contributory proof that an idea was more obvious than perhaps the patent office thought. I don't mind copyright because there are ways that one can demonstrate that one did not merely copy another's work, and independent creation is a defense.

5) Advice to Aspirants
by An Anonymous Coward

Kent, I am one of the lucky ones who programs professionally in Common Lisp. I certainly appreciate your hard work and the hard work of everyone else who helped to bring us the ANSI standard - which serves to reify much of the esoteric knowledge the Lisp community has developed in the many years since the language was born.

While I do not need to be sold on Lisp, I know many people who do not fully appreciate the power of the language. To a large degree, this is due to misconceptions about the language. Specifically, there seem to be a number of what I would call 'cultural misconceptions'. Because many people have never worked in a tightly interactive development environment with incremental compilation, language-level introspection, and real code/data equivalence (not to mention the differences between CLOS and what the rest of the world seems to have come to believe is the God-given definition of 'object-oriented' programming) - they don't really 'get' what makes Lisp so special and so powerful. More to the point, because the logistics of developing and deploying applications in Lisp is different than what the typical c/c++/perl/java developer knows, the hurdle to even investigating or considering Lisp as a real possibility seems unnecessarily high.

Could you talk a bit about how those who have a feeling that Lisp might help them with their hard problems could go about bootstrapping their way into finding out? How would you suggest getting started? What is a reasonable set of tools for experimentation, and where should a beginner start with the language? (The standard is a big document!) Also, could you give an example of the type of problem space and style of application delivery that demonstrates that Lisp is more practical than many seem to believe?

KMP: Well, one thing to note is that there's very little overhead to just downloading an implementation and diving in. Not only do the major commercial vendors like Xanalys and Franz offer high quality, no-cost trial versions of their proprietary software, but there are quite a number of free (non-proprietary) versions of Lisp as well. Information about these, as well as much other useful information about Lisp, can be found at the Association of Lisp Users (ALU) web site. I've also recently purchased common-lisp.info, which I plan to maintain as a repository for information about Common Lisp; the site doesn't have a large base of information yet, but it does have a list of the problem spaces in which you might consider using Lisp.

The ANSI Common Lisp standard, effectively available in webbed form as the Common Lisp HyperSpec, is indeed a big document (about 16MB and having about 108 kilohyperlinks downloadable). I think it's fairly readable as standards go. But you're right that it takes some work to get through and it wasn't really intended as a tutorial.

The ALU web site will also have pointers to books and online tutorials about Lisp. Books by Paul Graham and Peter Norvig on the subject are very highly regarded. I think there is always room for more, and I'm working on several, at least one of which I hope to complete in the not too distant future; feedback from you and others is useful to me in understanding what areas most urgently require treatment.

One resource that some people might find useful is an article I wrote called Accelerating Hindsight: Lisp as a Vehicle for Rapid Prototyping. This article is intended primarily for a Lisp programmer audience, to help them articulate some of the ideas you've asked about to others. It was not intended to be read by the audience you'd like to convince mainly because it appeals periodically to Lispy notation that might not be familiar to them, but it may still be of interest to the adventurous non-Lisp reader.

As your project becomes more sophisticated, and evolves from a personal toy to a real commercial product, it also doesn't hurt to ask an expert for help. My company offers consulting services that include helping companies manage the transition into Lisp. One of my major clients, The Software Smith approached me on just such a basis and the result has been very exciting both for me (getting to help them improve their system) and, I think, for them (getting to see more of how Lisp is supposed to be used). I don't want to turn this interview into a huge advertisement, but people can contact me for more information. If I'm either not competent to help you or am too busy to help you, there's a very good chance I can refer you to someone else who can help you.

6) Language feature trickle-down
by WillWare

I was a big Scheme/Lisp fan five or six years ago, but now I see most of my favorite Lisp-like language features available in Python, which is getting a huge amount of high-quality development mindshare these days. Some of the Lisp-ish features in Python that spring right to mind are functions as objects, closures, garbage collection, and dynamic-yet-strong typing, and convenient rapid-app development.

One needn't look far to find arguments that there is still something unique to Lisp that differentiates it even from very recent languages which have had ample opportunity to borrow from Lisp. But one rarely finds a really clear articulation of that uniqueness. Do you think concur with the view that Lisp is still unique, and if so, do you think that Lisp's putative advantage really is ineffable?

If there is an advantage but it's ineffable and therefore opaque to managers with purchasing power, that would explain why Franz, Harlequin, et al have had such a rocky road. Does the Lisp/Scheme community regard this as a worrisome issue? (Some folks on c.l.lisp clearly don't think so, but I don't know if they are just a noisy minority.)

KMP: I guess I think Lisp is unique, but whether it is or not doesn't affect its usefulness as a tool. I'll enumerate some things I like about Lisp, but Slashdot readers shouldn't assume that I'm asserting for each of these features that Lisp has a lock on these. Various other languages surely have some of these. But I am often heard to say: languages are ecologies. Language features are not a priori good or bad. Rather, language features are good or bad in context, based on how well they interact with other language features. Some of what makes Lisp what it is has to do with the features it offers, but some of what makes Lisp what it is has to do with how the features work together to make a coherent whole. Lifting some of these features out of context might sometimes work, but in other cases, it might not. To get a real feel for Lisp, or any language, I think you have to really use it.

Also, in my 1994 article Lambda, the Ultimate Political Party, I advance the hypothesis that languages are defined as much by their community as by their semantics. That is, languages are forever in flux, and the semantics you read about in a language spec is a point in a multi-dimensional space telling you the current location, but it does not tell you the velocity vector in that space. For that, you must look to the community. Even if two languages happened to occupy precisely the same point in design space, that is, if they had the same semantics, would they continue to over time? I think not.

For what it's worth, here are just some of the things I personally like about ANSI Common Lisp:

• Lisp is dynamic. The world is ever changing and it's useful to allow programs to change dynamically with it. I can load new or changed functions, classes, and method definitions into a running image that I'm debugging, or even in a deployed production application. When I do, the code that was running will immediately start using the new definitions. Classes can be redefined even if the new class has different slots, and, if I care to, I can control how the update is done from old to new slot arrangements for already-created instances. This kind of thing supports programs that must be continually running yet must be responsive to changes or even just bug fixes.

• Lisp is introspective. Not only can functions, packages, classes, methods be dynamically added, redefined, or removed, but programs can also inquire about whether aspects of the programming environment (functions, packages, classes, and so on) are defined, can manipulate those objects as data, can save them away, can transform or encapsulate them, etc. Also, the Lisp compiler is a standard part of the language and can be invoked even at runtime by applications that need to augment themselves. New programs can be created on the fly, then compiled and loaded and executed in the same running image as they were created, without ever exiting (and even without doing file I/O). This facilitates automatic programming and the development of layered languages.

• Lisp's syntax is malleable. There's nothing worse than being stuck in a syntax that you don't like in a language you're going to use for a long time. Lisp allows programmers to reconfigure the syntax rules for parsing characters into data and programs, as well as allowing macro technology that transforms one parsed program expression into another. And it allows control of how data is displayed during program execution and debugging. Moreover, this can generally be done in such a way that one programmer's customizations don't adversely impact another's. This makes interactions with Lisp more pleasant and debugging sessions more productive.

• Lisp doesn't force users to use variable type declarations in order to just get a program to run. The initial focus in Lisp is on getting programs working. You can add type declarations when you're done if you want to, in order to enable additional compiler optimizations. This facilitates rapid prototyping by first getting an application running quickly with low overhead, and then allowing an application to be tuned as a second pass operation.

• Lisp has a powerful class system, and a flexible meta-class system. The class system allows powerful slot and method definition, method combination, and a great many other detailed features. The meta-class system allows users to treat the object system as data that can be programmed, creating new kinds of classes.

• Lisp gives the user powerful tools for both signaling and handling errors. This means that when an error occurs, there are often a variety of ways to continue programs other than simply aborting or dumping core. Moreover, object-oriented error handling allows programs to represent errant situations, evaluate the options for how to proceed, and select an appropriate option under program control.

• Lisp uses automatic memory management. This means that when a programmer is done with an object, they just let go of it and the garbage collector reliably frees its storage. This means Lisp programs do not suffer from the memory leaks that commonly plague programmers in many other languages.

7) What will it take to make Lisp fashionable again?
by kfogel

For myself and a number of friends, Lisp/Scheme programming has for too long been a kind of mystical Eden, fading in our memories, from which we have been mostly banished in our professional lives. But we can still recall how it felt to work in a language able to shape itself to any pattern our minds might ask: coding was more interesting and more expressive, and the rate of increasing returns over time was tremendous, because fine-grained -- almost continuous -- abstraction was in the nature of the language. Life was just more fun, frankly.

Alas! In our jobs and even in our personal projects, we are often forced to use C, C++, Java, Perl, or Python -- not because we prefer to write in those languages, but for two much less satisfying reasons: first, everyone else knows those languages, so we'll get more developers with them. And second, you can't count on users and testers having the right environment to run programs written in Lisp/Scheme, so right away you take a portability hit if you choose to develop in them.

Do you think there is a chance of Lisp/Scheme becoming "mainstream" again? That is, when someone contemplates starting a project, it would be as realistic for them to consider Lisp or Scheme as, say, Perl, without worrying about losing developers or initial testers? What will it take?

KMP: First, let me say that I really appreciate the poetic description you offer in the first paragraph above. I very much think that captures how I and others think about the experience of using Lisp.

And as to the future of Lisp, I think the situation for Lisp is looking pretty upbeat these days. Enough so that my own infant business is building its tools in Lisp, both for sale and for our own internal use on products we produce.

There are a lot of implementations, both commercially maintained and "free", with a wide range of delivery options, from conventional executables to "remote" solutions: Some implementations support CORBA and/or COM interfaces, for example. Also, most implement some kind of sockets interface, and there are several Lisp-based web servers available that build on this. Lisp programs can dynamically load DLLs, or can be delivered as DLLs themselves. They can do "foreign function call" to functions in other languages. It can also communicate with databases, and so with other programs via databases.

As the world moves increasingly to high-bandwidth global connectivity, I think the issue of the delivery environment will become less important. People have been waiting for an e-Service based society to take off, and it hasn't quite done that yet, but I think it's coming. I can't see how it won't. The overall savings in quality assurance and support of not having to re-deploy an application in a hostile customer-premise environment will be a lot, just as your question implies. One will just bring an application up on the right kind of hardware, connect it to the net, and then forget about where the program is actually being used. That may be an oversimplification today, but I wouldn't waste my money betting against it for tomorrow.

8) Questions I've Come Across Learning Lisp
by Jon Howard

I was recently (April) hired-on as webmaster at Franz [franz.com], a commercial lisp company (we make Allegro Common Lisp [franz.com]) which has introduced me to lisp in a very loud way. Since joining these guys (and gals), I've been thoroughly indoctrinated - with my full consent - because of my belief that as computing hardware progresses programming in more abstract languages will allow for more creative and effective use of the platform. Sure, coding assembler on a new super-duper petaflop chip will still be possible and less wasteful, but who would want to code a million lines of asm to save a few (or even a few thousand) operations out of a few billion, or trillion when it will only net a difference of nanoseconds in the end? I'm less interested in making super-fast programs than I am in making artistic and super-functional programs.

I'm not expressing the views of Franz, every member of the company has their own beliefs on what makes for great programming - which is one of the major reasons I find this place so fulfilling, everyone has complex reasons for their design considerations, and everyone communicates them (something I've grown to appreciate from working in too many places where this was definitely not the case), and consequently I've been exposed to quite a few different techniques of Lisp coding since my introduction half a year ago. I'm constantly amazed that so many different styles of programming can be expressed in the same language, it's capable of accommodating any logical thought process that can be converted to code - and I doubt many of you often use recursion in a logical way on a daily basis, but even that can be done efficiently in lisp.

I'm still very new to lisp, and I was never a serious programmer in the past, but I've always been accustomed to asking questions, and here are a few that I'd like some input on:

• If you learned any other programming language, did you initially find the formalities of its structure to be a significant stumbling block to understanding the language as a whole? Was the same true of learning lisp?
• How much time do you spend debugging non-lisp code? How much on lisp?
• What language took you the most time to learn - was it your first?
• What feature do you consider to be the most important for an abstract language to support efficiently - and which features have you found to be most poorly implemented in lisp distributions?

I'd love to hear about what people think sucks about lisp and needs improvement - or can't be improved, so far I haven't found anything that I could complain about, the most difficult thing for me has been managing all the documentation on a half-century old language in the process of learning it. I've begun to love working in lisp, but I suppose being surrounded by a group so full of passion for it has helped contribute to my bias - if I'm wrong, help snap me out of it with a good argument against using lisp. ;)

KMP: I knew FORTRAN and Basic before I learned Lisp. And I've dealt with numerous languages of all kinds since learning Lisp. With most, the syntax itself is generally not a burden. Some languages have more pleasant syntaxes than others, but the human brain has an amazing ability to cope. Of all the many languages and syntaxes I've seen, about the only thing I've never been able to cope with is the "*" used to notate indirection in C. I understand thoroughly the notion of pointer indirection, and the difference between "pointer to array" and "array of pointers", but I find it forever hard to read and write that particular awful notation for some reason. Give me Teco or Perl any day.

Mostly, though, I think the issue of how hard a syntax makes it to learn a language is overblown. Humans have brains that are adapted to processing myriad special cases and can mostly cope with obscure syntaxes. The real issue is how hard it is for humans to pass on their knowledge to programs. People are good at judgment, and programs are good at repetition. Over time, though, judgment tasks become repetitive and it's time for programs to take them over. I like to write macros to package up things I do a lot, and the key to that is having a reliable mapping between program syntax and program structure. The last thing one wants is a macro language based on character syntax, since such syntax is too unpredictable. Lisp offers macros based on program structure, and that greatly reduces the number of programmer errors one makes in macro writing.

As to debugging, I try to use non-lisp code as little as possible because of how hard it is to debug. Most other languages don't have good visual representations of their data, so when I get in the debugger, the manner in which I am presented with errant data is usually low-level and hard to read. A great deal of my valuable time is spent painstakingly piecing structure back together. But in Lisp data objects have familiar visual representations and I find it's usually easier to see what has gone wrong.

What language took me the most time to learn? Probably Teco. There was a lot of trivia to learn there. What language took the least time? Probably FORTRAN, BASIC, Lisp, HyperTalk, and MOO. Fortran just because it was small. The others because they are highly interactive, which is a huge boon to learning.

Actually, I learned PostScript very fast, too. There are some excellent cookbooks on this. But I never learned to debug PostScript. When my programs erred, I mostly just wrote them anew and hoped they'd work then because debugging was too painful.

What do I consider it most important for an abstract language to support efficiently? My time. Time is the only true, non-renewable commodity. I eschew languages like C because they often waste enormous amounts of my time trying to develop and debug programs, and justify it on the basis of micro-differences in speed that have just never ended up mattering to me. I regard C as appropriate for use as an assembly language, but it doesn't provide enough high-level services for me. When I'm old and grey and look back on my life, I want to have done a lot of interesting things, not just have done a few interesting things but "boy were they fast".

I think it's important to pick a language not on the basis of how fast its implementations are today, but on the basis of how much they do what you want. Lisp has an undeserved reputation for being slow, which I think results from deciding to make it do things that there are not always known optimizations for at the outset. Like garbage collection. But as Lisp is used, people complain about the things that are slow, and fixes get found. So Lisp moves ahead. If Lisp had started instead only with the things it knew how to implement efficiently, it would be holding things back. I want my ideas to lead my technology and my tools, not to have my technology and tools leading my ideas.

9) Basis set for programming languages?
by PseudonymousCoward

As a Scheme and Common Lisp programmer, I got excited when I heard that the Java Virtual Machine would have automatic memory allocation and garbage collection. I thought it would be possible to build Lispish languages to run on the JVM. The rate at which Kawa has been developed, to implement a near-Scheme on the JVM has been frustrating to me. I attribute this at least in part to the absence in the JVM of a construct equivalent to Scheme's continuations. Do you think it is feasible to establish a "basis set" of programming language concepts on which all programming languages could be built, so that the distinctions between C, Scheme, etc would be "merely" syntactic? If yes, please enumerate your candidate set.

KMP: Well, continuations are just functions. What's really lacking to make this easier is good tail call support so that continuations can be called correctly without pushing stack.

I don't really have personal experience with using the JVM directly, but my experience with the MOO programming language led me to believe that there might be a problem with integrating tail calling and security, since sometimes security is implemented by asking "who called me?" and tail calls can mean that the apparent caller is not the real caller. So I asked my spies at Sun about this.

I'm told that the original security model for Java worked the way I expected (by examining the call chain), and that concern over consequent security matters contributed to the absence of tail calling support in early releases. But apparently it was conceded a long time ago that such support should be added some day, and that day simply hasn't come yet. So perhaps there is hope.

Even so, I'm not so sure no matter how hard you try that you can just paper over the many differences between languages and say that the only remaining issues are ones of syntax. I do think you can probably get to a point where all languages can compile to this machine, but that may not always mean that programs in one language are as efficient as those in another, or that data structures in one language are as naturally represented as those in another. For example, both Lisp and Scheme assume that small integers (that would fit in a machine number) are still integers; they don't have the int/Integer disjointness that Java has. A Lisp-to-JVM compiler could presumably hide this distinction, but it would be wrong to say that the only difference between Java and Lisp was syntax--there are really some material philosophical disagreements between the two languages.

10) Scheme as an XML Translation Language
by Evangelion

I've become fairly interested lately in using Scheme (probably mzscheme) and the SXML package as a way to do arbitrary XML translations in my free time (if I had any).

From the looks of it, the ability to create a reflexive mapping between an arbitrary XML document and an interpretable programming language is too powerful to be ignored.

Do you think that in the future one of the primary roles of Scheme/Lisp is going to be in manipulation of XML documents, or is this going to be relegated as an academic curiosity while the world struggles through parsing XML in Java?

KMP: Are those my only two choices? The second one sounds awfully bleak. I'd better choose the former.

I don't know whether you'll see XML as a formal part of either Lisp or Scheme any time in the near future, but a lot of that is because the standards bodies administering these are not extraordinarily active at this time. That doesn't mean the languages are dead, just stable. Ongoing work is mostly happening at the level of libraries, and such libraries can generally be written by anyone using existing primitives, without modifications to the core language.

Lisp manipulation of XML and HTML is something people have been working on for a long time. For example, the Document Style Semantics and Specification Language (DSSSL) was a purely functional, side-effect free variant of Scheme. Even XSL, the apparent replacement to DSSSL, offers the same kind of functionality. It just uses a more CSS-like page model and XML syntax. But, conceptually, it's Scheme inside.

In my recent professional life, I have personally written several XML parsers, all in Lisp, for various employers and most recently for myself and my fledgling company. My company's implementation is not available on the market yet, but when it is, I'm quite sure the chief competition will not be around the availability of mere "availability". Already there are a variety of libraries related to XML, XSL, and SAX floating around. And I'm quite sure there will be more to come. Competition will be over things like efficiency, robustness, representation, and optional additional features.

11) Lisp vs. the world
by hjs

What do you see as the unique strengths and weaknesses of Lisp?

What strengths does it specifically have over other functional languages (such as ML), over structured languages (such as C, Algol, etc), over object oriented languages (such as C++, smalltalk, simula, etc), and over scripting languages (such as TCL, perl, etc)? Can these other languages or classes of languages be enhanced to include these strengths? If so, how, and if not, why?

What about weaknesses? What do you see as the weaknesses of Lisp, both in general and in comparison to the above classes of languages? Can these weaknesses be eliminated? If so, how and if not, why?

I mean strengths and weaknesses not only in the formal sense of the language itself, but also in terms of its usability in today's world. For example, difficulty in delivering binaries or lack of accessibility of system libraries from within common implementations of a language would be considered weaknesses.

KMP: There are so many things I like about Lisp, but most of them come under the heading of "doing things in the right order."

For example, type declarations in many languages are required but in Lisp they're optional. I prefer to first get my program working, and only then to tune it to be more efficient by adding type declarations. What's the point of doing a lot of make-work declarations if you're not even sure you're going to keep the result? I do a lot of exploratory programming just to answer "what if" questions. I also write lots of little throwaway programs just to compute a simple result. I don't need such programs to run in 5 microseconds instead of 10.

I also view the process of programming as a series of "times" at which decisions can be made: "coding time," "parsing time" (Lisp calls this "read time"), "macro expansion time," "compilation time," "load time," and "execution time." Lisp gives me a great deal more control for each piece of code as to when it runs, so that it can run at the appropriate time when the data it depends on is known. Other languages, especially statically typed ones, often make me specify information too soon, before it is really known, which usually means "making up" answers instead of really knowing the answers. Sometimes that makes programs run faster. Sometimes it just makes them run wrong.

And I like Lisp's willingness to represent itself. People often explain this as its ability to represent itself, but I think that's wrong. Most languages are capable of representing themselves, but they simply don't have the will to. Lisp programs are represented by lists and programmers are aware of that. It wouldn't matter if it had been arrays. It does matter that it's program structure that is represented, and not character syntax, but beyond that the choice is pretty arbitrary. It's not important that the representation be the Right® choice. It's just important that it be a common, agreed-upon choice so that there can be a rich community of program-manipulating programs that "do trade" in this common representation.

I write a lot of macros because there are a lot of interesting things one can do with macros in Lisp. In other languages, macro-writing is a process of manipulating strings containing input syntax. That feels very unreliable and I've never liked that. Lisp's willingness to represent its code in known data structures makes macro writing feel a lot more reliable. And the presence of macros in Lisp generally means that the boring parts of coding get removed, because repetitive patterns usually get captured by a macro and hidden away, keeping the developer's attention on the "interesting parts", and making the activity of programming itself both more fun and more efficient.

Could other languages borrow some of Lisp's strengths? Sure. And they do. Java, Dylan, and I suspect even C++ have all borrowed ideas from Lisp. But that's ok. We'll make more. And anyway, it's not a zero sum game. Everyone benefits when there's this kind of cross-pollination, whether it's Lisp influencing other languages or vice versa.

Weaknesses of the language? Well, that's harder to say. I think the basic design is quite strong. Sometimes you see an implementation that has put more energy into some parts of the language than others, but usually that has created a market opportunity for another, so overall we have our bases covered.

For example, you might find some implementations that have big "hello world" footprint sizes compared to "hello world" in other languages. Some in the Lisp community, don't think this matters much, because disk and RAM are getting ever cheaper. "Real" applications (i.e., not "hello world," but something meaty) of 5-10 megabytes are pretty commonplace these days. Years ago, Lisp used to be seen as large, but due to such criticism, Lisp has held its size constant in the last decade while other languages and systems have bloated rapidly. So nowadays, Lisp is comparatively quite small. And even still, if you don't like the size you get from one vendor, it seems there's always another trying to squeeze into the niche of addressing your need. Corman Common Lisp (an up-and-coming commercial implementation) and CLISP (a GPL-style "free" implementation) have given special attention to this issue. So there's a vendor for everyone on the size issue. And, though I deal more often in Common Lisp in my day-to-day work these days, I would be remiss if I didn't mention that image size is also a key concern of the Scheme language community, so that's yet another way the size issue is addressed for those who see it as critical.

Some might have heard that Lisp, being dynamic, doesn't make use of static type information. This isn't quite right. In fact, the language doesn't require static type analysis, it merely permits it. This gives a lot of leeway to each implementation to address the specific needs of its own customer base. The CMU Common Lisp implementation has, for example, addressed the issue of type analysis in great detail and offered a clear demonstration that there are many exciting things that implementations of Common Lisp can do with type declarations if they choose to.

Why don't all implementations optimize all of these aspects--footprint size, static type analysis, etc.? The Common Lisp language is admittedly conceptually large and correct, efficient compilation requires considerable time and cleverness to implement. "Why not make the language smaller so it requires less work to implement?" is a query you hear a lot from the outside, and even from members of the Scheme community. The answer from the Common Lisp community amounts to this: Programs are written all the time, but implementations are written much more rarely. What the implementation does not do is left for the user. The more hard work the language does, the less hard work programs do. In effect, the thesis of Common Lisp is that bigger languages make for smaller sentences in the language. (To see that there is at least some intuitive basis for this, think about how long a novel like Gone With the Wind is in English, then try to imagine whether the same novel re-expressed in Esperanto would be longer or shorter.)

If a language offers only what a programmer could implement overnight, it gives its programmers not much of a leg up on their final application. Many members of the Scheme community boast that they have written a Scheme implementation, while many Common Lisp programmers have not. Common Lisp is surely harder to implement, but the Common Lisp community does not see as its primary purpose to put out legions of implementors, each with their own easily-created implementation. The Common Lisp community has chosen to be about commercial applications, and its designers have provided a "meaty chunk" of useful power for programmers to use, with the promise that if programmers write their programs to that standard, not only will those programs work well today, but as implementations get better, those same programs will work even better tomorrow.

[to be continued...]

Slashback brings you updates tonight on book reviews past, intentionally defective CDs, failing disk drives, and joining the HURD. Enjoy!

Spin control for some IBM drives? If you are one ofthe people who have the same results with IBM 75GXP hard drives that Sean Kelly did when he posed a recent Ask Slashdot, you may be interested in this report from legLess, who writes: "Pair Networks is swapping out every IBM 75GXP hard drive they have "[b]ased on an amazingly high failure rate." Pair is a big host: 114,000 sites all running on FreeBSD 4.1.1, including cdrom.com and Tom's Hardware. "We currently use and recommend Maxtor drives" they say. Big black eye for IBM."

GNU isn't Linux, either. Amid the stream of recent and upcoming software releases (Suse 7.3, Red Hat 7.2, Qt 3.0), it's sometimes easy for projects with smaller followings or more esoteric goals to get lost. BorrisYeltsin writes: "The Debian HURD iso images are now available from your local ftp.gnu.org mirror. There are 3 iso's available, so get downloading now!" (And read through the recent months' on the HURD Kernel Cousin too.)

Update: 10/16 14:20 GMT by T : Please note that the GNU Project maintains a list of ftp mirrors -- look for one local to you for best results all around :)

Placing warning signs along the road to consumerism brigc writes: "Good interview in the Chronicle of Higher Education with Jessica Litman about changes in the copyright arena since the publication of her book.

For those who were asleep, Litman's book 'Digital Copyright' does a good job of discussing why the copyright process got handed over to the industry and Congress has failed to protect the rights of the public."

Litman's book got a rave review from Michael a few months back; I suggest you check it out, and better yet ask you local library to put it up on display. Libraries have a strong vested interest in not ceding all control to copyright holders forever and ever amen.

It might pay to have a big fat mouth and ask for a refund on defective merchandise, too. anonicon writes: "Here's a heads up to the web site I'm running at http://www.fatchucks.com. I've started both a Corrupt CDs list for people who wish to report 'copy-protected' CDs or find out which ones they are, and an Indie Rec for people who want to recommend independent artists to the public. Thank you."

Kent M. Pitman has been programming in Scheme and Lisp, and contributing to the design of those languages, for a long time -- 24 years. He was a technical contributor and an international representative for the ANSI subcommittee that standardized Common Lisp, and in that capacity directed the design of Lisp's error system. Scheme may be better known as a teaching language, but both Scheme and Lisp have applications (as any Emacs user knows) that go far beyond this. Now's your chance to ask him about the pros and cons of those two languages, circa 2001 A.D. Kent also has an interesting, ambivalent take on Free software that's worth noting in an atmosphere where complex issues are often oversimplified and radicalized. Since he's someone who's helped develop standards, this is perhaps a timely issue on which to probe his opinion. It's also a good time to get acquainted with things he's written, which might interest you just as much as his programming. (Soap opera parodies, anyone?) So suggest questions for Kent below (please, one per post) -- we'll pass along the highest-rated ones for him to answer, and Kent will get back soon with his answers.

GregK asks: "Once upon a time (in the fun, fast 80's), you could find a different kind of X-box, a thin client that ran X!! and did little local processing. These boxes had a monitor, keyboard, and a thin box of electronics to communicate on an ethernet port and run an X11 server locally. Whatever happened to this kind of thing? I ask because I'm interested in putting a thin client in my bedroom, and leave my Linux server in the basement. A wireless connection could send the data over the two floors, and in an ideal world I'd have a color LCD monitor (24 bit or more) connected to a wireless transceiver. The monitor would not have a fan or a hard drive, but would be able to run X11. This would let me have an always-on connection without an always-loud presence in my bedroom. Now the best I can come up with is a modified, old laptop with the hard drive ripped out, and a boot disk/CD-R combination that would let the CPU boot and read the X software into memory. But there has to be better! Can anyone help?"

WillWare writes: "Using a scanning tunneling microscope at the Free University of Berlin, scientists have for the first time manipulated single molecules to perform a complete chemical reaction. (Here are STM pictures of the reaction happening.) ...the making of C12H10 molecules from C6H5I molecules, normally carried out on a copper catalyst and using thermal activation, has here been forced to proceed by employing one molecule at a time at a cryogenic temperature of 20 K. The researchers believe that new manmade molecules, never before seen in nature, can be engineered in this way, including the selective detachment or replacement of parts of larger molecules for individual assembling of molecular based nano-devices. The official article appears in Physical Review Letters, 25 Sept." Nanites. That's all I have to say.

jrg writes "The company, TranOrbital, Inc. has a project, TrailBlazer, to become the first (early 2001) commercial space mission to enter lunar orbit. They plan to do this for a fraction of the price it would cost NASA, plus they plan to map the entire surface of the moon in unprecedented detail using HDTV video cameras (finally, we get to see those alien bases! ;) ). If they can pull it off as cheaply as they claim, this might signal a new phase in the human utilization of space. "

Quite a number of you out there are into Lego Mindstorms, as evidenced by the number of book reviews that have been sent my way. Below are a couple of reviews, one from Kurt DeMaagd and the other from Will Ware. Click below to get their take on the O'Reilly book The Unofficial Guide to Lego Mindstorms Robots. The Unofficial Guide to Lego Mindstorms Robots author Jonathan B. Knudsen pages 247 publisher O'Reilly & Associates rating 9/10 reviewer Will Ware & Kurt DeMaagd ISBN 1-56592-692-7 summary Get the most out of your Lego Mindstorms The Unofficial Guide to Lego Mindstorms Robots Review by Will Ware

Last year, Lego released their Mindstorms Robotics Invention System. Using this, children and adults can build simple robots whose behavior can be programmed. The Mindstorms system is a major contender for Coolest Toy on the Planet.

The system contains a RCX programmable brick containing an H8/300 microcontroller, some pushbuttons, a little LCD display, and connectors for motors and sensors (light and physical contact). The user writes a program using a graphical programming language on his Windows box, and downloads it to the RCX via infrared.

Not surprisingly, substantial reverse engineering (1, 2) has been done by hobbyists, and it is possible to develop Mindstorms programs on a Linux box and to download the RCX brick from Linux.

Now O'Reilly has joined the Mindstorms fray, with a book full of fun and useful information about how to build and program Mindstorms robots. The book describes four different robots: Hank is a bumper car robot, Trusty uses light sensors to follow a line along the floor, Minerva has a movable arm, and two identical robots play a game called RoboTag. Along the way, the author discusses the physics and mechanics of robots, programming issues, and the available development environments for Mindstorms.

What's Good? There are detailed building instructions for each of the robots, showing photos at various stages of construction. The designs are simple and appear mechanically sound. There are discussion of the physics and mechanics of tank treads, steering, gears, and other things.

The book's chapters sequentially step through several different software development environments. The first chapter starts with the Windows-based RIS environment that comes on the Mindstorms CDROM. Later chapters give programming examples for NQC (Not Quite C), pbFORTH, Visual Basic, and the legOS operating system, which uses an EGCS cross-compiler to target the H8/300. There are more development platforms available, but these give a good sense of what's possible in Mindstorms programming.

The book has dozens of useful URLs, for both official Mindstorm sites and unofficial hobbyist sites. I particularly liked the fact that the author was aware of some of the recent research in robotics. For instance there is some discussion of Rodney Brooks' subsumption architecture, which is used for the RoboTag robots.

Later chapters of the book often expand on designs from earlier chapters, building more sophistocated robots in an accessible, incremental fashion. For the more adventurous hobbyist, the final chapter talks about building your own sensors and actuators, and how to connect them to the RCX.

What's Bad? Some of the photos are too dark and lack contrast. It would also have been nice if the photography had been in color, but black-and-white photos kept the book more affordable.

This book is for the casual weekend robot-building tinkerer, and it never promised to discuss real-time embedded issues in depth. Still, a few topics might have merited at least brief mention. Systems with real-time multitasking must frequently arrange for synchronization and communication between tasks, using mutexes and mailboxes and the like, which brings the possibility of deadlocked processes. Another danger is that an aggressively efficient compiler will sometimes optimize away reads and writes to hardware registers. The fix is to declare such registers with the volatile keyword.

Review by Kurt DeMaagd

While Lego Mindstorms were officially released for a teenage crowd, they have become popular with a wide variety of technically competent people in many age groups. This widespread fascination has opened up a whole new world of opportunities for using Mindstorms. At the same time, the documentation and tutorial included with the Lego kits provide very little information about how to get the most out of the sets. This book fills the void by providing several start-to-finish robot designs, software to run them, and a wealth of other tips and tricks.

After a brief introduction to robotics and how Legos fit in, the author discusses the basics of using Mindstorms to create them. Both chapters present a problem, provide step by step building instructions, provide the necessary information to program the solution, and finally go into greater detail about the Lego features used to solve the particular problem.

While the chapters did an excellent job of presenting this information in general, they fell victim to a problem that would plague the entire book: some of the building diagrams were nigh unto unreadable. Attempting to build a robot based on fuzzy black and white photographs can be quite a chore. Fortunately, none of the robots were so complex that they robots were completely unbuildable.

The first few chapters presented robots programmed with the default RIS programming environment. In chapter four and following, he shows how to program using languages such as Not Quite C, Forth, Sprit.ocx for Visual Basic--or optionally Visual C++ or another ActiveX-aware language--and legOS. Since much of these sections was documenting API's, it was certainly not the most exciting read, but it does provide concise, easily to reference documentation.

Not Quite C, as the name implies, is a C-like language that can be used to program Mindstorms robots. It overcomes many of the limitations of the default RIS programming environment, most notably the lack of variables. One of its biggest advantages is that it does not require the user to install a new version of the firmware on their RCX unit. In general, it provides an excellent balance between power and usability.

The remaining three means of programming presented in the book are fairly mediocre options. PbForth requires the user to download a new firmware version, and the language itself is very archaic in modern software development terms. Using Sprit.ocx is a viable option for people used to programming in Visual Basic or Visual C++, but the control structures are very clunky and non-intuitive. legOS, while it is probably the most powerful option, takes a significant amount of time to set up and develop applications with.

Two of the projects referenced while discussing the various programming languages were particularly interesting, both of which outlined infrared communication. The first program creates a simple remote control for controlling a robot via the IR port on the RCX. The other example, perhaps the most interesting in the book, was creating two robots who played tag with each other. These two robots also communicated with each other via their IR ports.

The last chapter, targetted toward the hard core Mindstorms users outlined how to create additional sensors for Mindstorms. It sketched out such possibilities as a passive light sensor, a Hall effect sensor (magnetic fields), and a touch multiplexor (allowing you to have more touch sensors than normally allowed on the RCS unit).

In general, the book provides a vast array building and programming tips, tricks, and methods. He gives basic information for the person who is just starting, and introduces the advanced user to the vast network of people and product that have made Mindstorms far more than a child's toy.

Purchase this book at fatbrain.

Table of Contents

• Preface

• 1. Welcome to MINDSTORMS
• What is a Robot?
• Mobile Robots
• What is MINDSTORMS?
• What Now?
• Online Resources

• 2. Hank, the Bumper Tank
• About the Building Instructions
• Building Instructions
• A Simple Program
• Wheels
• Bumpers and Feelers
• Gears
• Multitasking
• Online Resources

• 3. Trusty, a Line Follower
• Building Instructions
• Some Tricky Programming
• The Light Sensor
• Idler Wheels
• Using Two Light Sensors
• Online Resources

• 4. Not Quite C
• A Quick Start
• RCX Software Architecture
• NQC Overview
• Trusty Revisited
• Online Resources

• 5. Minverva, a Robot with an Arms
• Building Instructions
• Programming
• Directional Transmission
• Pulleys
• Mechanical Design
• Two Sensors, One Input
• Where am I?
• Online Resources

• 6. PbFORTH
• Replacement Firmware
• pbForth Overview
• About Forth
• pbFORTH Words
• An Expensive Thermometer
• Minerva Revisited
• Debugging
• Online Resources

• 7. A Remote Control for Minerva
• Two Heads are Better Than One
• The Allure of Telerobotics
• Building Instructions
• Programming the Remote Control
• Programming Minerva
• Online Resources

• 8. Using Sprit.ocx with Visual Basic
• You May Already Have Visual Basic
• About Spirit.ocx
• Calling Spirit.ocx
• Immediate and Delayed Gratification
• Programs, Tasks, and Subroutines
• Tips
• Retrieveing the Datalog
• Online Resources

• 9. RoboTag, a Game for Two Robots
• Building Instructions
• Subsumption Architecture
• Online Resources

• 10. LegOS
• About legOS
• Development Tools
• Hello, legOS
• Function Reference
• New Brains for Hank
• Development Tips
• Online Resources

• 11. Make Your Own Sensors
• Mounting
• Passive Sensors
• Powered Sensors
• Touch Multiplexer
• Other Neat Ideas
• What About Actuators?
• Online Resources

• A. Finding Parts and Programming Environments

• B. A pbFORTH Downloader

• C. Future Directions

• Index

Sebbo writes "The latest article at BeDope has coverage of the iToaster confusion. It includes a nice photo of Be's VP of Developer Relations, Tim Self, demoing BeOS R4.5's new Death Ray app on the president of Microworkz. " Ya know, it's just good to have this whole thing cleared up. For record, AOL might buy Microworkz, and Microworkz does not run a BeOS/Linux hybrid, but an OS based on the ideas found therein.

Andreas Pour sent linkage to a page where you can get the KDE mascot in T-shirt form (half the profits go to KDE). Hubert Figuiere sent us pictures from the Paris LinuxExpo if you weren't in France. Brian sent us How Stuff Works. Its actually not bad. cpfeifer wrote in to send us some spoofed book covers including Taking Down the Internet in 30mins for Dummies and IP Spoofing for Dummies. More here. An anonymous reader sent us Prozac Pez if you've been having a rough day. Dwonis sent us a point-form description of Geeks, Twits and Nerds, and the differences between them. aspodf wrote in to show us what happens when Red Meat and Star Wars come together at last. CowboyNeal sent us a link to Career Path which has a Personality Quiz that tells if you are a Jedi Master, or a Sith Lord. I think Neal ended up an Ewok *grin*.

WillWare writes "Here's a nifty microcontroller project being done by Ryerson Amateur Radio Club in Canada. They are building a SIMM board with a Motorola Dragonball (same processor as the Palm Pilot), 4 meg of flash, 8 meg of DRAM, some digital I/O pins, ports for Ethernet and RS-232, and able to drive a 320x240 LCD panel. This board is intended as a target for their MMU-less Linux port, which has previously been running on Palm Pilots. There has been mention on the mailing list of the possibility of running a Python interpreter on this board. This would be a huge win for rapid app development on embedded controllers. "

Andrew Tannenbaum was the first to inform us that Archie Comics has dropped its threat of legal action against veronica.org In related news, Captain Ajax writes "Ajax.org, making good on its vow, is sponsoring an initiative that coordinates grass-roots efforts to stop domain-names from being wrongly usurped by well-monied corporations and other unethical individuals. " I hope Captain Ajax's initiative will put an end to this nonsense.

Christian Reiniger of the new Linux Game Development Project has written up a nice piece that you might want to read if you want to see more games on Linux, and how this new project will aid that. The way I see it, the apps are coming, and in many cases, already here. We just need the games. The following was written by Slashdot Reader Christian Reiniger The Linux Game Development Center Rationale

Linux is gaining much attention these days. People who were anti-Linux for a long time suddenly discover that it has changed much the past few years, ultraconservative magazines feature positive stories about Linux at prominent places and The Big Ones in the computer business are almost crowding to support the former "hacker OS".

Good press is always welcome - but can Linux live up to its new image? Can it avoid to dissapoint the people finally giving it a try?

Well, the "It doesn't have a nice, easy to use desktop" and "There are no applications for it" arguments are vanishing in a puff of colorful smoke and the "It's too hard to install" problem is quietly dissolving. But there's still that nasty "But I can't play my favourite games in Linux!" thing.

Linux has games. Linux has good games. But that other operating system has several orders of magnitude more good games than Linux. That's bad. And difficult to overcome, as it's not only because of technical reasons. But we, the free software community, have have a long history of solving But we, the free software community, have have a long history of solving problems and shipping around obstacles. There is no reason why we should not be able to solve this issue, too.

So what's the current situation, what needs to be done and what can be done? Here is a short overview of the major issues:

• Despite Linux's rapid growth - both in terms of user base and existing software - it still is not generally perceived as viable platform for high quality games. Some of the often cited problems are without doubt true, but most of these are already at the verge of being solved and the others mainly need more public discussion.
• While many game-related SDKs and applications exist or are in the make, there is no comprehensive overview of them available.
• As all of these SDKs have their strengths and weaknesses, much can be gained by making them as modular and interoperable as possible, so that game developers can combine them to an almost optimal solution.
• For both commercial game developers wanting to port games to Linux and yet-inexperienced Open Source® developers aspiring to write free games, easy to read documentation and online help via mailing lists and/or irc are very valuable.

Quick Summary

In essence we are suggesting that this new Linux Game Development Center be a kind of meta-project. It would be dedicated to advocating Linux as gaming platform, collecting knowledge about Linux game development and using it to help all interested people, providing facilities for discussion to Linux game developers and, last but not least, encouraging and helping existing free (Open Source®) game SDK projects coordinate with one another.

Please note that this is not an attempt to impose standards or rules on anyone. We just want to do what we can to help everybody coordinate their project with the others and to encourage all game SDK developers to develop compatible libraries.

This is also a call for developers, users and game SDK projects to join our efforts.

History

In the beginning ... there were many unrelated games SDK projects started by many different groups with little or no inter-group communication or coordination.

The initial initiative of starting the Linux Game Development site came from Ian Crawford (you can read his announcement of the site here).

It was first meant as a meeting and coordination point for people developing native and free Linux games, but its scope was soon widened to support Linux game development in general - the phrase "This site aspires to be the headquarters for all Linux game development" is from that time.

Cut - Switch to the PenguinPlay mailing list. Shortly after Ian's announcement of the site, Sam Lantiga suggested on the PenguinPlay mailing list that people could get together on IRC to discuss the future of Linux game development. His idea was considered as "really good" and after the first meeting the thing was extended to all people involved in pushing game development for Linux. Here are the archives of past meetings and the plans for future ones.

Well, the irc meetings became a regular event (each Saturday) and the possibility to have a real-time discussion through irc gave a big push to our work. We started discussing on how we could coordinate our efforts better, how to make Linux more appealing to professional game developers etc. After a few meetings we came to the conclusion that it would be best to merge the SDK projects (ClanLib, CrystalSpace, GAMES and PenguinPlay) to one, giving it the full support. It seemed to be the right thing, but we were a bit uneasy with it, as merging projects is a very, very difficult task.

Then Charles Durst threw in an proposal for a clearing house project, i.e. a project that would give developers from different game SDK projects a good way to communicate with each other, remind these developers to keep the different SDKs compatible to each other etc. He first proposed that PenguinPlay could become this "meta-project", but we found Ian Crawford's "Linux Game Development Center" much more fitting.

We started working on the homepage for this and Charles wrote an announcement text we wanted to post on Slashdot or Freshmeat and several newsgroups. However, as we assembled material for the homepage, discussed its structure etc it slowly mutated from the "Linux Game SDK Coordination Center" to a site for Linux game development in general - the "Linux Game Development Center" or LGDC for short. Ian's original site laid the foundation for this (as it was aimed at helping people to develop actual games) and the transformation was completed when the "Linux Game Breeding (LGB)" (aimed at creation of new projects around Linux GameDev) and "Linux Gaming Awareness (LGA)" (aimed at advocating Linux to commercial game developers) projects joined in.

So here we are. The Linux Game Development Center is a project from Open Source® game developers, maintained by them and dedicated to all people interested in the subject. Located at www.linuxgames.org, it serves as a sister site to www.linuxgames.com, the already well-established site targeted towards game players.

The Proposal

The new Linux Game Development Center would:

• Maintain a collection of links to various game SDK projects and a "news page" of the current status and functionality of each.
• Help coordinate efforts to increase compatibility and perhaps create "glue" software between the libraries produced by different game SDK projects.
• Help game SDK developers coordinate with one another (via mailing lists and perhaps IRC get-togethers), and share algorithms and code. This could even help SDK developers abstract out new layers of common or overlapping functionality between projects.
• Help to fill the functionality gaps that are currently preventing any combination of game SDK libraries from being comprehensive enough for many professional game developers to use.
• Help to direct game developers to the right tools for their particular tasks. Making it easy to find software for a particular purpose, within certain platform, language or license requirements. We are considering using existing web-based knowledge base tools such as WikiWikiWeb or faq-o-matic, as well as tables of the features and limitations of each available package.
• Collect the general feedback that game developers might want to give the Linux community about any porting problems they might have. And helping to start, extend or fix projects to meet those needs.
• If neccessary initiate and host "please port this to Linux" petitions and mane the commercial game developers aware of the demand.
• Find volunteers willing to port commercial games to Linux and act as mediator between them and commercial game houses.
• Provide facilities for discussion between commercial game developers and Linux users on how support for Linux can be increased in the future.
• Help rally game SDK development efforts to port existing game libraries to needed, unsupported platforms.
• It could help direct interested people to other projects as needed to help with bugfixing, porting, and documentation (especially with respect to interoperability between projects).
• It could even have a relationship to game SDK projects and Open Source® games somewhat similar to the relationship Debian has with the packages that it collects. It could collect easy-to-find and easy-to-install packages of game SDKs and try to make it easy for a new developer to choose the one(s) that best meets their needs. It could even help develop policies to ensure clean interaction between libraries wanting to be added to the collection.

While game development for Linux would be an important goal of the web site, the most important goal would be the development of quality cross-platform game libraries. For that reason, developers of games and game SDKs for platforms other than Linux would be more than welcome to join us. Especially if they are interested in porting software to or from Linux.

In the end, there would still be multiple, competing game SDK packages, but that should be OK as long as at least one comprehensive open-source solution can be cobbled together from the pieces. As we have seen with multiple distributions, and even the KDE/GNOME projects, competition can sometimes be a very good thing ... if you can see past the flame wars.

The biggest problem with having multiple, competing projects is the resultant (developer and user) confusion. What we are proposing is a Linux Game Development Center that is aimed simply at reducing that confusion by helping people to find, evaluate, combine and use the available tools, or to develop new, missing ones.

Request

At this point, we are mainly looking for:

• More people to work on the web-site (in particular people who have ideas for ways we should do it with existing or new web server and/or database technologies).
• Other game SDK related projects that should be added, or who want to help, or who should at least join the linuxgames mailing list(s).
• Other Game or Game SDK developers who want to be in on the discussions, prioritizing, development, or who just want to influence the direction of the Linux Games project in one way or another.

All interested people are invited to join the linuxgames mailing list and participate in the discussions (send a blank message to linuxgames-subscribe@sunsite.auc.dk)

Current Linux Game Development Projects

These are the current Linux Game Development projects we have been able to locate and invite to participate. If your favorite project is not included, let us know and please join us.

• 3dfx HowTo
• ALSA - Advanced Linux Sound Architecture
• ClanLib
• CrystalSpace
• Daryll Strauss' Linux 3D page
• DUMB
• GAMES - GNU Animation Multimedia Entertain ment System
• GGI - General Graphics Interface
• GSI - General Sound Interface
• Linux game development webring
• Linux Game Programming HowTo
• Linux Game Programming Megasite
• Linux Game Tome
• LinuxGames.Com
• Mesa
• MGL
• PenguinPlay
• SDL - Simple DirectMedia Layer

Charles Durst, Christian Reiniger

Psarchasm wrote in to note that NetStat has recieved a makeover, no MCI, but 15 other spots are generating good net traffic reports. Mike Evans wrote in to send us a link to RFC2441, A tribute to Jon Postel by Danny Cohen. An anonymous reader wrote in to send us an article where you can read about a biochemist who is now in legal trouble for distributing genetically altered seeds that grow Oranges containing THC. He designed them because cops confiscated his car 15 years earlier. Bill Bumgarner wrote in to say that A Sherlock Plugin for searching Slashdot is now available. Bob McCown sent us a link to an interesting Pen Based Computer. Another anonymous reader wrote in to send us a link to an excellent Linux Introduction Series over at Avault on Penguin Games. Nick sent in an oldy but goodie, a link to the Unix Haters Handbook. It comes with its own Unix Barf Bag.

Morten has written in with an excellent summary of the Wine project, and an updated status report. This is definately a critical project and deserves attention. Click below to read what he has to say about it. the following is a feature written by Morten Wine, The Windows Emulator Actually, Wine is two things at the same time. First of all, it is a binary emulator that will let you run your Window 3.1/95/98/NT binaries without having to have a copy of Windows on your computer. You will require the DLLs your program uses, but Wine intends to supply replacements for all the standard ones.

When used this way, Wine loads your program and jumps to its entry point. Wine then intercepts all calls to the system DLLs and substitues suitable X-Windows calls. Assuming that we can substitute calls that are efficient, your program should run at regular speed. Moreover, we can integrate Windows programs and regular programs such that cut-and-paste operations work as expected.

(Other projects -- such as Bochs and to a certain extent Dosemu -- want to give you Windows running in an emulated PC. We wish them luck, but for this particular purpose we don't think that is the right way to go. They will have to pay Microsoft for that Windows copy and the integration between programs will suffer.)

Wine, The Windows API Implementation Secondly, Wine is an implementation of the Windows API allowing you to compile Windows programs into Unix binaries -- if you have the source code, of course. Thus, Wine (Winelib, actually) is a GUI toolkit, but since we don't intend anyone to develop directly for it, we don't see it as competing with GTK, Motif, Qt, and you name it.

We hope that Winelib can help transfer usable, free Windows programs into the Linux world with only little work. We also hope that developers -- who might want to spend the money doing a real port -- could use Wine as a quick way of entering the Linux world with their already-written programs.

Status Wine already runs your favourites: FreeCell, Solitaire, WinMine, and MSHearts. (Hey, that was 90% of the reason to use Windows already -- way to go, :-) MS Words and Excel are close to being useful; for some versions and people, they already are. Success has been reported for the Power Point viewer, but Power Point proper still needs some work. Also, the Forte Agent news reader is reported to be functioning well enough to be useful. More information can be found right here.

Since many programs are build with toolkits like MFC, we believe we are "close" to getting a lot of programs working. Time will tell.

Problems There are three major problems in the Wine development: The Windows API is really, really big. Wine is thus a large scale operation. Luckily we can proceed one function at a time. Much of Windows is undocumented -- and programs depend on undocumented stuff. This is by far the worst problem. Implementation of some system call can be quite difficult when you have no clue what it does. In my humble opinion, the US Department of Justice should have demanded full disclosure of all documentation regarding all function calls ever called by a Microsoft application. (Recall that Explorer clearly was an application at the time it came out. The "integrated part of the OS" mumbo jumbo came later.) There really is no good reason why Microsoft's application writers should have such information denied competitors. Lack of Windows API knowledge. Many of the Wine developers don't know what they are doing, yours truthfully included. We have some Microsoft documentation, some books, lots of Unix experience. Then we start coding. This would work better if Microsoft's documentation was correct and complete, but it certainly isn't. Wine Development The Wine project operates a bit differently from other Linux projects. Developers tend to come and go. We live in USENET space although nowadays we also have the Wine HeadQuarters. (Sorry. We're not a multi-million enterprise; the /. effect will probably kick in sooner rather than later.) New versions come out biweekly and are edited by our fearless leader, Alexandre Julliard.

(PLUG type=shameless)Please help with Wine: test your favourite applications, regardless of whether Linux alternatives exist. Tell us what breaks and how. Better yet, send us patches for bugs and missing functions. Is it worth it? Well, read what Linus says about Wine:

Wine, on the other hand, is in my personal opinion one of the most important linux projects currently under development. The ability to seamlessly run windows binaries among linux applications (all showing on the screen at the same time, with cut-and-paste working between them and eventually even maybe some kind of drag-and-drop setup) would be a huge advantage, mainly because windows has what linux lacks: end-user applications.

Other benefits from participating in the Wine project:

• You will get the occasional junk email about alcoholic fluids made from grapes.
• The resident comp.emulators.ms-windows.wine troll will tell you that you are a member of one of the world's secret communist parties -- I kid you not! Their main activity seems to hinder Wine development, if our source can be trusted.
• "No, I am not playing games -- I am performing valuable testing of Wine."

Thank you for you time.

-- Morten Welinder, terra@diku.dk. A member of, but not speaking on behalf of the Wine team.

Bob McCown writes "Im not sure if I like this or not. Several ISP's have been ordered to reveal the names of some of their users. Apparently they used a message board to attack employees of a Canadian company. Given the nature of the "threats" I tend to agree, but I also believe that this is a slippery slope, and could set a precedent for future abuse. "

Bob McCown wrote in to tell us that NEC is lowering their prices on their Flat LCD screens. Anyone who has ever been near one of these beasts knows that this is one step closer to putting them on everyone's desktop. The 14" is now a mere(?) $999 ($200 cheaper).

Niall Kavanagh writes "Thing Microsoft and Intel have it bad? Think again. Argentina is asking InterPol to arrest four IBM execs. who they accuse of bribing public officials to get a state contract. Details on TechWeb"

Niall Kavanagh writes "WTO ministers agreed to keep the Internet a duty-free zone - at least for a year. The US is pushing for a permanent tariff ban on online transactions. No net-tax on those CD-NOW goodies! Details on at TechWeb"

This isn't exactly new, but Yahoo reported it today so I figured I'd share this with those of you who haven't heard about the Counterfeit Pentiums that have their clocks tampered with. Apparently its a bigger problem than most of us thought... thanks to Niall Kavanagh for sending us a link.