Slashdot Mirror
The Art of Unix Programming
On the surface, this book is a gentle introduction to programming; but in reality it is an attempt to explain the Unix aesthetic; at times I felt as if I were reading less a technical guide than an art history book, a chatty account of Gothic Architecture as told by someone who helped build a few churches himself. ESR articulates a set of design principles for Unix, which because of succintness deserve reprinting here.
- Rule of Modularity: Write Simple Parts connected by clean interfaces.
- Rule of Clarity: Clarity is better than cleverness.
- Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
- Rule of Simplicity: Design for simplicity; add complexity only where you must
- Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
- Rule of Transparency: Design for visibility to make inspection and debugging easier.
- Rule Of Robustness: Robustness is the child of transparency and simplicity.
- Rule of Representation: Fold Knowledge into data, so program logic can be stupid and robust.
- Rule of Least Surprise: In interface design, always do the least surprising thing.
- Rule of Silence: When a program has nothing surprising to say, it should say nothing.
- Rule of Repair: Repair what you can--but when you must fail, fail noisily and as soon as possible.
- Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
- Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
- Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
- Rule of Diversity: Distrust all claims for one true way.
- Rule of Extensibility: Design for the future, because it will be here sooner than you think.
ESR shows how to follow these general principles while writing Unix programs. The central metaphors of Unix (that everything is a file, and data-streams that can be piped and redirected) are intuitive, and maximize transparency and separation. About pipes, he writes:
A subtle but important property of pipes and the other classic Unix IPC (Interprocess Communication) is that they require communication between programs to be held down to a level of simplicity that encourages separation of function. Conversely, the result of having no equivalent of the pipe is that programs can only be designed to cooperate by building in full knowledge of each others' internals (p 81, Chapter 3)
Interestingly, the real opposing aesthetic to the Unix aesthetic is not Microsoft (which really is a hybrid), but Macintosh:
The Macintosh's unifying idea is so strong that most of the other design choices we discussed above are either forced by it or invisible. All programs have GUIs. There is no CLI at all. Scripting facilities are present but much less commonly used than under Unix; many Mac programmers never learn them. Mac OS's captive-interface GUI metaphor (organized around a single main event loop) leads to a weak scheduler without presumption... Mac OS applications are not, however, invariably monster monoliths. The system's GUI support code, which is partly implemented in a ROM shipped with the hardware, and partly implemented in shared libraries, communicates with Mac OS programs through an event interface that has been quite stable since its beginnings. Thus, the design of the operating system encourages a relatively clean separation between application engine and GUI interfaces.
ESR's criticism right here (and throughout the book) is not necessarily condemnation. In fact, ESR recognizes that Macintosh as a competing aesthetic has a lot to offer that Unix does not. For example, Mac file attributes (in which a file has both data and resource "forks") provide mechanism for richer GUI support. Thus, we have (ESR says) developers "who work from the interface inward, rather than the engineer outward." In contrast, the Unix byte-stream metaphor may make it hard to conceptualize the meaning of operations such as Create, Open, Read, Write and Delete.
With Unix, the Rule of Least Surprise suggests that the developer delegate interface functions to a GUI or to another program. Instead of creating a built-in editor inside an application, the developer should allow the user to choose which editor to use. Instead of making a robust and easy-to-use interface, the Unix developer should produce a command line utility first, and then let someone else create a separate and independent GUI layer. (Rule of Separation). Xcdroast is a perfect example of a GUI layer for the command line program, cdrecord.
The Command Line Interface (CLI) may scare off new users, but it offers endless scripting and batching capabilities for programs (and smooth IPC). Also, it offers expressiveness to developers. "The Unix programmer," ESR writes, "is likely to see defaulting away from expressiveness as a sort of cop-out or even betrayal of future users, who will know their own requirements better than the present implementer. Ironically, though the Unix attitude is often construed as a sort of programmer arrogance, it is actually a form of humility -- one often acquired along with years of battle scars" (p.304, "Interfaces,")
ESR distinguishes between interface complexity and implementation complexity. Unix projects often involve tradeoffs on these two. The Unix developer prefers a lean and simple implementation at the expense of a usable interface; ESR writes:
Programs that mediate between the user and the rest of the universe notoriously attract features. This includes not just editors but Web browsers, mail and newsgroup readers, and other communication programs. All tend to evolve in accordance with the Law of Software Envelopment, aka Zawinski's Law. 'Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones that can'".
The book devotes considerable space to showcasing Unix best practices (which in most cases, means readable config files, limiting the program's scope, providing access to sufficient debug information and making it easy to hack). He offers sensible advice about when to use minilanguages ("don't extend your way to it, one patch and crufty feature at a time"), why to limit the number of available command-line switches (each one increases debug time), why optimization has few payoffs ("Moore's law implies a 26% performance gain just by buying new hardware every six months") and why bottoms-up programming can work better than top down design(because "it gives you time and room to refine a vague specification" when "...you are programming in an exploratory way, trying to get a grasp on hardware or software or real-world phenomena you don't completely understand.").
One of the most instructive chapters was about J. Random Newbie, a fictional programmer fresh out of college who understands basic programming concepts and the value of reuse. ESR presents a scenario of how Newbie nonetheless ends up rewriting rather than reusing, relying more on his own code than that of others. When he changes jobs, "he is likely to discover that he can't take his toolkit with him" and may even find it to be useless at a new job with different proprietary tools, languages and libraries. Thus, ESR writes:
Programmers have reuse (and other good practices that go with it, like modularity and transparency) systematically conditioned out of them by a combination of technical problems, intellectual-property barriers, politics and personal ego needs. Multiply J. Random Newbie by a hundred thousand, age him by decades, and have him grow more cynical and more used to the system year by year. There you have the state of much of the software industry, a recipe for enormous waste of time and capital and human skill -- even before you factor in vendors' market-control tactics, incompetent management, impossible deadlines, and all the other pressures that make doing good work difficult. (p.420, "Reuse")
This book is a good introduction for this newbie programmer (as well as a warning about what to expect). It offers practical advice about which license and language to use, how to set up documentation, how to decipher the standards process, how to check things into CVS and how to submit a patch to an open source project.
The chapter on Unix's history puts the current SCO/IBM controversy into perspective. Unix has always been dogged by exertions of commercial control, and ESR accurately conveys how the software world is constantly swinging back and forth from periods of intensely-creative free-spirited openness to periods of commercial control.
I was struck by several of ESR's observations: that Linus Torvald's "refusal to be a zealot" was a contributing reason why Linux was able to succeed; that both the patch utility and email probably did more to advance the Open Source movement than mere "consciousness raising."
In summary, this book is a great help for the student programmer. It synthesizes a lot of ideas and insights from other programming gurus, and is full of insights, aphorisms and fun digressions (no surprise to readers of ESR's other works). The experienced programmer, on the other hand, might find the book more profound than practical. Invoking the Zen metaphor (and even including Unix koans at the book's end), ESR shows us how the abstract nature of programming provides insight into problem-solving, design and yes, even a kind of enlightenment. The book, available for free on the Net, is probably better to read on vacation or an airplane or as a welcome break when stumped by a programming problem. More practical books on Unix programming exist (I happen to recommend Mark Sobell's), but ESR's book will stay with you long after you have finished reading, providing countless hours for reflection and appreciation of Unix's Unix-nature.
Robert Nagle (aka Idiotprogrammer), believes that plone is the best thing since garage door openers, and is a strong supporter of music sharing. You can purchase The Art of Unix Programming from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Yeah, I stumbled onto web version, too.
I bought the dead tree version as way of saying thanks to ESR. It's a great read, though far more qualitative than quantitative.
I thought his treatment of the various personalities (RMS, Linus) was fair, and his notes about where Unix got it ronngg was interesting. Clearly biased, yet not grossly so.
My theory, though, is that all of the "better" aspects are generally lost.
Couple of trivial examples, the idea of having logical and physical file systems highly interchangeable and configurable is irrelevant for most users. The market has said that they want a floppy named "A:\" and a hard drive named "C:\". And bother me not with these 'script' things. Give me icons or give me death.
IMHO, the task is not to have the geeks show that the brand of suit in the boardroom is irrelevant. No amount of careful, articulate argumentation can do that. The task is to get the wearers of the suits in the boardroom to think, and I daresay even Zen Master Foo at the end of the book would be hard pressed to accomplish that.
Put another way, Unix and Open Source both assume a sophisticated user, and the market has repeatedly argued otherwise.
Possibly the sheep enjoy the shearing?
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
OK, he's familiar with the program, but the effect is, at times it really looks like it's all he's familiar with. I mean, if the design principles it illustrates are so prevalent, it shouldn't be hard to find other examples that avoid this obvious slant, no?
Here are some quotes from TCatB (October '99). ..."
"Microsoft will not have an enterprise ready operating system, because Windows 2000 will not ship in a useable form. (At 60 million lines of code and still bloating, its development is out of control.)"
"Windows 2000 will be either canceled or dead on arrival. Either way it will turn into a horrendous train wreck, the worst strategic disaster in Microsoft's history."
"the most likely scenario for late 2000/early 2001 has Linux in effective control of servers, data centers, ISPs, and the Internet,
Raymond is a true visionary. He can almost see beyond his own nose. Why anyone takes him seriously is beyond me.
This is bogus:
It's just as easy to define and use a serialization format for IPC via shared memory as it is do so for IPC via pipes. Sure, if you just copy objects from the stack or heap to shared memory without any abstraction (such as a vtable), then all programs will have to know about the common representation, but that applies equally to stream-based IPC mechanisms such as pipes or sockets.
Honestly, kids, don't bother with this book. Read K &R, Steele, Stevens, Richter, and Stroustrup instead.
Anyone who's ever worked with any of Eric's code (and there's a lot less of it than he usually lets on) knows that ESR is the world's LEAST qualified person to write a book about programming as an art. A nice example of ESR's code is fetchmail, one of the most security-hole ridden POP clients ever produced; so poorly coded that it should likely be thrown out since it can't be fixed. Well, it can be fixed about once or twice a month I guess, if history serves.
Mod me down if you wish, but ESR is a blundering fool, and his efforts to promote Linux only make the Linux community look like similar fools. After all, what kind of community would want THAT for its leader?
Customers who bought The Art of UNIX Programming also bought:
History of Classical Music by Britany Spears
Gov't By The People by George W. Bush
Improving Your Golf Swing by Stephen Hawking
Interpreting Modern Dance" by Stevie Wonder
Gee, remember when fsck came along? It was a lot easier than icheck, dcheck, ncheck, and your last resort: adb. Sysadmins really had to know their stuff before fsck.
Bruce
Bruce Perens.
Well worth the read, even if you aren't familiar with / think you don't like Lisp. There's another page with links to followups too.
I haven't read the whole thing, but I've read parts of it. Some of it is quite good, but other parts... Well, I'll just say that he doesn't let the facts stand in the way of the point he's trying to make.
Really, it just blew me away how biased he was in some areas. For example, about two years ago, I was reading the draft edition. In his section where he was discussing various operating systems, he compared Windows to UNIX. He made some very broad, very negative statements about Windows that simply weren't/aren't true. The link at the bottom of the page asked for corrections and suggestions, so I sent him a short email with some additional information. He replied, and admitted that he didn't really know much about Windows since he didn't use it very much, and that the information was all second hand. I pointed out some factual errors, along with references to back up the corrections. The discussion seemed very positive at the time. However, now that the book has gone to print, I have looked over that section to see how it turned out. Not much has really changed -- some of the most blatant mistakes have been corrected, but many factual errors of which he was fully aware remain in the printing edition. Most importantly, the tone of the section is very authoritative and negative towards the technical aspects of Windows, when ESR admits that he really doesn't know much about it.
I guess that tells me what I wanted to know about the book. It may have some wonderful insights about the "UNIX way" (which is actually pretty cool), but the author isn't interested in making me wiser and better informed -- he's pushing an agenda. His statements are heavily slanted towards what he wants people to believe, and he is willing to bend the facts to strengthen his case.
Personally, I think he would have no trouble defending his position with solid facts. There is a lot of good in the "UNIX Way" of doing things. He shouldn't need to twist things to get his point across. This simply discredits him, and by association, discredits the Open Source movement in general.
There are plenty of books out there about UNIX, more than I'll ever be able to read in the limited amount of time I have for bedtime reading. I'm not going to spend my time on one that I can't trust to give me the straight facts.
Time flies like an arrow. Fruit flies like a banana.
I got sick of patching fetchmail for one buffer overrun after another. Unfortunately, there's nothing else out there that will do what it does, so until someone gets fed up with and rewrites it, we're stuck with it.
I especially like how the passwords are stored in cleartext in the various rc files, if you want fetchmail to run as a daemon. Yeah, real secure design there.
Fetchmail is a great victory for how open source can successfully work to rescue old code and turn it into something new and viable. It should not be used as an example of sound design.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)