Feature:Thoughts on the Linux Documentation Project

Matt Welsh has written in to talk a bit about the future of a project that he is very familiar with. The LDP ^[?] has been essential to help newbies (and even experts) learn what they need on their quest to become gurus. Hit the link below to read what Matt has to say about where the project ought to head. The following was written by Matt Welsh , co founder of the LDP Thoughts on the Linux Documentation Project In the last few months, the Linux Documentation Project has been in a bit of a rough spot, and it seems that things have not been moving along as smoothly as they were for the last 5 years. I have seen the latest "LDP Proposal" on the Open Source Writer's Group website, and I have a few thoughts on the current state of the LDP and what should be done to fix it.

As the original co-founder and coordinator of the LDP, I did a lot of work to make the project a popular source of Linux documentation. I started the LDP website, moderated the newsgroups, ran the mailing lists, acted as the liason between LDP writers and CD-ROM vendors and paper publishers. So I'd like to share my thoughts about what the LDP needs to get back on track.

In a nutshell, my approach would be minimalistic. I have always found that when organizing a group of volunteers on the Internet, doing less is always better than doing more.

Here's what I mean: The LDP is primarily a vehicle for Linux enthusiasts and developers to share their knowledge about the system with other Linux users. People are motivated to contribute to the LDP because they know that by having their docs on the LDP website, and on many CD-ROM disrtibutions, and even in printed books, that many Linux users are likely to read what they have written. This is much better than burying the docs on a website somewhere and relying on a search engine. The LDP is the de facto standard place for people to go to find out about Linux -- first.

As such, it is *vital* that it is as easy as possible for people to contribute to the LDP. Participation in complex standards processes, voting organizations, or high-traffic mailing lists should never be a requirement. Likewise, the tools used to write LDP docs should be easy to use, widely-available, free, and well-supported.

Ideally, we should allow people to write LDP docs in any format they wish -- but I have found that this makes it quite difficult for the LDP maintainers to publish the docs in a common format. My compromise was to always allow people to contribute docs in plain ASCII, but the preferred tool was Linuxdoc-SGML (now SGML-Tools). SGML-Tools is very easy to use and automatically generates HTML, ASCII, TeX, DVI, PostScript, and texinfo from a single source file. Also, it works with any text editor. Any system which imposes additional responsibilities on authors makes the barrier of entry too high, and people will be less likely to contribute.

Another important aspect of the LDP is that it is vendor-neutral and run by volunteers. For the LDP to be a mouthpiece of a large (or small) Linux distributor or other company would defeat its purpose. I am not suggesting that for The Puffin Group to "take over" the management of the LDP would be a bad thing -- but it is important to make it clear that the LDP is open to contribution by anyone, and is not a closed, privately-run organization motivated by corporate profit concerns. Otherwise, the LDP loses its identity as an open organization which exists to serve the Linux community as a whole.

My strong conviction is that the LDP exists for only two purposes: First, to provide widely-distributed, online publication of Linux documentation (through the website, USENET, and other means). This frees authors from having to concern themselves with making docs available: they simply send the materials to the LDP maintainer and the rest is taken care of. The second purpose is to act as liason between authors and organizations which wish to publish LDP materials on CD-ROM or in print. It is important that LDP authors are represented to publishers in this way -- publishers rarely want to deal with authors individually, but would rather deal with the LDP "as a whole".

There are some pitfalls which I think the LDP needs to be careful of when moving forward:

1. Too much bureaucracy. Turning the LDP into a complex organization with offices, roles, voting procedures, and standards does nothing to further the goal of Linux documentation; all it really does is feed the egoes of the people involved. No other successful Linux project requires this level of bueaucratic organization. Keeping things informal makes it easier to move forward and to be flexible in light of change. I have seen too many Linux projects die (or take too long to get any useful work done) because of political infighting based on organizational procedures rather than meaningful technical details.
2. Changing tools. Unless there is a compelling, immediate, and inevitable reason why some new documentation format is required, don't waste your time trying to convert all of the docs and educate all of the authors about how to use the new tools. Adopting Linuxdoc-SGML was hard enough in the beginning; it only happened because it was so easy to use and we provided a lot of examples and documentation for the tools themselves. I haven't seen any compelling reasons why DocBook (or any other tool) will make Linux documentation any better. At the very least, accept LDP docs in *both* Linuxdoc-SGML and DocBook indefinitely. Forcing authors to adopt a new tool is the best way to scare people off. The LDP isn't about tools; it's about documentation. Don't let the tools weenies take over.
3. Too much -- or too little -- leadership. Leadership in the LDP is really an oxymoron, because the group doesn't need any real leadership. All it needs is someone to work hard maintaining the documentation archives, running mailing lists, posting to USENET, and working with the various paper and electronic publishers who want to redistribute LDP materials. What it also needs is someone who is communicative, open-minded, and clear-headed enough to direct the group's energy towards productive means, rather than endless political discussions. What the LDP doesn't mean is a group of people to overlay a complex organizational structure over something which is essentially a loose-knit collection of enthusiastic writers.
4. Using the wrong copyright. My feeling is that the LDP should adopt a *class* a copyright licenses which it will accept for LDP docs, similar to the Open Source Definition. This will allow individuals to select their own preferred copyright license for LDP materials, within a certain range of allowable licenses. Forcing authors to adopt a single license simply means you will have fewer contributors. Some people distrust the GPL, others distrust the BSD license, and still others prefer to craft their own. You need to be flexible enough to accept a range of copyrights from authors.
5. Too much planning. Setting out a roadmap for a collection of documents that need to be written is the best way to ensure that nobody will ever write them. Writing docs gets done in bits and spurts, and can look like a pretty big job if someone draws up a big outline for a mega-tome which the LDP plans to work on. The HOWTO project was so successful because people could write as much -- or as little -- as they wanted to on a topic of particular interest to them. So what if it's not organized into a coherent whole? With the aid of indices and search engines, people will eventually find what they're looking for. Also remember that ownership of LDP documents is a lease: if someone fails to maintain or update their document over time, it's perfectly acceptable for someone else to revise it or rewrite it altogether. This happened several times with documents that I originally authored.

In short, I think that the only thing the LDP needs to do to get on track is to retain the essential structure it has had for the last five years. Making things any more complicated will only raise the barrier of entry to new authors, which will eventually cause the project to die out. The LDP worked so well because it took almost no effort for someone to contribute: all they had to do was e-mail me the source for a new HOWTO. As soon as you become embroiled in discussions about new tools and bureaucracies, the LDP stops getting useful work done and starts becoming "yet another overburdened Linux project". My advice is to keep it simple.

I appreciate any comments.
Thanks,
Matt Welsh, mdw@cs.berkeley.edu

Comments

I like simple

[Shotgun]

Maybe someone could write a HOWTO to make things simpler for me. I have several HOWTOs that I've printed, and as I've read them I've edited. Simple things. A misspelling here...reword a sentence for clarity there. Things that have made the text more readable for me.

How do I funnel suggested changes/corrections back to the author with minimal fuss? Shouldn't this be the first HOWTO one should read? Maybe the directions on how to submit changes should be at the top of every howto? Yes the author will be diluged with spelling error notifications for a week or so, but as the corrections are made the notifications will move quickly from 'you misspelled...' to 'the correct syntax for the command is..' or 'a clearer way of saying this is...'

So, make it simple for people (me) to submit editing. And put the directions for doing so on title page of every HOWTO.

Is the LDP any good?

[Matt Welsh]

I have noticed a few people comment that the LDP is sparse and out-of-date. That may indeed be true, but consider this: the LDP was the original (and is still the only) volunteer-run documentation effort for Linux. As it turns out, it is very difficult to get volunteers to write complete documentation for a system the size of Linux and keep it up-to-date; anybody who doesn't believe me is welcome to provide me with a counterexample. (Note that the FSF documentation is very incomplete and often mismatches the version of the software which it is packaged with.)

The original LDP planned to write a small set of long manuals (such as the Linux Network Administrator's Guide or my own Linux Installation and Getting Started). What we found was that Linux moved too fast, and there was too much to cover, to keep these manuals recent. The HOWTO project was a shift towards letting people write short documents on their favorite topic of interest -- somebody might write the Net-HOWTO while somebody else wrote the Serial-Cheese-Grater-HOWTO. This was a fantastic success, judging by the number of HOWTOs and mini-HOWTOs we have now. But this doesn't solve the problem of keeping the material up-to-date.

I don't see any way to force volunteers to maintain documentation; all you can hope for is to get writers who care to maintain their docs and have some incentive to do so. Perhaps there is now a niche market for professional writers and publication companies to step in and help maintain Linux docs, but that's not the scope of the LDP.
The benefits of the LDP are that it's free, vendor-neutral, and anybody can contribute. So, you get what you pay for.

Matt Welsh

Documentation where it's needed, MAN

[Malc]

I find that the LDP is good when I need an introduction to something new: it seems to have some examples on how to get started, plus a wider over-view (e.g. look at the Linux xDSL HOWTO, a few paragraphs on using DSL under Linux, but lots and lots of info about DSL itself). But after that I need something far more concise: MAN pages. Why aren't these being maintained properly? What is GNU's fascination with crappy INFO. MAN pages are where documentation needs to be focused as these are falling behind, or simply not even being updated as they've been replaced by INFO. MAN is the most useful documentation when you kind of know what you're doing.

SGML, DocBook, and, Is the LDP any good?

[tig]

The LDP docs, in the old days of Sunsite and few or no RPM's and debs, were crucial to anyone doing
Linux. Times have changed, and the LDP (IMHO) needs to organize documentation btter I think.

Most People do work in the context of a
distribution these days, and distributions do
things differently. I've often found files at
different places than in the docs. The docs maybe
too detailed for me to spend my time on...I want
to code, not administer my systems, nor compile
everything from scratch. What I am saying is that
the granularity of the docs is not fine enough--I'd be happy to know how to use Linuxconf to set up routing with a small sidebar as to what is going on behind the scenes. I dont want to read
the whole Net-3 howto, unless that is my interest.

Finer Granularity implies better structured documentation, in the sense that it is not enough
to say "section", but section on routing with references to the route giving the error message set ..., so that I can zoom in on exactly what I want. The more general point here is that finer granularity needs to be intelligent
granularity, so that searching is smarter, and natrual language searches are guided.

Now Docbook provides a lot of these tags. You can write API and command man-pages, there are tags like command, syopsis, errormsg, funcsynopsis,
screen, screenshot, input(what the user inputs) etc. Some of these tags are common to all documentation, some ought to be removed, and be put into domain specific XML dtd's, like those involving class framework API's, etc

Docbook is too big to use. A whittled down set of tags, finer grained and more functional than linuxdoc, combined with domain specific XML dtd's
which can then be combined with docbook XML using
namespaces ought to be developed. What would have to be agreed upon, or volunteers found to tag, is
a minimal docbook subset, and a method of combining DTD's. Conforming to these specifications, developers would be free to create their own DTD's, or convertable doc formats, and we would all reap the benefits of structured documentation.

Finally, from a users point of view, there is the documentation finding chaos. Umm lets see
do man bla, is that fails, and even if that succeeds for a FSF software, do info bla, umm we're still not there, wait there is /usr/doc, and stuff is in html there, umm no its in PDF/PS, or oh, in tex and I need to process it....Now
obviously this chaos will remain as everyone has
their favorite formats. But searching is out of the question, and at the very least, we need a RDF
source(like newhoo) of all the resources on the system which could then be used to display all places where documentation is in the system in something like one of the KDE/GNOME help browser, or
the info browser...