Feature:Thoughts on the Linux Documentation Project

Matt Welsh has written in to talk a bit about the future of a project that he is very familiar with. The LDP ^[?] has been essential to help newbies (and even experts) learn what they need on their quest to become gurus. Hit the link below to read what Matt has to say about where the project ought to head. The following was written by Matt Welsh , co founder of the LDP Thoughts on the Linux Documentation Project In the last few months, the Linux Documentation Project has been in a bit of a rough spot, and it seems that things have not been moving along as smoothly as they were for the last 5 years. I have seen the latest "LDP Proposal" on the Open Source Writer's Group website, and I have a few thoughts on the current state of the LDP and what should be done to fix it.

As the original co-founder and coordinator of the LDP, I did a lot of work to make the project a popular source of Linux documentation. I started the LDP website, moderated the newsgroups, ran the mailing lists, acted as the liason between LDP writers and CD-ROM vendors and paper publishers. So I'd like to share my thoughts about what the LDP needs to get back on track.

In a nutshell, my approach would be minimalistic. I have always found that when organizing a group of volunteers on the Internet, doing less is always better than doing more.

Here's what I mean: The LDP is primarily a vehicle for Linux enthusiasts and developers to share their knowledge about the system with other Linux users. People are motivated to contribute to the LDP because they know that by having their docs on the LDP website, and on many CD-ROM disrtibutions, and even in printed books, that many Linux users are likely to read what they have written. This is much better than burying the docs on a website somewhere and relying on a search engine. The LDP is the de facto standard place for people to go to find out about Linux -- first.

As such, it is *vital* that it is as easy as possible for people to contribute to the LDP. Participation in complex standards processes, voting organizations, or high-traffic mailing lists should never be a requirement. Likewise, the tools used to write LDP docs should be easy to use, widely-available, free, and well-supported.

Ideally, we should allow people to write LDP docs in any format they wish -- but I have found that this makes it quite difficult for the LDP maintainers to publish the docs in a common format. My compromise was to always allow people to contribute docs in plain ASCII, but the preferred tool was Linuxdoc-SGML (now SGML-Tools). SGML-Tools is very easy to use and automatically generates HTML, ASCII, TeX, DVI, PostScript, and texinfo from a single source file. Also, it works with any text editor. Any system which imposes additional responsibilities on authors makes the barrier of entry too high, and people will be less likely to contribute.

Another important aspect of the LDP is that it is vendor-neutral and run by volunteers. For the LDP to be a mouthpiece of a large (or small) Linux distributor or other company would defeat its purpose. I am not suggesting that for The Puffin Group to "take over" the management of the LDP would be a bad thing -- but it is important to make it clear that the LDP is open to contribution by anyone, and is not a closed, privately-run organization motivated by corporate profit concerns. Otherwise, the LDP loses its identity as an open organization which exists to serve the Linux community as a whole.

My strong conviction is that the LDP exists for only two purposes: First, to provide widely-distributed, online publication of Linux documentation (through the website, USENET, and other means). This frees authors from having to concern themselves with making docs available: they simply send the materials to the LDP maintainer and the rest is taken care of. The second purpose is to act as liason between authors and organizations which wish to publish LDP materials on CD-ROM or in print. It is important that LDP authors are represented to publishers in this way -- publishers rarely want to deal with authors individually, but would rather deal with the LDP "as a whole".

There are some pitfalls which I think the LDP needs to be careful of when moving forward:

1. Too much bureaucracy. Turning the LDP into a complex organization with offices, roles, voting procedures, and standards does nothing to further the goal of Linux documentation; all it really does is feed the egoes of the people involved. No other successful Linux project requires this level of bueaucratic organization. Keeping things informal makes it easier to move forward and to be flexible in light of change. I have seen too many Linux projects die (or take too long to get any useful work done) because of political infighting based on organizational procedures rather than meaningful technical details.
2. Changing tools. Unless there is a compelling, immediate, and inevitable reason why some new documentation format is required, don't waste your time trying to convert all of the docs and educate all of the authors about how to use the new tools. Adopting Linuxdoc-SGML was hard enough in the beginning; it only happened because it was so easy to use and we provided a lot of examples and documentation for the tools themselves. I haven't seen any compelling reasons why DocBook (or any other tool) will make Linux documentation any better. At the very least, accept LDP docs in *both* Linuxdoc-SGML and DocBook indefinitely. Forcing authors to adopt a new tool is the best way to scare people off. The LDP isn't about tools; it's about documentation. Don't let the tools weenies take over.
3. Too much -- or too little -- leadership. Leadership in the LDP is really an oxymoron, because the group doesn't need any real leadership. All it needs is someone to work hard maintaining the documentation archives, running mailing lists, posting to USENET, and working with the various paper and electronic publishers who want to redistribute LDP materials. What it also needs is someone who is communicative, open-minded, and clear-headed enough to direct the group's energy towards productive means, rather than endless political discussions. What the LDP doesn't mean is a group of people to overlay a complex organizational structure over something which is essentially a loose-knit collection of enthusiastic writers.
4. Using the wrong copyright. My feeling is that the LDP should adopt a *class* a copyright licenses which it will accept for LDP docs, similar to the Open Source Definition. This will allow individuals to select their own preferred copyright license for LDP materials, within a certain range of allowable licenses. Forcing authors to adopt a single license simply means you will have fewer contributors. Some people distrust the GPL, others distrust the BSD license, and still others prefer to craft their own. You need to be flexible enough to accept a range of copyrights from authors.
5. Too much planning. Setting out a roadmap for a collection of documents that need to be written is the best way to ensure that nobody will ever write them. Writing docs gets done in bits and spurts, and can look like a pretty big job if someone draws up a big outline for a mega-tome which the LDP plans to work on. The HOWTO project was so successful because people could write as much -- or as little -- as they wanted to on a topic of particular interest to them. So what if it's not organized into a coherent whole? With the aid of indices and search engines, people will eventually find what they're looking for. Also remember that ownership of LDP documents is a lease: if someone fails to maintain or update their document over time, it's perfectly acceptable for someone else to revise it or rewrite it altogether. This happened several times with documents that I originally authored.

In short, I think that the only thing the LDP needs to do to get on track is to retain the essential structure it has had for the last five years. Making things any more complicated will only raise the barrier of entry to new authors, which will eventually cause the project to die out. The LDP worked so well because it took almost no effort for someone to contribute: all they had to do was e-mail me the source for a new HOWTO. As soon as you become embroiled in discussions about new tools and bureaucracies, the LDP stops getting useful work done and starts becoming "yet another overburdened Linux project". My advice is to keep it simple.

I appreciate any comments.
Thanks,
Matt Welsh, mdw@cs.berkeley.edu

Comments

LDP quality impedes further Linux growth

[wmeyer]

Matt, it IS true. The LDP info is sparse and out of date. All the statements you make about volunteer projects are true, but isn't it ironic that Linux itself is the same kind of project, and yet its development appears to flourish, even as LDP languishes?

Linux desperately needs good, detailed, and current documentation for users and developers alike. There need to be some commitments authors make before being granted custody of a document. In particular, there needs to be a commitment to currency.

It's true that volunteers cannot be forced, but unless someone who is highly motivated to see LDP succeed takes charge and contacts each of the authors on a semi-regular basis, the project will collapse from apathy. Authors need to feel that they are part of a cohesive gropup, I think.

My own pet peeve is the Serial Programming HOW-TO, which has not received a revision for over 18 months, and was written by someone who admits his understanding of the topic was poor.

Once an author stakes a claim as maintainer for a document, there should be a requirement that the author revisit the document, perhaps quarterly, and do two things: first, review and correct or amend, and second, comment back to whoever is the coordinating member of the group that the quarterly work has been done. If the maintainer fails to maintain, then the document should be returned to the pool of unsupported documents. A web site with a list of ownership and status on the docs should not be too onerous a task for someone to maintain (see the Lazarus site for a model.)

I am new to Linux, but have been writing software for 25 years, on other OSes. I find the state of LDP docs, and of Linux docs in general, very frustrating. I have shelved my plans to move some of my company's development to Linux simply because on Linux I lose the ability to adequately schedule and predict development time. I doubt that I am alone in this.

There seems to be a pervasive assumption that we all studied Unix in college. Sorry to disappoint, but I was a fine arts major, and came to software design after having self-educated on hardware design. Coming to software from a hardware background, I expect a certain rigor and detail which is seldom found in software manuals, at any level.

I look on Linux as a potential alternative to Windows. So far, the stress must be on potential, as I cannot and will not commit the welfare of my employer's products to a system for which answers are often very hard to find.

I like simple

[Shotgun]

Maybe someone could write a HOWTO to make things simpler for me. I have several HOWTOs that I've printed, and as I've read them I've edited. Simple things. A misspelling here...reword a sentence for clarity there. Things that have made the text more readable for me.

How do I funnel suggested changes/corrections back to the author with minimal fuss? Shouldn't this be the first HOWTO one should read? Maybe the directions on how to submit changes should be at the top of every howto? Yes the author will be diluged with spelling error notifications for a week or so, but as the corrections are made the notifications will move quickly from 'you misspelled...' to 'the correct syntax for the command is..' or 'a clearer way of saying this is...'

So, make it simple for people (me) to submit editing. And put the directions for doing so on title page of every HOWTO.

I LOVE THE LDP

[Lord Kano]

If it were not for the LDP, I'd never have been able to get up and running with Red Hat 4.2 wayyyyy back when I bought it. Since then I've been able to install and play with several other versionas and distros. Yesterday I picked up Mandrake 6.0 and I owe my linux knowledge and prowess in very large part to the information available through the LDP.

IMO the LDP should be under the complete control of those who created it. You see how Linus is the patriarch of kernel development, that is how the LDP should be run.

Who can be trusted to be loyal to the project more than the one(s) who created it?

I think that a GPL type license for documentation would be a good idea, just perhaps stronger provisions to make sure that the original author of a document gets credit where it's due.

"Said documents can be copied, transmitted, edited, published by anyone provided

1. Original author(s) is (are) given credit for creation of said document.

2. Any changes made to the document are commented, and that reasonable efforts be made to give a copy of the changed document to the original author.

3. No fee be charged, beyond that of the price of the media and packaging, for distributions of the documents.

Just my 2

LK

Right On, MAN!

[joeslugg]

To this day, I cannot accomplish simple navigation
through a texinfo file. With 'man', I can scroll
up down pageup pagedown beginning end, its easy.

INFO just seems so prickly to me; I hate having
to use it. 'man' is so ubiqitous (can you say
de-facto?)

Is there something about 'man' that is somehow
non-free? or anti-GNU? Why /did/ GNU come up
with INFO?

Oh, and small note to developers: maintain your
man pages! (OK, maintain your INFO pages too, I
guess). We users ARE reading them!

Authorial Convenience vs. Reader Convenience

[The Welcome Rain]

While it's a good thing to cater to the authors of HOWTO documents as much as possible, it's also good to consider the reader's convenience. I wouldn't want to be confronted with a melange of TeX, HTML, PostScript, ASCII and SGML docs in one hierarchy, nor would I care to go searching through multiple hierarchies if the author decided to provide the docs in only one format.

Of course, SGML covers that requirement admirably -- one can generate any number of representations from SGML source -- but unless authors are constrained to provide SGML docs, one of two problems will arise:

• The user has to hunt for the proper doc, then has to know how to read it. This last requirement will cause more problems as Linux widens its user base; it may already be untrue of a good percentage of Linux users.
• The LDP has to convert the doc to SGML. This is difficult, certainly not easy to automate since SGML is a more general format than other common source formats, and it won't scale well as the number of HOWTO's grows.

These are strictly long-term concerns, however. For now, the LDP has not reached a state of completeness where it can afford to discard any contributions. Please, though -- no MSWord.

--

That's not a GNU-style license!

[azz]

These are reasonable ideas, but they're not GNU-style.

1. Original author(s) is (are) given credit for creation of said document.

No, that's BSD-style. Copyright and courtesy require this, not the GNU licenses.

2. Any changes made to the document are commented, and that reasonable efforts be made to give a copy of the changed document to the original author.

Again, that's courtesy.

3. No fee be charged, beyond that of the price of the media and packaging, for distributions of the documents.

No, the GNU-style licenses say that you can charge as much as you like, but you must make the source (SGML here) available. For instance, under a GNU GPTL (General Public Text License), O'Reilly or similar could produce a nicely printed copy of the book, but they'd have to say where you can get the SGML to make your own from.

"I want to use software that doesn't suck." - ESR
"All software that isn't free sucks." - RMS

LPD = good.

[SirSlud]

Good one. A well thought out article, with a fair amount of factual backup. I've used the LDP more than a few times and am glad it's there. I'd hate to see it go.v

I agree with Matt on what seems to be his key concern - that being that beauracracy is often the torpedo to sink the ship. Especially with the LDP, there doesn't seem to be any compelling argument for any kind of authoritative structure to govern it. I've always been a fan of the "something is more than nothing" attitude, and even if 3 in 10 articles contributed are of dubious quality, at least the other 7 are there.

Incidentally, I think I remember some sort of disclaimer on the site saying something to the effect of "we try to make sure contributions are technically correct, but they may not be." This should be something a reader keeps in mind when we read /anything/ - documentation, a manual, the newspaper. It's too bad, all too often, we forget these things and take everything we read on faith.

Is the LDP any good?

[Matt Welsh]

I have noticed a few people comment that the LDP is sparse and out-of-date. That may indeed be true, but consider this: the LDP was the original (and is still the only) volunteer-run documentation effort for Linux. As it turns out, it is very difficult to get volunteers to write complete documentation for a system the size of Linux and keep it up-to-date; anybody who doesn't believe me is welcome to provide me with a counterexample. (Note that the FSF documentation is very incomplete and often mismatches the version of the software which it is packaged with.)

The original LDP planned to write a small set of long manuals (such as the Linux Network Administrator's Guide or my own Linux Installation and Getting Started). What we found was that Linux moved too fast, and there was too much to cover, to keep these manuals recent. The HOWTO project was a shift towards letting people write short documents on their favorite topic of interest -- somebody might write the Net-HOWTO while somebody else wrote the Serial-Cheese-Grater-HOWTO. This was a fantastic success, judging by the number of HOWTOs and mini-HOWTOs we have now. But this doesn't solve the problem of keeping the material up-to-date.

I don't see any way to force volunteers to maintain documentation; all you can hope for is to get writers who care to maintain their docs and have some incentive to do so. Perhaps there is now a niche market for professional writers and publication companies to step in and help maintain Linux docs, but that's not the scope of the LDP.
The benefits of the LDP are that it's free, vendor-neutral, and anybody can contribute. So, you get what you pay for.

Matt Welsh

Re:Is the LDP any good?

[Gerv]

I am involved with an effort to write a book for Linux newbies. I was pushing really, really hard for them to use the LDP docs as a base. After all, I thought, there was no point in reinventing the wheel. But, there seemed no point unless we could get the official blessing of the LDP moderator so we could guarantee our work wouldn't be wasted. The authors' consent (to avoid forking) would also be nice.

So, what did I do? I mailed the authors of the documents (that's you, Matt, and Larry Greenfield), and the LDP maintainer (at the time, Greg Hankins), asking them if we could take it on.

No response. From anyone.

In the face of that, the group decided to go their own way and write it all from scratch. A wasted opportunity.

I also wrote a whole bunch of ideas down for simple, easy, non-controversial things we could do to make access to the LDP info easier. That want to Greg and Guylhem Anzar (sp?). No meaningful replies. (A few of them are in a separate message.)

So, what The LDP really needs is a coordinator who will answer their e-mail, and *do* something!

Gerv

Documentation where it's needed, MAN

[Malc]

I find that the LDP is good when I need an introduction to something new: it seems to have some examples on how to get started, plus a wider over-view (e.g. look at the Linux xDSL HOWTO, a few paragraphs on using DSL under Linux, but lots and lots of info about DSL itself). But after that I need something far more concise: MAN pages. Why aren't these being maintained properly? What is GNU's fascination with crappy INFO. MAN pages are where documentation needs to be focused as these are falling behind, or simply not even being updated as they've been replaced by INFO. MAN is the most useful documentation when you kind of know what you're doing.

Problem with the LDP

[Dr. Evil]

The reason I never contribute is because I don't feel qualified. Those who are qualified probably don't have time to contribute. Perhaps what is needed is a branch project, or an intermediate step... the "Hacker's LDP" or maybe a new document format... no longer the HOWTO or FAQ, but a new class of "implementor's notes" or something. I guess Mini-Howtos somewhat cover this...

In essence, this would be a centralized version of what people find on deja.com or on people's homepages. If the author wants to compile these notes into the HOWTO, then they can.

It's just an idea. It shouldn't take much to set this up... if somebody has the facilities.

The only issue I can think of is moderation and making sure that the feedback is not redundant... or that redundant feedback is appropriately categorized.

Comments?

LDP home and support at MetaLab

[pjones]

I'd like to duck into this discussion to say that whatever forms the LDP needs to take, we'll be glad to keep hosting and supporting the project at http://MetaLab.unc.edu/LDP/ or at a variety of URLs for a variety of ways of solving the documentation issues. This includes keeps docs in various formats and/or databases (mySQL mainly but we're open to supporting other forms including XML.

Love and Linux,
Paul

SGML, DocBook, and, Is the LDP any good?

[tig]

The LDP docs, in the old days of Sunsite and few or no RPM's and debs, were crucial to anyone doing
Linux. Times have changed, and the LDP (IMHO) needs to organize documentation btter I think.

Most People do work in the context of a
distribution these days, and distributions do
things differently. I've often found files at
different places than in the docs. The docs maybe
too detailed for me to spend my time on...I want
to code, not administer my systems, nor compile
everything from scratch. What I am saying is that
the granularity of the docs is not fine enough--I'd be happy to know how to use Linuxconf to set up routing with a small sidebar as to what is going on behind the scenes. I dont want to read
the whole Net-3 howto, unless that is my interest.

Finer Granularity implies better structured documentation, in the sense that it is not enough
to say "section", but section on routing with references to the route giving the error message set ..., so that I can zoom in on exactly what I want. The more general point here is that finer granularity needs to be intelligent
granularity, so that searching is smarter, and natrual language searches are guided.

Now Docbook provides a lot of these tags. You can write API and command man-pages, there are tags like command, syopsis, errormsg, funcsynopsis,
screen, screenshot, input(what the user inputs) etc. Some of these tags are common to all documentation, some ought to be removed, and be put into domain specific XML dtd's, like those involving class framework API's, etc

Docbook is too big to use. A whittled down set of tags, finer grained and more functional than linuxdoc, combined with domain specific XML dtd's
which can then be combined with docbook XML using
namespaces ought to be developed. What would have to be agreed upon, or volunteers found to tag, is
a minimal docbook subset, and a method of combining DTD's. Conforming to these specifications, developers would be free to create their own DTD's, or convertable doc formats, and we would all reap the benefits of structured documentation.

Finally, from a users point of view, there is the documentation finding chaos. Umm lets see
do man bla, is that fails, and even if that succeeds for a FSF software, do info bla, umm we're still not there, wait there is /usr/doc, and stuff is in html there, umm no its in PDF/PS, or oh, in tex and I need to process it....Now
obviously this chaos will remain as everyone has
their favorite formats. But searching is out of the question, and at the very least, we need a RDF
source(like newhoo) of all the resources on the system which could then be used to display all places where documentation is in the system in something like one of the KDE/GNOME help browser, or
the info browser...

The One Feature the LDP could really do with...

[Gerv]

"Last updated" dates. Placed beside the name of each document in the list. At a glance, you could tell whether a certain HOWTO has any chance at all of being able to solve your 2.2-specific problem.

I went through once and took the last-modded date out of each document and looked through them. Average age of the docs was about a year to a year and a half. An age in Internet time.

They would get updated if it were made easier. I know many people who have busted a gut trying to get stuff submitted, or permission to make "official" changes, but been rebuffed or ignored.

Gerv

LDP needs an active maintainer above anything

[Anonymous Coward]

I've written a pretty big and complete howto on hwoto setup a bunch of diskless clients and a root-over nfs-server. This howto is available from: http://electron.et.tudelft.nl/~jdegoede/ btw. It is about 2 months old. And I've submitted it about 2 months ago to the LDP hwoto maintainer i got one reaction about there already being an HOWTO (an outdated and very summier one if i might add). The problem is that after that I've kept him sending mails and never got any response. I want to get the howto intpo the LDP but this seems to be impossible since there is no maintainer, and yes it is written in sgml, has the right copyright etc. In the meanwhile I've pointed a few people add the howto who we're having problmes with root-over-nfs and they said: "But i checked the LPD and it wasn't there" And probably a lott of other peolle who I never mett on the net could have used the info too. So imho what the ldp needs most of all is an active maintainer. Regards, Hans