Mark Edel Answers Project Leadership Questions

Okay, here (as a refreshing break from all the political stuff) is what amounts to one of the most lucid primers on software development management I've ever seen, contained in NEdit leader Mark Edel's answers to your questions. If you ever expect to lead an Open Source project (or a closed source one, for that matter), you need to read this from start to finish, possibly even print out a copy and read it over and over. Great stuff!

Project coordination
(Score:5, Interesting)
by techmuse

First, thanks for producing Nedit. I've been a big fan of the program for years, and it's always one of the first things that I install. I'm wondering how you go about coordinating the development and testing of code on an open source project. Do you create well defined APIs for modules to interact with, and then ask people to work on modules that use those APIs? Do you put one person in charge of each segment of the project? How do you resolve disagreements over the features and functionality of the project? PS. What happened to the // or /* */ commenting macro in Nedit? I miss it!

Mark:

Everyone working on NEdit is a volunteer, and that makes the development process very different from what you would find in industry. I can't say that I even fully understand it. What is most peculiar and unnerving about it, is that you can't actively recruit someone for a given task. People appear out of nowhere and volunteer to work on things that you'd never dream of asking. Whereas, putting something on the wish list seems to kill off all enthusiasm for it.

Good APIs have indeed played an important role in coordinating and advancing NEdit's development. By far, the most successful of these is the one for syntax highlighting. Almost all of NEdit's language support is user contributed. Between the built-in patterns and the ones available on the web site, NEdit has highlighting patterns for more than sixty languages. There's probably something to be learned from the fact that a similar API for language sensitive indent has been so much less successful at attracting developers. Smart indent macros are harder to write, and require you to know the NEdit macro language, whereas highlight patterns only require understanding of regular expressions. I can only guess that there is a critical threshold where the quantity of work required exceeds the perceived benefit.

The majority of contributions to nedit are unsolicited. The contributor is relying on existing APIs, documentation, and program structure, and either blindly hoping that we'll integrate the change, or aiming for a target audience who will use it as an add-on. For larger projects, where a developer comes to me with an idea and a willingness to implement it, we send email back and forth. It usually takes many exchanges to define the scope and functionality of a change and decide how to fit it into the existing structure of NEdit. For significant projects, it makes a big difference if developer is willing to take some responsibility for ongoing maintenance of the new code.

What all this adds up to, is lots of rejected contributions, a few bad feelings, and even code forks. I'd like to say that we resolve disagreements via the developer discussion list, but in practice, I am acting like a dictator. The worst part about it is knowing that I am making mistakes, and slighting contributors because I'm too busy with other work to process their contributions in a timely manner.

I would prefer a model where this responsibility is shared among a larger group of people, either as a team of core developers, or even better, where we each take responsibility for smaller parts of the overall project. In areas where we've managed to divide up the project like this, we've so far done stunningly well.

(The C/C++ commenting macros are still there in the current version. They are language sensitive, so they show up only in appropriate language mode(s). If you still don't see them in C mode, try temporarily renaming your .nedit file, saving your preferences, and cutting and pasting them from the new .nedit file to your original.)

Using Motif/other 'unpopular' libs
(Score:5, Insightful)
by JanneM

A lot of people are very hesitant to install a whole set of libraries to run only one application -- almost no matter how good the app is -- when there are 'good enough' alternatives for the standard libs they already have.

Do you feel that NEdit has suffered from not using more popular libraries, and does it matter to you?

Mark:

Life with Lesstif and Open Motif is not as bad as some of the comments I saw in moderated-out questions would suggest. Users do not need to install these toolkits to use NEdit. The pre-built binaries come with Motif linked in statically. The "horrible bloat" that this static linking adds is about half a megabyte.

The real problem that Motif has caused for NEdit has been in its relationship with the various Linux distributions. Another question which didn't make it here, was titled "Why so Unstable?". I'm surprised there weren't more of these. Both Debian and Red Hat, refused to have anything to do with Motif, and would not accept statically linked binaries of NEdit. What they did do, however, was worse than just leaving it off. They packaged NEdit with early, unstable, versions of Lesstif, so Linux users who saw NEdit during this period, thought it was a pile of junk. An odd consequence is that NEdit has a larger user base in Europe than in the U.S., because Suse did distribute the binaries for a while.

Now that Lesstif is up to speed and Motif has been forced to open up, I'm hoping NEdit will finally appear in good form on the popular Linux distributions. Unfortunately, Red Hat is now so big, I haven't found anyone there who will pay attention to such a request. The one message I sent them got no reply, and I'm afraid of making a pest of myself.

Converting NEdit to a different toolkit will be a huge amount of work for very little gain. I am no fan of the Motif programming model, and I would not suggest it for new projects. However, NEdit has a huge investment in working around all of Motif's oddities, and I'm sure we'll face a similar battle with Gnome or Qt.

Users. Developers. And How to Find Them.
(Score:5, Interesting)
by omarius

I run a small (one developer: me!) project at sourceforge: a text-based web bulletin board called vaxbb.

Although sourceforge says it's been downloaded 63 times, I've received nary a comment or email of any sort. Granted, it needs a lot of work before vaxbb will be an install-n-go program, but I definitely think it fills a niche (I started writing it because I couldn't find a free bb that I liked the look of).

So, after all that intro, my question is: Does a project have to be super-slick before people will use or contribute to it? How does one find developers w/out a huge user base. . .or get a user base without having a fully-developed program?

Mark:

(I'll answer this question and the next one together)

How do you "advertise"?
(Score:5, Interesting)
by OlympicSponsor

Reading newsgroups and mailing lists, I occasionally come across people who have problems that could be solved with software I have written and made available. I have a hard time coming up with a way to point them towards my software without coming across as some kind of big-headed, know-it-all, "my software cleans your floor and does your taxes" loser. What to do?

Mark:

I'm probably the wrong person to ask these questions, because my answer is just going to depress you, and because I consider myself a complete failure in this area. I really believe that NEdit is the best editor out there, and that everyone should be using it; and yet, they aren't. I've hit people over the head with it, publically made a fool of myself over it, and utterly failed to sell a free product that can measurably make peoples lives better.

Probably I'm just delusional, and the ability to operate in terminal mode is actually the most important characteristic that Unix users want in a text editor.

Let me digress even further:

In the early Macintosh years, my wife and I ran a small software business. Our one product was a utility called SnapJot, of which we managed to sell something like 5,000 copies. The hard lesson we learned from the experience was that the game was all about marketing, and hardly at all about software. Our sales were ALWAYS directly proportional to our marketing budget.

Being engineers, we began with the faulty assumption that the quality of the product meant everything. We wrote the software, packaged it with a simple black and white manual, and submitted it to a bunch of magazines for review. The result was a one tiny entry in a new products listing, and a note from one reviewer claiming unspecified bugs. Sales, near zero.

Making no significant changes to the software, but switching to professionally printed color packaging, we got good reviews from all of the Mac magazines (5 mice from MacUser!). Sales, still near zero.

We began paying for listing in the MacWarehouse catalog. Sales!

More catalog adds, a few small magazine adds Sales, Sales, Sales!

A year or so after we closed up shop, I remember an interesting tidbit of software industry news. After selling on the order of a million copies of the heavily marketed SoftRAM '95, the Federal Trade Commission forced Syncronys Softcorp to admit that it was selling a product which literally did NOTHING.

Having learned this lesson the hard way, I don't feel nearly as put upon by the music/software/film/book publishing industries as the general Slashdot readership seems to. Ask yourself, are you seeking out MP3s of new unpublished artists, or desiring exactly those on whom the accursed music industry has lavished its marketing dollars?

Is the Free Software community different? Yes, particularly if you're the author of something which fits a well defined need. If you write a driver for a yet unsupported piece of hardware, you need only hint at it in an appropriate newsgroup, and users will pile up at your doorstep. If you write something less well defined, though, your voice can be easily drowned out by the ever increasing background noise level of the Internet.

There's definitely luck involved in whether something catches on or not, but having a polished package and pounding the digital pavement will certainly help. I find selling personally painful and degrading, and I'm terrible at it. Good salespeople require a special kind of thick skinned personality that enables them to face rejection, something I think very few of us nerdly types possess. NEdit has certainly suffered for it (the dumb name hasn't helped, either).

I have to ask...
(Score:5, Interesting)
by pb

What data structure do you use to store the text internally, and why? What trade-offs exist in your approach?

Mark:

NEdit stores text in a simple gapped-buffer structure, common to many text editors. The gapped-buffer is a block of memory with a moving hole in it. NEdit maintains the gap's size and continuously relocates it as part of the insertion / deletion process. This organization maps exceptionally well to how people actually use a text editor, and is very effective at minimizing the number and size of memory relocations that need to happen as users add and delete text. For comparison, another common approach to storage in a text editor is to break the text into linked blocks so that any given modification affects only a limited number of blocks. While this better limits worst-case memory movement; it also breaks up and scatters the stored data, destroying its locality; so the virtual memory system can't swap it as efficiently.

While I feel good about the data structure itself, the bigger question is whether the data should be maintained in virtual memory at all. The advantages are simplicity, good performance, and robustness in the face of file system problems. Because NEdit only touches a file on open and save and does not depend on its contents being stable, your work is much less likely to be trashed by mishaps like overwriting, system crashes, loss of network connections, etc.. The disadvantage is that if you have a really big file you want to edit, you may have to increase your swap space, or maybe give up altogether and use a different editor. In practice, text files are absolutely miniscule on the scale of todays systems, and rarely does anyone approach that limit, but I certainly wish the limit wasn't there.

Common sense of FS/OSS development
(Score:5, Interesting)
by dilger

Here's two questions:

1.From which (if any) of the commonly accepted principles of FS/OSS development does your project deviate?

Mark:

The advice, "Release early, release often", contradicted everything I had been taught about software engineering. For most of nedit's life, it has had carefully planned alpha and beta tests, with as much as a year between new versions. When I first read "The Cathedral and the Bazaar", though, something clicked. I had only just started to realize some of the odd nature of what we were doing. I'm still in awe of the power of distributed debugging. The reliability we've achieved with NEdit is beyond anything I could have hoped for had the testing been limited to a small group of people. NEdit simply doesn't crash. I also realized that some of the things I was doing were actually making it harder for people to contribute, and I've been slowly trying to adopt more of the other non traditional practices of the FS/OSS movement.

Since most of NEdit's development happened before the Cathedral and the Bazaar paper, and before the whole movement gained momentum, I haven't yet had enough experience testing those principles to contradict them. We've only recently reached the goal of having the development sources continuously accessible.

One noticeable difference between NEdit and other FS/OSS projects is that the code is neat and thoroughly documented. As I mention in a later question, enforcing this has not been fun, and has probably been counterproductive, as well. It's just something I've found hard to give up on, particularly where I'm going to be the one with the responsibility for maintaining a given section of code.

2.How much time do you spend writing code vs doing more administrative/moderation tasks?

Mark:

I hardly get to spend any time actually coding. Given the limited time I have to spend on NEdit, at all, my first priority is integrating what other people are doing. I'm thankful that other people have stepped in to help with some of the management. I'm really only in charge of the code, now. I'm also thankful to the user community as a whole for participating in the ongoing discussion and answering eachothers questions. I used to get completely bogged down in email, to the point where I couldn't do anything productive at all.

Software Enginnering for OSS?
(Score:5, Interesting)
by Midnight Warrior

Now that you've successfully created a wonderful, stable editor, how many of the following software engineering techniques were used on the project, or would have been helpful?

1.Clear, concise statement of work (since NEdit was developed with U.S. Govt funding)

2.A Requirements Management process (measurable details about end-user goals)

3.Requirements Traceability Matrix (did all the planned stuff get implimented?)

4.Design Documents (GUIs plus interactions with black-box libraries or external packages)

5.Work Breakdown/Size Estimates (what are the pieces of the design that need attacked, what skills are required for each, who is attacking each piece)

6.Project Tracking (some automated way of allowing outsiders to see how it's going)

Colleges are teaching this. Companies who want government contracts are starting to do this kind of stuff. Would these software engineering techniques help OpenSource biggies stay on track and on time even though there is no financial interest to do so? The upcoming [cough] Linux kernel [cough] comes to mind.

Mark:

I'm glad I've never had my productivity forcibly "enhanced" with any of those processes (our software was for research). You skipped the best one, though, SASD (Structured Analysis, Structured Design). First you spend a year creating reams of useless diagrams and unreadable documentation. Then at the "end" of the project (which is actually the beginning) you try to write the code to fit the model you've generated. Unfortunately, without the benefit of bottom-up experimentation you now discover that the libraries you are using don't work the way you expected, and a whole different architecture would have been better. To get a working program, you then write the software conventionally and fudge up all of the silly diagrams to match.

I do believe strongly in documentation, though. Software is communication as much as it is algorithms. Even if you're writing device drivers or embedded systems, most of what you're doing interfacing with other people's work. On large projects, with lots of people or expensive resources, project tracking is what keeps half the company from sitting idle, while one overscheduled programmer contemplates suicide. Of course, I see no reason to force it on individual programmers. It's something managers should do, themselves.

License
(Score:5, Interesting)
by angst_ridden_hipster

First, I want to say that I've been using Nedit for probably seven years, and I've always thought it was the best thing out there. Someone mentioned that it eased the transition from PC/Mac to Unix, and that was certainly true for me. So thanks for your great work!

When I first used Nedit, I was working for a government-related agency, which had restrictions on what kinds of software could be run. Because Mark Edel hosted the source on a xxx machine, I was able to weasel around some of those restrictions. This, of course, was not before GNU but was well before the Open Source movement gained its groundswell of popularity. When the movement did gain popular momentum, it was looked upon with distrust and suspicion within the agency where I worked. There were offhand references to Communism and anarchy whenever people heard my enthusiasm. Likewise, when I wanted to give out source to some utilities I wrote, I was summarily shot down and chastised -- "this was paid for by the Government!" I was told (which was exactly my point; they just had a different conclusion as to what that meant).

So... how did you convince [organization] (or the [government agency]) to allow source distribution in the first place? Did you run into any difficulties when you decided to change the license? Did you have to get permission within the organization? If so, what did you need to do to get approval?

Mark:

Note how I have edited your question; and compare NEdit's documentation, web site, and copyright notices between those earlier releases and today. You might detect that ALL mention of the institution which supported a good part of the original development of NEdit has been removed. This was part of a peculiar (to say the least) compromise that we finally reached to appease a handful of government bureaucrats and lawyers, fanatically paranoid about liability.

Needless to say, it is difficult for me to discuss this subject without violating at least the spirit of this idiotic agreement. I hope someday I can tell the story, since parts of it are really unbelievable.

That we could get away with publicly releasing those earlier versions was not such an extraordinary feat. The institution in question is merely funded by the government, but run by a consortium of universities. Still, we were on shaky ground doing it. Even though sharing software is integral to scientific collaboration, their battle to release software freely still rages. Having been part of it, it's hard for me to think about it objectively. The processes that produced the various versions of the original license were bizarre. When it came to getting the GPL, I mostly just outlasted the opposition. I had little to lose, since I didn't want to put further work into nedit under the old license. Even so, it took a year of being shunned, misdirected, lied to, and even vaguely threatened, to make it happen.

Design and testing
(Score:5, Interesting)
by Grab

One of my pet peeves is lack of design, documentation, commenting and testing. I'd appreciate your views on how much you believe these are necessary. Do you design before you code, or do you just dive in? Do you like to provide loads of support documentation for users and co-workers, or are you of the "the-code-is-my-documentation" school?

More importantly, how do you enforce any such standard on your project, given that you've got other people submitting you code patches which may be technically perfect but visually obfuscated? And has this caused any friction, since managing coders is a "herding cats" scenario, and criticising someone's coding style is often taken very personally?

Mark:

I feel strongly about those issues, too. As I said earlier, I think software is as much about communication as it is about algorithms. The essence of programming, to me, is breaking down a problem into components, until each component is easily understood on its own. I know I have succeeded when I can clearly describe, in words, what each component does, and how it does it.

The coding standards in NEdit are much higher than what I see in other Open Source projects. That causes friction between myself and many contributors. It may well be that what I am doing is wrong. As I mentioned in a previous question, there is an almost mystical power in having a user base willing to test and assist with debugging. By exercising my gatekeeping too early, I may be stifling experimentation and making unnecessary work for myself and for the contributors.

The question of design is a bit more complicated. NEdit began as a working, throwaway, prototype for some data analysis software, and evolved slowly and continuously into its current state. I wouldn't recommend that as a design methodology, but it is a common course for Free Software projects to take. Through all the modifications, it is still very manageable code; nowhere near the hairy edge of maintainability around which much "better-designed" software seems to hover. Incremental projects like these openly defy traditional software engineering wisdom. Why do they work? Small size may have something to do with it, and the magical effect of "the Bazaar". From my own experience, I think the difference might just be commitment. Simply taking care to maintain the modularity and comprehensibility of the code may be all it takes to stop it from plunging over the waterfall.

-- Mark Edel

Comments

Congratulations

[Mindwarp]

A very well written article.

I can wholeheartedly agree with Mark's comments about the need for good coding discipline, communication and documentation within a project. In my experience it doesn't matter how brilliant your developers may be; if there is a lack of discipline when it comes to design and documentation then it is extremely hard for that project to succeed. As an example, my current project (a global system for a large investment bank) started out as a group of smart but inexperienced college graduates. Every one of the initial project releases was plagued by 'show stopper' bugs which used to take weeks to track down due to the disparate coding/design styles and lack of documentation. Over the past two years the project has, due to the efforts of the senior developers to instill the required discipline in the team, matured to the point where releases are now extremely close to 100% reliable. The bottom line is that we now have happy users, happy developers, and happy pay-checks come bonus time!

For further reading in a similar vein, I can recommend the "Software Engineering Classics" boxed set (yes, I know it's Microsoft Press. Please don't hold that against them, as these books contain much useful information and insight into Project/Team leadership within Software Engineering.)

The individual books are:

1. Debugging the Development Process Steve Maguire

2. Dynamics of Software Development Jim McCarthy

3. Software Project Survival Guide Steve McConnell

The ISBN number for the box set is 0-7356-0597-1.



--

Damn he's sharp

[Greyfox]

His comments on design and testing in particular were dead on. Just recently in an interview I had a company raise an eyebrow at my relative lack of experience with structured design methodologies. In fact, in a decade of experience in the IT field, I've yet to have a company request, much less mandate the use of any academic structured design methodology. I suspect that this is due to the fact that, as Mark suggests, you end up generating reams of documents that you don't adhere to when you finally hack the program up in the end.

At the same time you do need a certain level of design before you start coding, or you'll end up rewriting everything over and over again. Most programmers I know do some ad-hoc design work before they start coding. You really need to understand the problem before you can break it down into managable pieces, and most programemrs I know seem to use the "design" step to do that. Enforcing at least some level of design is a hard task for any project, commercial or Open, but it is a necessary one if the project is to succeed.

Coding standards

[Coz]

Amen, Brother Mark!

I hate reading half a dozen files and having to figure out half a dozen different sets of personal coding quirks. I've gotten so I can almost identify contributors to some projects by their coding style - which is not, to me, a desirable characteristic.

When I'm leading a development team (on crass commercial software development) I try to get some standards for comments, indenting, and naming conventions in place early on, and point out discrepancies during code reviews. Yeah, there's more typing when you have to name a class for the piece of the system it's in - but it's a heck of a lot better than having the C++ preprocessor "pick" a class with a colliding name (or even worse, having some static data element collide).

Folks may complain about the coding standards you hold them to, but it's obvious from the quality of the product that it's working. Bravo!