Making Software Suck Less, Pt. II

Not long ago, chromatic wrote about one aspect of the quest to create software that doesn't suck. This time he's back with a proposal that's simultaneously practical and idealistic. Namely: If you as a whiz at a certain language or other aspect of programming want the world of programming to be better, you can help other people become programmers, or better programmers, with well-considered instruction. I hope someone in the Computer Science department at a high school near chromatic gives him a call, because he outlines here something more important than a "learn to program" curriculum.

Making Software Suck Less, Pt. II Coders Code Whether You Like It Or Not The insidious thing about Free software, the really subversive part, is that it takes so little to start writing it. Anyone with a few tools, spare time, and the wherewithall to start hacking can. Witness the long listings of low-version-number IM clients and MP3 players on software announcement sites. People like to code.

Every new project represents a coder who wants to write free software, but chooses not to work with an existing project for whatever reason. How many more people would like to contribute but don't know where to start? Hundreds, even thousands of free software projects could use another coder, some testers, and someone -- anyone -- offering suggestions and attaboys. For each veteran programmer, battle-hardened and wizened by experience, a dozen novices spend evenings honing their skills.

Mentoring Beginning Programmers The obvious solution is to match availability with opportunity and enthusiasm with experience. The free software community can produce better programmers by giving new recruits mentors to emulate. It offers the possibility for programmers to learn by improving existing projects, instead of reinventing wheels. This doesn't require expert programmers. It takes people with practical experience, patience, and the willingness to invest time in another person's education.

Though this article draws from experience with free software, there's no reason similar procedures could not succeed in commercial settings. Computer science sophmores (and higher) would benefit from internship programs organized similarly.

The Usual Suspects Candidates for mentoring exhibit a combination of at least three different characteristics. These are expressable as three different archetypes: the self-taught hacker, the computer science student, and the new programmer. Individual personalities and experiences also come into play. These are gross generalizations, but serve the purpose of categorizing the types of information to present.

Self taught hackers learn by experimentation. They dissect existing code, copying and modifying implementations. Their technique gradually coalesces from dominant influences. Hackers often amass a library to improve their education. One danger for members of this class is that they may pick up questionable practices from questionable code ("cargo cult programming").

Computer science students learn theory and are expected to teach themselves various languages and implementations. (This category does not cover degrees like 'information services.') The education covers ideal solutions, often emphasising aesthetics and mathematical perfection. Students may not be exposed to practical issues in specific languages and techniques, focusing on working code.

New learners enter the world of coding with a strong sense of need. Management might pick a hapless victim for a project of undefined scope. A home user may just wish to do more with her computer. People in this category often don't know where to turn. They may not all grow into dedicated coders, but can fend for themselves with some guidance and direction.

Besides these categories, recruits may include artists, musicians, writers, and testers. Though direct mentoring from programmers may not awaken nascent coding abilities, these folks are certainly welcome. Even experienced programmers new to the idea of free software or to a class of software can benefit from directed guidance. (Many people in this position just need to see how to contribute.)

Course Requirements Mentoring must provide examples of secure, clean, and idiomatic programming. It should also include issues related to the practice of programming -- time management, software design, tools and classes of tools, and project leadership. Additionally, mentors should introduce their students to a network of peers, whether on IRC, through mailing lists, or in user groups.

Benefits of Mentoring Besides the warm glow of humanitarianism, being a mentor bestows personal benefits. Your project immediately gains an extra set of hands and eyes. Each new person brings a fresh point of view, with different expectations and stories for the code and unique experiences. At first, the additional overhead of explaining will slow progress; wise mentors invest slightly more time to turn these lessons into improved documentation and tutorials. After a short period of training, the project gains another person familiar with internals.

The discipline of explaining your personal technique, especially the more reflexive elements of your coding style, will also improve your skills. ("Why do I do it that way?") Describing system architecture and complex codes in words forces you to organize your thoughts. It can even clarify future design decisions. Your student must be free to ask questions and to challenge your assumptions. Either strengthen your arguments or discover a better alternative.

Leaving the Nest If you've mentored correctly, at some point you'll have taught every lesson you can. The student will have met the right people and will have tasted the fruit of hard work. Your example will still teach him, but he will make his own way. You will be peers.

Prepare for this from the beginning. Allow the student to lead subprojects with the freedom to make mistakes but the supervision to produce working results. Provide opportunities for personal growth. Your job is to remove obstacles from the path of enlightenment. Writing good software is hard, but teaching people to contribute is very rewarding.

Comments

Hey CmdrTaco!

[Shoeboy]

Rob,

I'm willing to be you mentor.

I've seen the database structure for slashcode. I know you need some help. I'm willing to offer it.

Our first lesson will be on denormalization for better performance. I'll try to be a patient and dilligent instructor, but the fact that you fetch the sig from the users table for every comment rather than storing it in the comments table may render that impossible. It's quite possible that I may have to beat you. Particularly when I look at some of the 3 and 4 tabe joins you do for really common operations. I know Hope College isn't much of a school, but surely at some point they explained that joins were slower than single table selects, right?

Anyway, I'm more than willing to help you with your database design and implementation problems.
Drop me an email.

--Shoeboy

Re:Hey CmdrTaco!

[Shoeboy]

A select is slowed down by joins, true, but its also slowed down by the size of your tables.
That's more true for the length of your table then the width. As long as you're doing index scans rather than table scans, width isn't all that important. In fact, on systems that support page level locking but not row level locking, it's sometimes in your interest to widen columns.

The whole purpose of normalization is to reduce the size of your database by eliminating redundant data. Reducing the size of the data searched will then speed up the selects.

But (assuming that your database system uses b-trees to store and index the data) it won't speed index searches by much. And if you're doing something besides an index search on a large table, stop.

For example, if you are an online music store and you have 250,000 music titles in your database, but the vast majority of those titles comes from 7 different record companies, it doesn't make a whole lot of sense to repeat the record company name in each title row. I would join the little table to the big table and index the record company id in the title table itself.
Actually, having a record company id is redundant data. Using the name as the primary key is the proper method. ID's are actually non-relational. It all depends on how you query your database though. Data warehousing schemas and oltp schemas are very different beasts.

I think slashdots problems are more related to mySQL's locking mechanisms. I have to deal with this sort of thing myself. IMHO, if you have an online database at all, you should completely separate the "read" database from the "update" database. In other words /. should have all the comments posted to another database and implement a once-per-minute table swap with the table used to display comments.

Yeah, I've been made aware of this. I took a look through the MySQL manual, and I'm shocked that people describe it as a real database. It's barely more advanced than Access. I personally recommend Microsoft SQL server for this sort of application. It's simple, fast, full featured, highly tunable, quite stable (versions 7.0 and above, 6.5 and below is a stability nightmare) and scales very well up to 8 processors. On commodity intel hardware, it's the fastest database there is. (numa machines don't count) Once you outgrow the capacity of a 8-way proliant, it's time for DB2 or Oracle.

It's a pity the free databases all suck so much. PostgresSQL has actually improved a lot, but the pricey overhead for new connections makes it unsuitable for client-server environments.

--Shoeboy

Re:Hey CmdrTaco!

[Shoeboy]

Hi,
Your comment on denormalization demonstrates a lack of real world experience. Denormalization on databases where reads massively outnumber writes (which is slashdot) is a very useful thing.
As far as making it difficult to make future modifications goes, this is complete B.S. As long as you encapsulate your data retrieval and manipulation through stored procedures (which slashcode doesn't do) all changes to the database schema are easy to make and very few modifications to front end or middleware code are ever required.
The simple fact is that no perfomance critical system should ever run in 3rd normal form. Period.
--Shoeboy

Re:Suck Less

[rw2]

Of course with most free software, since there is no one promised a product that withdraw their money if it doesn't happen there aren't many deadlines.


Except that tons of software is written as a kind of modern day duel. Why else would someone write a message board when ten already exist? To scratch an itch? Partially, but that only explains why they write it, not why they gift it. They gift it in order to win the acclaim of their peers.

In order to win said acclaim you must be current. The deadlines are therefore sometimes much more onerous because they are less tangible. More like a nightmare where you are terrified and running, but you don't necessarily know from what instead of a deadling where you are running, but it's only because your boss said so and he has no spine, no brain and no power. The worst he can do is fire you and then you'll just go somewhere else for more money.

--

Don't teach them to program.

[Black Parrot]

Don't teach them to program at all -- teach them to solve problems. After you have a sensible solution strategy, expressing it in the syntax of a given programming language is trivial (boring) work.

The problem that I see over and over and over again, among both students and "professionals", is that they sit down and start throwing code at a problem without knowing a solution strategy for that problem. And for many of these people, if you try using the Socratic method to bring out their solution strategy you'll find that they not only don't know what strategy they're trying to encode, but that they also don't even want to talk about strategies -- they want to talk about the details of the incorrect code they've already written.

I'm not against hacking, and I certainly don't think every program anyone writes needs to be supported by an engineering discipline, but I do feel, very strongly, that people need to think of programming languages as a tool for expressing a solution to a problem, not as being the solution itself.

--

My own efforts to help other programmers

[goingware]

First, my effort to improve the quality of free software can be found at the Linux Quality Database. While the database hasn't started being written yet, I have started writing articles on the top of quality assurance and writing better free software. The first such article actually to be posted is Using Test Suites to Validate the Linux Kernel.

For quite some time before that I have been writing GoingWare's Bag of Programming Tricks, a collection of articles on the business and practice of programming. Of most interest in learning how to program well would be:

• Study Fundamentals Not Tools APIs or OSes
• Properly managing memory returned by transcode() in the Xerces library (while about an obscure problem, it illustrates more general principles in a useful way)
• Secrets of the Debug Meisters: MacsBug Tips and Tricks (MacsBug is a Mac OS assembly debugger)

I worked on an article on C++ programming style that is not done yet, and has some errors in it. But I'll post the URL here with the understanding that you're to take is as a preview and not gospel truth:

• Pointers, References and Values - Passing Parameters, Returning Results and Storing Member Variables, with Musings on Good C++ Style

Finally, if you program in C or C++, you need to be using one of the following tools, appropriate to your development system and platform:

• Bounded Pointers for GCC
• Spotlight for Mac OS PowerPC
• BoundsChecker for Windows
• Purify for Windows and Unix (but I think it doesn't support Linux)

If you use Java, you don't have to worry about dangling pointers, but you do have to worry about memory leaks (quiz: why do you in a garbage collected language?), deadlocks and so on. For that we have:

• OptimizeIt

Don't let another day pass without availing yourself of one of these tools. I know most aren't Free Software and some are very expensive. I want to tell you that the money I spent on Spotlight is some of the best money I have spent on anything in my life. I wouldn't dream of shipping a Mac application to my clients unless it tested completely cleanly under Spotlight; on the other hand, the first time I'm given a client's code to work on, Spotlight usually reveals a multitude of sins.

Mike

One simple "trick" for not sucking

[Wordman]

Most of you will read this suggestion and snicker at it. I know I did when I first heard it. I've since learned how blind I was.

There is one simple thing you can do to make your code a lot better. It takes a bit of discipline, but once you do it for a while, it becomes second nature. It is this: whenever you write new code, make sure that you walk through every new line with a debugger. That's it.

At first blush, this seems kind of like a "duh" sort of step. After all, most people run their code through a debugger to check it. The key to it, though is the phrase "every line". Put breakpoints in every branch of the code you write. When you get into that branch, remove the breakpoint. When you think you are done, check for any breakpoints that are left. I was extremely surprised the first time I did this. I thought I had been thourough, but I hadn't checked three different branches. And there were bugs in two of them.

What I have learned after doing this for a number of years is that programmers pay more attention to the important part of a loop or branch and ignore the fringe cases. What ends up happening, then, is that the fringe cases contain a proportinally large number of the bugs. Worse, because they are fringe cases, they don't happen often, leading to bugs that are hard to reproduce.

Re:Suggestions for better software

[Vanders]

For a troll, I have to agree with you. Open Source software very rarely seems to go through a proper build, test, fix stage. They just throw the software out to the users and let them find, and in a lot of cases fix, the bugs. This isn't good.

I'm a software tester by trade, and I code in my spare time. There are some cardinal sins that any peice of software can commit. They tend to be:

• Not having a clear set of features that you want in your code for at least version 1.0
• Not writing proper specs before any code is written
• Not having testers, or having testers and not listening to them
• Not having a defined roadmap or plan. At a minimum, this should cover the planing, and upto the second build of your software

Now on smaller projects, it's usually fine to sit down and start coding. With larger projects, you're going to be royally screwed about 50% ofthe way through your project, when you find you'll have to re-design part of your project to add functionality you didn't know you needed.

If you don't have testers (And no, outside of unit testing, the coder cannot teste their own code properly!) your software will be full of bugs you just don't know are there, until your users start using the software under normal conditions. You'll then find you need to recode large parts of your software to fix even small bugs.

Release early, release often, is good provided you have proper controls in place before your first release!

Re:Suggestions for better software

[Lucretius]

The problem that you speak of is not new at all, and is definitly not confined to the realms of computer programming. When it comes right down to it, nobody wants to do the dirty work that is required to get a project done (i.e. bug squashing, documentation, general project managements) , they would rather go out and do the new and fun stuff (which in this case is adding new features).

This is just a fundamental flaw in human nature. People want to get all the glory and credit and nobody wants to do the background work that is required to make something actually work. Look at politics... everybody wants to be president, nobody wants to be his aides that actually run around and get the shit done (well, some people do, but most don't).

This was also a problem in ancient Rome, when you actually look at it. Everything used to be paid for by donations from the rich folk. However, everyone wanted their money to pay for something big and wonderful that everybody loved (such as theathers and circuses, etc), but nobody wanted to pay for the smaller bits of infrastructure that held everything together, as they wouldn't be remembered eternally if they gave money for the roads to be resurfaced or for general maintenance of the aquaducts.

Its the same thing with programming. Nobody thinks they'll get any credit for the bug fixes that they work out that makes the product actually work, they want to be known for the feature which they implemented (even if nobody actually needs it or wants it in any way).

The Way of the Bard

[Ricdude]

A more applicable model might be similar to the way of the bard:

Seven years learning
Seven years playing
Seven years teaching
Repeat.

The idea is that the would-be bard spends some period of time under the apprenticeship of one learned in the art. After acquiring some basic skills (melody, harmony, scales, etc.), and some examples of how to combine those skills (a repertoire of songs), the bard is then sent off to earn a living in the world. While doing so, the bard experiments with new ideas, merging them with the body of knowledge acquired from the master. After some time, the bard takes on apprentices, and shows them what they've learned from their master, and from their own experimentation.

All too often, what's lacking in the free software world of half finished IMs and mp3 front ends, is the "combinations" of those skills. A lot of these programmers barely understand pointers and event driven programming, yet they're building full GUI programs as a method of learning more about how to program. Frequently, they bite off more than they can chew...

Some software has to suck...

[11thangel]

There are too kinds of software that you'll find on download sites. Software that is intended to get the job done and software that is intended to help the programmer learn wtf he is doing. You'll very often find more of the latter than of the former, but each plays its own role in the programming community. If you see a piece of software that sucks, dont just delete it, email the author and tell him that it sucks. Let him know what you think is good and bad, so he can learn. Who knows, you may be helping out a future alan cox.

Suggestions for better software

[atrowe]

I've got a suggestion that I believe will help improve the quality of software, especially open source software. It seems that whenever an open source project is out there, coders seem to be overly focused on adding new features rather than improving/fixing existing ones.

Have you tried to use Mozilla lately. It's loaded with bloat and crappy features I'd never want to use, but when it comes down to it, Mozilla is absolutely horrible as a web browser. When the Mozilla project was begun, I thought that it was an noble idea, and was eager to start contributing code, but I soon realized that human nature goes against the open source model by seeking out self gratification and promotion, often against the benefit of the group as a whole. This is an inherent flaw in the open-source philosophy. Everyone is seeking their own selfish gratification in being recognized for adding a new feature to a program, but no one is willing to try and make the existing features work properly.

I'd like to see this issue addressed more often in programming circles, because something desperately needs to be done.

The problem with bad code

[Faizdog]

Is partly due to the inexperience/naiveness of the coders. But another factor IMHO that i've noticed in my experience has been big is simply human laziness. So many coders I know don't document, "can't be bothered to", they just write code that works, but don't worry about "elegant," algorithimically well thought out code. You can never just sit down and begin to code. And please don't tell me you can, I can just sit down and crank out a buncha programs too, but I mean well thought out plans. ( I dont' mean the BS stuff like Requirement's Engineering, Software Architecture, etc. although that stuff is sometimes useful), but just to sit down and think about what you wanna do and what's the most efficient way to do it.

I'm not trying to be troll at all, so please don't think that. I'm just saying something that I've noticed. Plus I can't emphasize the importance of documentation. Currently I'm expanding upon a project that has a few thousand lines of code, and it's hardly documented. Hacking the code and trying to figure out what it's doing is tedious. Please document, it doesn't mean you are a bad programmer, actually means you are a good one.

I tried this with assembly coding, once

[kyz]

I'm on several mailing lists, including ones regarding oldskool music ripping. One particular person had written a ripper in a high level language, and wanted to know more about assembler language so he could make it rip much faster. I quite happily offered to help teach him assembler.

He understood most of the assembler instructions, by my comparing them to the high level constructs, and showing how they are built. However, the point he got really stuck on was optimisation (which is really the point of asm coding nowadays), particularly memory access. He didn't see why accessing unaligned data was bad. So I showed him how memory is logically addressed versus the physical bus requests the CPU has to put out. I think this was the straw that broke the camel's back. He said 'thanks, but I think that's too hard for me', and he's now a Visual Basic coder. (No, really. This isn't a joke.)