Programming Things I Wish I Knew Earlier

theodp writes "Raw intellect ain't always all it's cracked up to be, advises Ted Dziuba in his introduction to Programming Things I Wish I Knew Earlier, so don't be too stubborn to learn the things that can save you from the headaches of over-engineering. Here's some sample how-to-avoid-over-complicating-things advice: 'If Linux can do it, you shouldn't. Don't use Hadoop MapReduce until you have a solid reason why xargs won't solve your problem. Don't implement your own lockservice when Linux's advisory file locking works just fine. Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job. Modern Linux distributions are capable of a lot, and most hard problems are already solved for you. You just need to know where to look.' Any cautionary tips you'd like to share from your own experience?"

Comments

Comment your code

[Nkwe]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do. Remember that comments are not for describing what the code technically does (that is what the code is for), comments are for what the code is intended to do. Try and comment the decisions you made when developing the code, specifically why you took the approach you did and why you didn't use other options.

Re:Comment your code

[kantos]

Commenting is not enough... you need to make sure you clean up the comments in a file before resubmitting it... dead comments can be more dangerous than dead code... as dead code at least doesn't run... dead comments mislead the person following later into believing a lie a lie that could potentially have major impacts on the software. Lastly... write meaningful test cases... if your tests only prove the happy path, that's OK... but if they prove that the happy path is the only path.. that is better.

Re:Comment your code

[Anonymous Coward]

Commenting code isn't enough, it's just a small part of the design and documentation process. Comments are there to tie the code to the relevant part in your design document, which really is a part of programming people should put more effort into.

Re:Comment your code

[Xtifr]

Put enough comments in your code so that five years from now you (and others) can remember what you indented [sic] the code to do.

But not so many that you (or others) will find it more work than it's worth to change the comments when the code changes.

I prefer code with no comments to code with actively misleading comments, and I hate code with no comments! :)

Re:Comment your code

[negativewashout]

This is the best, IMHO. I, too, have looked code and gone, "WTF was this idiot thinking???? ...Oh wait... that's my code."

And agreed - dead comments have to be updated. I didn't always do it perfectly, but I pretty much got in the habit a few years ago of commenting as I go, right by the code in question. One of those times when it's good to be a bit verbose.

Re:Comment your code

[Manip]

I could never find the correct degree of commenting. Didn't help that in school they made us comment in an idiotically verbose way to the point of "this is a for loop to find needle in haystack" and we lost marks if we failed to. My reaction to that level of stupidity was to lose interest in comments entirely which obviously later bit me in the butt.

I really need to find some kind of "idiots guide to when to comment and when NOT to comment."

Re:Comment your code

[sourcerror]

Think of comments as an API documentation*. If something complicated is going on, I usually include several usage example as well. Just look at the documantation 3rd party libraries, and try to follow that granularity and style.

Re:Comment your code

[Richard Steiner]

Sure, assuming that everyone who ever uses that code is using the same code repository.

I've had to modify code professionally that was written over 30 years before I encountered it (older Fortran 66 stuff), and the comments in the code were invaluable to me. The folks who wrote it had no idea that their code would be running at another company when it was written.

Re:Comment your code

[ZorroXXX]

If the comments are out of date, then the code has been modified by someone who was too lazy to update the comments.

When code and comments disagree, both are probably wrong. -- Norm Schryer

Re:Comment your code

[gringer]

My current annoyance with python:

1. Copy code from one section
2. Paste code to another section
3. Different indentation changes code function
4. ...
5. Bash head on keyboard

Re:Comment your code

[Just Some Guy]

In other words you document the intent of every paragraph of code, and a high level explanation of how it does it.

No offense, but I despise that style of commenting. I can tell what your code is doing by reading the code. What I don't know is why you're doing it that way. Why are you looking for needles? Have you proven needles are the only thing in your haystack made out of iron so that the code won't detect nails, screws, and hatchets six months from now? Is there a reason your haystack is a list and not a binary tree? These are the things I need to know to maintain your code. I don't need to be told how a loop works.

Re:Comment your code

[AuMatar]

Yes, because what's clear to you is going to be clear to everyone else who ever reads it. And it'll be perfectly clear to you in 6 months or 6 years when you try to alter it. Yup.

There is no such thing as self documenting code. If you ignore comments and/or documentation, you're a shit poor programmer and a blight on the profession.

Re:Comment your code

[russotto]

There's a reason architects use blueprints.

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

Re:Comment your code

[Lotana]

Solution: Stop mindlessly copying and pasting.

I lost track of the amount of code I've seen that is a straight duplicate from some area that does vaguely the same thing. Then all the variable names don't match the new context, comments are not applicable and assumptions are not necessarily correct. Doesn't matter what language it is written in.

Of course the person that does it never bothered to look through it and only relied on compiler error messages to detect the obvious "not declared" errors. What ends up is a mess that is nearly guaranteed to have bugs.

If tabbing the new pasted code into place forces you to actually read it and think about what you just did: then it is well worth it!

Re:Comment your code

[DavidTC]

That's what the GP meant. Comment why, not how.

You only need how when you do something that people might miss, such as a switch fall-through, or not understand, like some tricky Boolean operation.

Everything else should be about what the code is attempting to do, and why it's trying to do it that way. When that's obvious, you don't need any comments at all. (Although function headings are still a good idea.)

Re:Comment your code

[MadKeithV]

There's a reason architects use blueprints.

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

Yes it does.
Consider all the useless implementation work you've done while your design was wrong (before you reworked it). It's just as important in software as in building.
Also, having blueprints / a design allows you to separate the work between many different teams with clear, agreed-on-beforehand interfaces between the sections that each team is working on. This is something that a lot of developers seem to miss (I know I did for the longest time) - perhaps because a lot of us tend to prefer working and thinking alone?

By the way, working out a design is not a solo 1-hour effort. It's just as much "development" as actually writing the code. Often you may actually want to write code while doing it just to make sure that what you are thinking of does work. But that's not the final software, it's still just the design. Don't give in to the temptation to "improve" that design prototype until it ships as finished software though...

Re:Comment your code

[erixm]

Come to work knowing what you are doing. This may mean cramming in your off hours. Don't say that you don't know how to do something. Say that you do and then learn it!

Please ignore that. Honesty is always better - and if your team doesn't welcome honesty, try to change the team's attitude. If you trust each other your results will be better. If you don't know how to do something, say you don't and THEN learn it.

Lesson #8

[pedantic bore]

Don't ask for advice about programming on slashdot unless you have a pile of salt grains ready.

Re:Lesson #8

[thoughtsatthemoment]

That applies to other sites as well. So many people are trying to preach than genuinely teach. The internet is great for seeking specifics, but for insight on programming, it has to come from within yourself.

3 things I've learned.

[Anonymous Coward]

Sometimes it's easier and faster to code from scratch than it is to use off-the shelf software - especially in the age of "frameworks".

In that train of thought, its often better to toss and rewrite (or write new programs) than it is to extend existing programs.

It's easier to implement a whole new framework than it is to convince your boss that writing anew is actually faster.

Making use of a database

[sphealey]

A database is not a bitbucket. Re-building basic database functionality in an external app is not a good idea. Applications, frameworks, languages come and go; data remains forever [1]. Business logic is part of the database. If you find yourself adding more and more "application servers" to get performance than you have a fundamental problem with your architecture (and probably a fundamental misunderstanding of how databases work). While it is not impossible to learn and implement good data management/database development practices using Microsoft tools, such a result is seldom seen in the wild.

sPh

[1] Per Tom Kyte of Oracle, whose first database job at the Department of Agriculture involved working with datasets stretching back to 1790.

Re:Making use of a database

[WarwickRyan]

> Business logic is part of the database.

I used to think the same thing, but that was before I ever had to solve a no-trivial problem.

What do I think now? Basically that business logic belongs in a domain layer, but that shouldn't mean that the database is treated like a black box. You should make use of constraints within the database server to ensure a base level of data consistancy. Default values (especially on auditing columns), correct datatypes, relationships, versioning and not-null constraints all belong within the database. Basically anything which can be supported without resorting to triggers or stored procedures.

Databases are excellent at sorting and filtering data. A massive amount of engineering talent has been invested in shaving miliseconds off of their algorithms. So let your database handle that. Don't filter in the application layer and then start complaining about performance. Need more capacity? Cluster. Throw more databases at the problem. Not more application servers. Make sure they're running on properly specified physical hardware, built to best match the actual usage pattern of the database.

Re:Making use of a database

[WarwickRyan]

Main reason is that you're hiding logic in the data layer. You don't want to do that. You should encapsulate all business logic in one place. Which, for systems of a certain complexity, is a domain model / layer.

Stored procedures are generally hard to test, and, unless you're using something like SQL Server's CLR integration the available functionality is limited. Not being object oriented, it's harder to manage complexity.

I've built some pretty complex logic in the past with SPs, and they're extremely useful when you need to do some data-only task with maximum performance. In my experience you'll usually get the task done better and faster (in coding time) with your programming language (assuming that you're working with a domain model etc etc).

Triggers are a similar thing, though it's good to treat them as a case of "I'll only use them if forced to" or if the alternative involves a massive amount of work (and if it does, then the triggers are the least of your problems) OR big risks to data integrity (think database which doesn't support cascade deletes).

Re:The hard way is more fun

[hedwards]

Eh, isn't that the basic difference between a programmer and an engineer? When I took my programming courses, there was a distinct focus on writing code with the smallest amount of new code possible, using preexisting libraries whenever possible and if there was any suspicion that the code might be useful later to split it off into a subroutine or method of some sort as done in the particular language. Sure it's a good idea to be able to write your own libraries and solutions, but it's a waste of time once you get to the point where you know how to do it. And generally using a library that's shared by other projects is much less likely that you're going to stumble on an undiscovered bugs with your program.

2-port programs, Linux, PIL, expensive hardware

[tepples]

If you are writing a program that touches more than two persistent data stores, it is too complicated.

I disagree. Is a program too complicated if it has 1. input, 2. output, and 3. logging? Is a program to prepare images for an online store too complicated if it reads 1. raw source images and 2. an overlay image and writes 3. finished images?

If Linux can do it, you shouldn't.

That'd be fine if we all ran Linux. But in an organization that already has to run Microsoft Access for other reasons, we have to take Windows into consideration. And I don't think Ted Dziuba was talking about just using Windows as a shell to run Linux in VirtualBox OSE either.

Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.

Our programmer is far more experienced in Python than in bash, and if I felt like it, I could benchmark PIL against subprocess.Popen(['convert', ...]).

if the physical machine is not the bottleneck, do not split the work to multiple physical machines.

Yet PC game developers split a 4-player game across four PCs when one could do, and increasingly, PS3 and Xbox 360 game developers are following the same path.

It is far more efficient to buy your way out of a performance problem than it is to rewrite software. When running your app on commodity hardware, don't expect anything better than commodity performance.

If you are writing software to be used internally, sometimes springing for better hardware is worth it. But if you are writing software to distribute to the public, you can generally assume your customer has commodity hardware unless your software costs at least 1000 USD a seat.

Thing I wish _others_ would know

[melted]

Do not make things super-modular and generic unless they 100% have to be. In 99.9% of the projects no one, including yourself, will use your stupid dependency injection, and logging / access control can be done just fine without AOP. Don't layer patterns where there's no need. Aim for the simplest possible design that will work. Don't overemphasize extensibility and flexibility, unless you KNOW you will need it, follow the YAGNI principle (you ain't gonna need it).

Re:Thing I wish _others_ would know

[Anonymous Coward]

(too lazy to log in)

hear hear.

i'm working in an over-architected codebase now and all the Design Patterns thrown in haven't done squat for making it flexible or modular,
they've just made it obtuse.

This is te problem with Linux

[bogaboga]

"Modern Linux distributions are capable of a lot, and most hard problems are already solved for you. You just need to know where to look."

First off, I must say this piece says a lot about the Linux ecosystem. Specifically that this system's documentation is anemic at best. Why won't we have something like:

"What do you want to do?...with an associated answer...this kind of arrangement surely cannot hurt the Linux ecosystem.
 

Overengineering can be a good thing

[caywen]

You know, I find that as I get older, I am able to avoid overengineering things a lot better than when I was twenty something. There's nasty effect, though. I'm learning a lot less in depth about systems than I normally would.

Overengineering is terrible for a project, but it often is highly educational.

Re:The hard way is more fun

[jellomizer]

Well if you are coding professionally. You are wasting time going the hard route unless there is a real good reason to do such. Newbees want to show off that they can do all these things. Mature programmers know they can but if there is an easier way then go for it as it gets your project done faster and you look better to your boss who can give a rats ass on how you did it. Just as long as there isn't any consequences in the future.

Re:Don't code in C

[topham]

No. Sorry, I strongly disagree.

C is like a swiss army knife that can be used to assemble a chainsaw, a sword a lightsaber, etc. You'll have to carve out all the pieces, file them down and put them together yourself, but the swiss army knife will help you a long the way.

But don't forget the bandaids.

Re:until you find out why Python doesn't do the jo

[daveime]

Look, the PHB usually wants the code to run on HIS server for HIS use only. That's what he pays you for. Not to code it in the most cross-platform friendly language-du-jour and take 2 years to iron out all the bugs.

He doesn't give a shit if it'll run on the Xbox 360 or a Linux-ready Dead Badger.

And neither should you, unless you are some kind of anal retentive who spends all day arguing the merits of absolute versus relative positioning, fixed vs percentage tables, and worrying whether your code will run on every machine conceived in the next 50 years.

I have news for you, it won't.

Hell, I got an N900 6 months ago, and it's already EOL'd as far as updates to the OS are concerned.

I suppose I could wait till there's a port of Android or something, because the three coders on the project are doing a fine job, in less than a year I'll have the same functionality as a 3210.

Comment your data too!

[LihTox]

I'm in a different boat from most commenters here, I think, because I am a scientist writing simulations; some simluations run a long time and create a lot of data which would be costly to reproduce, and what I wish someone had told me early on was that I should comment my *data files*, not just my code. Each file should include the exact parameters used to create it, an explanation of what each column represents, and preferably there should be a way of knowing what version of your simulation code was used to create it. A couple of times in grad school I had toss out months of data after I discovered a bug in my code, and didn't know when the bug showed up and which data was affected by it.

(I'd welcome other advice from simulationists too; I've never had an advisor who was particularly programming-savvy, even though programming was always a large part of my research, and so I always had to make it up as I went along.)

Re:Comment your data too!

[SplashMyBandit]

Yep. I learnt that lesson too when recording astronomical observations for research. You must design your software to *automatically* put in *all* the relevant information - just in case you need it. Rely on a human to put it in and you get mistakes that take a huge amount of time to untangle (if at all possible, sometimes you just don't have enough information to deconflict bad human-recorded values). My general rule is, "Never lose information". Once information has been thrown away it becomes a huge effort to try reconstruct it. Better to always keep as much as you can around to begin with. This is where using database to record 'state' stuff makes sense, and then have that point to the data files that are too big to go in the database. Then you can slice-and-dice the database tables as you need to. As a researcher I avoided writing software to existing databases (I now love Postgresql) but I wished I'd used Postgresql a lot earlier.

Re:Comment your data too!

[Anonymous Coward]

I'd welcome other advice from simulationists too; I've never had an advisor who was particularly programming-savvy, even though programming was always a large part of my research, and so I always had to make it up as I went along.

Use subversion/CVS/git/whatever. Do so *frequently* and not every 10 months.

Keep the manual updated - too often people have left our group rather suddenly without really knowing beforehand, and the only documentation left was in the code.

Use LAPACK/BLAS wherever possible. 20 years from now these libraries will still be around and *much* faster than your own code. And they will still compile (not true for some of the code I am working with - needs special old compilers because of some "clever" hacks by people long gone)

Do not only write "programming" comments but also describe what the physical background/model is. Provide citations of the relevant articles.

If you produce lots of data then make use of unformatted files (or whatever C calls them). Saves terabytes in the long run, and converting them to ASCII when necessary is trivial.

Do sanity checks wherever possible.

Do integrity checks wherever possible.

Write meaningful error messages. They *must* include where the error happened (subroutine e.g.), what type of error and *expected behaviour*

Provide some sort of debug mode where people can step through the program and review intermediate results (*You* can use a debugger. The next guy maybe not).

Make sure that you/users can include comments in the input files.

Save early, save often. That means write the program as modular as possible. If possible, split into subprograms chained by shell scripts and write the intermediate output to disk.

That's all I can think of for know...

Emotional Things I Wish I Knew Earlier

[daseinw]

we could not figure out whether the author was an incredibly elaborate troll or just a run-of-the-mill idiot.

Reading this comment of his reminds me of something I read recently:
Physicists stand on each other's shoulders. Engineers dig each other's graves.

I've never understood why so many software developers feel the need to disparage one another in an attempt to prove their intelligence/superiority. There are plenty of tough problems out there and we all can learn something from one another, no? I've definitely been guilty of this in my tech career but lately I'm wondering more and more, why does the person who has a different solution always have to be an "idiot?" Why isn't he/she just someone who has a different take on solving this particular problem?

Now, I'm not saying that engineers do this more than any other group but out of all of my friends (some of whom are doctors, lawyers, teachers, etc.) it certainly seems like a more common event among software developers.

Re:Emotional Things I Wish I Knew Earlier

[azmodean 1]

I think there are several factors that contribute to this:
1. Programming is a very popular and easy to enter field.
2. It's actually pretty easy to get by as a programmer without really understanding what you are doing.
3. Regardless of how much you hear about it, modularity, reusability, and highly structured programming do not have good penetration in Software Engineering.
4. Because of #3, it is all to easy for otherwise competent programmers to paint themselves into a corner and generate software with really messy architecture and/or implementation.
5. Programmers OFTEN have to clean up after other programmers.

So, due to #1 and #2, there actually are quite a number of really bad programmers running around.
Due to #3 and #4, there are quite a number of otherwise decent programmers who produce working but unmaintainable code.
Due to #5, most programmers have ample opportunity to experience a great deal of pain from other programmer's incompetence.
Due to human nature, programmers tend to assume that all that bad code comes from #1 and #2 rather than #3 and #4.

And more speculatively and unrelated to the above:
6. Lots of programmers tend to hang out on Usenet, internet fora, mailing lists, and IRC, where harsh criticism is de rigeur, and internalize the habit of harsh criticism in their professional lives.

Re:Some tips from a C guy.

[spiffmastercow]

That's essentially what the K&R C book steps you through, and I'd say it's the best programming book ever written (or at least the best I've read). I don't get to do much in C these days, but the stuff I learned 10-15 years ago in C has made me a much better programmer. It's sad how programmers these days give you a blank stare when you ask if you passed something by value or reference.

Re:Oh if you find yourself repeating some code

[Haeleth]

Sadly, if you're working in Java then you will frequently be forced to write the same 8 damn lines over 70 times, and there will be literally no way to avoid it unless you resort to automatic code generation, which is a fancy name for using a separate program to do the copying-and-pasting for you.

Version Control

[BluePeppers]

Put everything in version control. Everything. EVERYTHING!

Well. You could skip /home, but I know a roll back of /etc has saved me a couple of times on config upgrades.

Remember that once code is deleted, you can't get it back. However, version control changes that. Version control is one of the most vital tools for anyone developing/working with a computer.

Oh and git rocks and stuff :)

Meaningfull messages.

[fahrbot-bot]

I don't know if this is still around, but a long, long time ago, I got this, and nothing else, from the shell when it died:

Assertion botch: This can't happen!

Nice.

The Truth

[TheCount22]

A few rules of thumb for a startup environment:

1. Don't overengineer! Overengineering wastes time on things that may never be used. Features should be customer driven.

2. Functions and methods should be as small as possible. You should make it an obsession to split methods and functions into the smallest possible components. Only then can you have good code reuse. Don't start thinking I will split it when I need it, you never will!

3. Never ever reinvent the wheel. Reinventing things that exist is overengineering.

4. Don't optimize ahead of time. When I say that I don't mean don't use a hash table instead of an array where it makes sense. I mean don't try to avoid exception handling or function calls or other minor optimizations. If it has an impact on readability don't do it. Optimization always comes last. Often you'll find there are only 1 or 2 "hotspots" in your code. If you spend time optimizing these "hotspots" after your application is built thats when you'll get the best return on your investment. Another gotcha with optimization is using technologies that can't deliver the level of performance you expect. You should test to make sure the underlying components you plan to use will perform as expected before you start coding.

5. Don't cram as much code in a single statement as possible. Every compiler I know about today will produce identical code whether it's one statement or 5 statements. It makes it hard to read so don't do it!

6. Allocate time for testing. No one writes perfect code.You want to give a good impression to your customer so don't skip this step.

7. Make unit testing an obsession. Always add unit tests for new code, it reveals errors in your code. When you find a bug in your code add a unit test to test for it. If in the future someone decides to rewrite some function or method you wrote because it's not elegant enough they will not reintroduce old bugs.

8. Don't rewrite code if possible. Refectoring is almost always easier and less error prone.

Re:One more tip

[radish]

Right - but a real programmer realizes that languages always come and go, that the real skill is PROGRAMMING not C# or Java or C++ or Python or whatever. If you have a real understanding of the fundamentals then learning a new language is usually easy (and often kinda fun).

Re:The hard way is more fun

[Firehed]

Hardware can certainly be the correct answer to the problem. Are you disk-bound? Then you need faster disks. Can't fix that in software. Saturating your pipe? You need more bandwidth.

Yes, you can do things to try and cut back on your use of those resources, but that's ultimately going to produce much less effective results unless you're at massive scale (read: you have your own datacenter) where a 1% performance gain frees up the equivalent of several hundred servers.

Hardware is also a good way to buy yourself a bit more time while you're working on fixing the real bottleneck, such as reworking an expensive database query. Sometimes you just need a couple extra weeks.

Don't read this post as advice to write bad software and only focus on vertical scaling, as that's not what I'm saying at all. But solutions that take you from 1-10k users are very different than going from 10k-1m users, and different still from 1m-10m. Trying to code for 10m when you don't have any users a) is a waste of time, b) is expensive (they rely on the assumption that you have lots of hardware anyways), c) won't work at all, and d) won't help you get users. Don't do dumb things, but users don't give a damn how well your software can scale if it sucks, and if they think it sucks you'll never find out how well it scales.

Re:The hard way is more fun

[Anonymous Coward]

Why in God's name the supposed "number one image processing solution for Linux" can't even handle JPG out of the box is beyond me.

Perhaps because the ImageMagick developers are professional enough to realise the best thing is to re-use the pre-existing, debugged, validated JPEG library from the Independent JPEG Group? Which is pretty much the approach this article is advocating.

Has ImageMagick improved?

[SL Baur]

Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.

I think the worst mistake I made as Mr. XEmacs was attempting to unify our graphics support to call ImageMagick libraries instead of the custom stuff we were using (and later restored when ImageMagick was backed out).

Does it work any better now? The last time I looked at display(1) a couple of years ago, it still wasn't close to long lost and patent challenged xv(1) that got shut down by the GIF patent war.

Re:The hard way is more fun

[danieltdp]

One build bridges and the other one codes. Oh, wait!

Re:social advice, especially applicable to geeks

[russotto]

There is no silver bullet. You are no Superman. You're not going to change the world.

I'm not, no. But some people have, and they may be reading this right now. How do you know you're not addressing the next Bill Gates?

Anyway, your advice is way off. The people with the most _social_ success are raging egotists and shameless self-promoters. There's right ways and wrong ways to do it, and the wrong ways lead to you being written off as a pompous ass -- but not being such will never lead to social success.