Programming Things I Wish I Knew Earlier

theodp writes "Raw intellect ain't always all it's cracked up to be, advises Ted Dziuba in his introduction to Programming Things I Wish I Knew Earlier, so don't be too stubborn to learn the things that can save you from the headaches of over-engineering. Here's some sample how-to-avoid-over-complicating-things advice: 'If Linux can do it, you shouldn't. Don't use Hadoop MapReduce until you have a solid reason why xargs won't solve your problem. Don't implement your own lockservice when Linux's advisory file locking works just fine. Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job. Modern Linux distributions are capable of a lot, and most hard problems are already solved for you. You just need to know where to look.' Any cautionary tips you'd like to share from your own experience?"

Comments

Re:faggot

[IllusionalForce]

>>> cunt nigger
  File "<stdin>", line 1
    cunt nigger
              ^
SyntaxError: invalid syntax
>>>

Thank you for this great tip!

Comment your code

[Nkwe]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do. Remember that comments are not for describing what the code technically does (that is what the code is for), comments are for what the code is intended to do. Try and comment the decisions you made when developing the code, specifically why you took the approach you did and why you didn't use other options.

Re:Comment your code

[Anonymous Coward]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do

I indented the code to make it readable. That's so obvious I don't need a comment to remind me.

Re:Comment your code

[Anonymous Coward]

"indented the code to do"

So the real question is tabs or spaces?

Re:Comment your code

[Anonymous Coward]

> Remember that comments are not for describing what the code technically does (that is what the code is for), comments are for what the code is intended to do.

Corollary: Never debug comments, only code.

Re:Comment your code

[kantos]

Commenting is not enough... you need to make sure you clean up the comments in a file before resubmitting it... dead comments can be more dangerous than dead code... as dead code at least doesn't run... dead comments mislead the person following later into believing a lie a lie that could potentially have major impacts on the software. Lastly... write meaningful test cases... if your tests only prove the happy path, that's OK... but if they prove that the happy path is the only path.. that is better.

Re:Comment your code

[hedwards]

Well, yes commenting on your intent is a good idea, but it's not really enough. You also ought to be commenting on _why_ you're doing something rather than an alternative it makes it a lot easier later on when you're debugging and wondering why you picked that solution. Makes it a lot less likely that you fix the code and reintroduce whatever it was that you were trying to avoid in the first place. But when it comes to the what, if you have to put a what comment into your code, then you've done something horribly wrong and need to rewrite it ASAP.

As for the indents that people are talking about, tabs are more reasonable now than they used to be, but you're giving up screen space for them. Not really the problem it was when you were limited to 80 character columns, but it still has it's benefits and costs.

Re:Comment your code

[Anonymous Coward]

Commenting code isn't enough, it's just a small part of the design and documentation process. Comments are there to tie the code to the relevant part in your design document, which really is a part of programming people should put more effort into.

Re:Comment your code

[Xtifr]

Put enough comments in your code so that five years from now you (and others) can remember what you indented [sic] the code to do.

But not so many that you (or others) will find it more work than it's worth to change the comments when the code changes.

I prefer code with no comments to code with actively misleading comments, and I hate code with no comments! :)

Re:Comment your code

[negativewashout]

This is the best, IMHO. I, too, have looked code and gone, "WTF was this idiot thinking???? ...Oh wait... that's my code."

And agreed - dead comments have to be updated. I didn't always do it perfectly, but I pretty much got in the habit a few years ago of commenting as I go, right by the code in question. One of those times when it's good to be a bit verbose.

Re:Comment your code

[Manip]

I could never find the correct degree of commenting. Didn't help that in school they made us comment in an idiotically verbose way to the point of "this is a for loop to find needle in haystack" and we lost marks if we failed to. My reaction to that level of stupidity was to lose interest in comments entirely which obviously later bit me in the butt.

I really need to find some kind of "idiots guide to when to comment and when NOT to comment."

Re:Comment your code

[TheLink]

> As for the indents that people are talking about, tabs are more reasonable now than they used to be, but you're giving up screen space for them

Why would you be giving up screen space with tabs? On most text editors for programming people can adjust the tabs to display as whatever indentation they want. So if someone wants to view tabs as 2 spaces and another prefers 4 and another prefers 8, they don't have to change anything in the code if their editors are already set up for that.

The disadvantage of tabs is where you have some stuff that requires you to use spaces in (text string etc) and for some reason you want certain parts of it to align with your indents. Or you are mixing your indentation - spaces and tabs (try not to do that OK? ).

Re:Comment your code

[Anonymous Coward]

Read "clean code" and write well crafted code that defines its intent clearly without relying on all too often "out of date" comments.

Re:Comment your code

[hibiki_r]

The problem is that it's easy to follow this rule and forget another important one: If the code you are writing needs a lot of comments, chances are you need to try a different solution, because you are trying one that has the complexity in the wrong places.

The best code has enough comments to be understandable, but it also doesn't have anywhere near as many comments as there is code.

Re:Comment your code

[DMiax]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do.

I know, Python right?

Re:Comment your code

[russotto]

Commenting code isn't enough, it's just a small part of the design and documentation process. Comments are there to tie the code to the relevant part in your design document, which really is a part of programming people should put more effort into.

It's been said for years, but it is almost never done. When it is done, it's most often (IME) done _after the fact_ because of some requirement to produce the paperwork. Perhaps it's time to give up on it. Is there a real reason for insisting on a design document, or is it just some sort of self-flagellation on the part of programmers?

Re:Comment your code

[wootest]

Or to put it another way, because I see you've been misunderstood: Let the code describe what the code does. Methods, classes and variables have names, use them.

If you still need to, let the comments describe *why it does that*. If you need to explain at every turn why it does that, either you're doing things in a weird way or you're over-documenting. You'll risk turning a blind eye to unwarranted comments and spend time maintaining and garbage collecting out-of-date comments.

Even when you do comment, skip the essays. Write down what's important. Spending five paragraphs telling a story about a bug in a library and then leaving out the bug ID in their tracker is to optimize the wrong way. Spend up to two sentences summarizing why and what and what it means, but don't fall through the rabbit hole so the reader doesn't have to.

Re:Comment your code

[Twinbee]

About the 'other options', I often find that I need to keep two different pieces of code that essentially do the same thing, but there are advantages and disadvantages to each approach (often speed versus flexibility). I obviously have to choose one approach, but I keep both just in case I eventually decide to go for the other approach.

Hence, I usually have more commented code than actual code.

This would be fixed if Visual C++ etc. actually allowed one to force loop *unswitching* (different to loop unrolling!). Alas it does not, so you get very messy code, and/or loss of speed.

Re:Comment your code

[sourcerror]

Think of comments as an API documentation*. If something complicated is going on, I usually include several usage example as well. Just look at the documantation 3rd party libraries, and try to follow that granularity and style.

Re:Comment your code

[dodobh]

Put comments in your commit messages. That way the comments never need to be updated, and they can be removed from the source itself.

Re:Comment your code

[Haeleth]

Read "clean code" and write well crafted code that defines its intent clearly without relying on all too often "out of date" comments.

If the comments are out of date, then the code has been modified by someone who was too lazy to update the comments.

Such a lazy person is also unlikely to have made well-crafted changes that kept the code's intent clear.

Now you have unreadable code without comments. Please explain to me why this is a step forwards.

Re:Comment your code

[Anonymous Coward]

I think it is to futility try to prevent scope creep. I've never seen it work, but some people insist upon it anyway.

Re:Comment your code

[Richard Steiner]

Sure, assuming that everyone who ever uses that code is using the same code repository.

I've had to modify code professionally that was written over 30 years before I encountered it (older Fortran 66 stuff), and the comments in the code were invaluable to me. The folks who wrote it had no idea that their code would be running at another company when it was written.

Re:Comment your code

[Anonymous Coward]

It's obvious to you now, but what about in ten years when you realise you should have had the opening brace on the same line as the if statement?

(Now to sit back and watch the fireworks!)

Re:Comment your code

[Korin43]

I don't know. I prefer to comment, just in case:

while 1:
        # Indentation
        dosomething()

Re:Comment your code

[aLEczapKA]

If you need to comment your code you did something wrong and you should refactor it.

Inventor of C++ when asked 'How do you debug your code?', said: 'I don't, if I have to debug the code to understand it, it means the code is wrong and I rewrite it'.

Great book on the subject Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin.

Re:Comment your code

[hachete]

Always comment your assumptions.

Give usage examples (scripts or modules).

Files that are used.

Conditions when module/script dies.

Major bits of functionality - like, this module forks, or it makes RPC calls to some external system.

Make sure the mother fucking error-statements actually MEAN SOMETHING, a line number whence they came, what caused the error etc.

Re:Comment your code

[garyebickford]

Five years?? Hell, half the time I can't remember how the dang things works two months after I finished the first version! :P Two months, that is, of working on some other things that are totally unrelated. I totally depend on my own documentation, including both 'formal' formats like *doc, and my own little cookie crumbs, commentary and admissions of "this particular block of code truly sucks but it has to be done this way to work with that other stuff, that was done before we changed directions, and now we have to work around the decisions we made before we knew enough about the environment to know - and refactoring would take more time than anyone has."

Re:Comment your code

[TeknoHog]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do

I indented the code to make it readable. That's so obvious I don't need a comment to remind me.

(pun indented)

Re:Comment your code

[nacturation]

dead comments mislead the person following later into believing a lie a lie that could potentially have major impacts on the software.

// the following code delivers cake to the subject

Re:Comment your code

[Anonymous Coward]

Your comments will illustrate why you have been replaced by a cheaper, more efficient, less verbose coder.

Re:Comment your code

[daeley]

// the following code delivers cake to the subject

// the preceding comment is a lie!

Re:Comment your code

[Anonymous Coward]

Comments to give the contract for the method or class you're writing. Stick to it, or if you refactor you must keep the comment up to date. Comment public interfaces before you write them. (Not so important for private methods).

Separation of concerns. We don't build hi-fi systems where the loudspeakers know how to guide the CD player's laser, so don't do the same kind of thing in code.

Beware property accessors, especially getters, that have side effects! Asking your for your name should not result in you growing 10 inches.

Immutable value types are good.

Design Patterns are good if used properly, but don't go gung-ho on them. Use them where needed.

Beware function creep of components, especially if they start re-implementing huge swathes of other system or touching directly the database tables of other parts of system - eeek!

Encapsulation!! Don't expose the underlying storage for your classes or people will go in and directly modify it.

instanceof or "switch(someAbstractEntity.getTypeNumber())" are evil. They should go the same way as GOTO. If knowledge is outside the class that is responsible it is very hard to track down and maintain.

Things such as Microsoft's DataSet may be convenient, but passing business entities around as one is not good. It is not type-safe. It breaks encapsulation completely. Because the business entity by necessity can have no business logic you end up forced to write procedural code where all your business methods are static methods on a facade (module in old-procedural-speak). Also all the code gets tied to the database schema.

Re:Comment your code

[danieltdp]

Aren't comments part of the code?

Re:Comment your code

[ZorroXXX]

If the comments are out of date, then the code has been modified by someone who was too lazy to update the comments.

When code and comments disagree, both are probably wrong. -- Norm Schryer

Re:Comment your code

[SL Baur]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do.

/* This is hairy. We need to compute where the XEmacs binary was invoked
          from because temacs initialization requires it to find the lisp
          directories. The code that recomputes the path is guarded by the
          restarted flag. There are three possible paths I've found so far
          through this:

          temacs -- When running temacs for basic build stuff, the first main_1
            will be the only one invoked. It must compute the path else there
            will be a very ugly bomb in startup.el (can't find obvious location
            for doc-directory data-directory, etc.).

          temacs w/ run-temacs on the command line -- This is run to bytecompile
            all the out of date dumped lisp. It will execute both of the main_1
            calls and the second one must not touch the first computation because
            argc/argv are hosed the second time through.

          xemacs -- Only the second main_1 is executed. The invocation path must
            computed but this only matters when running in place or when running
            as a login shell.

          As a bonus for straightening this out, XEmacs can now be run in place
          as a login shell. This never used to work.

          As another bonus, we can now guarantee that
          (concat invocation-directory invocation-name) contains the filename
          of the XEmacs binary we are running. This can now be used in a
          definite test for out of date dumped files. -slb */

OK. So now everyone knows how Lisp programs written with a core in C initialize themselves, right?

And as much as people may joke about it, XEmacs was tested to ensure that it worked as a login shell prior to release.

Re:Comment your code

[Anonymous Coward]

// the following code delivers cake to the subject

// the preceding comment is a lie!

// the above comment explains the joke

Re:Comment your code

[Just Some Guy]

Remember that comments are not for describing what the code technically does (that is what the code is for), comments are for what the code is intended to do.

Amen. The best practice I ever adopted was adding comments like "As per bug #1351", so that a year later I can remember why I picked took a seemingly screwy approach (that maybe implements a workaround for a customer's weird proprietary stuff) instead of doing it the obvious way.

Re:Comment your code

[theshowmecanuck]

Granny's Pearls of Wisdom. For those who never saw Java Ranch.

Re:Comment your code

[cayenne8]

"It's obvious to you now, but what about in ten years..."

Hmm....if you're at all interested in your career, progress both in level and SALARY/BILL RATE, you need to resign yourself to the point that you will not be at the same job 10 years from now to look at code again. The only way to make it these days..is to job hop every 2-3 years.

Re:Comment your code

[gringer]

// the following code delivers cake to the subject

// the above comment explains the joke

== changelog ==
* removed redundant comments

Re:Comment your code

[Thangodin]

If you are new to coding, don't be a bedroom programmer. You are no longer writing a 10,000 line app alone in your bedroom. You may be working on a million line app with a team. Change your habits accordingly. Learn to work with other people.

Programming is one of those things that humans are not quite smart enough to do. This means you. Check your ego at the door. In the early 90's, IBM estimated that 80% of large projects in the industry (one million lines or more) were "abandoned in disgust". This should give you some idea of what you are up against.

Come to work knowing what you are doing. This may mean cramming in your off hours. Don't say that you don't know how to do something. Say that you do and then learn it!

Put in comments where they are needed, and maintain them. You will forget what you were doing within three months. The harder it was to code, the more you need the comments.

Use descriptive variable names. Try to organize your data into conceptually simple variables where possible.

If you have to complicate a mathematical formula by breaking it into sections appropriate for inner and outer loops, put the formula in the comments. It may even be worth putting in an ASCII diagram if you are working with geometry.

If you can't see the bug, it's because you have become blind to the code. Get someone else to take a look. The mistake may be embarrassingly obvious to a new set of eyes.

If speed is a factor, preprocess the data. Offload runtime cycles to preprocessing.

Maintain an up to date user manual for all tools and apps. Add to it as you add features, update it as you update the features.

Avoid magic numbers where possible, and put any magic numbers you do use into defines, again with descriptive names.

If you can, avoid virtual methods and pointers in streamed objects. This way you can bulk load them and bulk write them. Indices often fast enough, or can be converted to pointers if need be after loading.

If you have lots of booleans, consider a bit array.

Try to write reusable code. Code for the general case when possible, but...

Normalize your data and objects. Don't waste memory and time maintaining variables you don't need. Don't repeat yourself.

Your key indexes should be integers, never strings. Yes, I have seen databases keyed on memo fields--they were tragically slow.

If updating an existing project, get the client to sign off on what is not to be changed or fixed, and make certain that the QA department gets this list. Otherwise bugs will creep onto the list that you are not actually required to fix, expanding the scope of the project.

Build test harnesses whenever you can which can be turned on with a simple switch. This will make regression testing a lot easier.

Re:Comment your code

[jc42]

So the real question is tabs or spaces?

Use a mixture. That way, nobody with tabs stops set different from yours will be able to make any sense of the code.

I've seen a lot of this, actually. For example, I've seen a number of cases where people leave the tab stops set to the default (typically 8, far too large), and then set their indentation to something different, usually 3 or 4, but I've also seen 2 and 5. This way, the editor will generate a mixture of tabs and spaces whenever you indent more than the first tab stop. I've questioned this in a few cases, and invariably I've gotten a red-hot flaming response from the programmer(s), telling me in no uncertain terms what an idiot I am for not seeing the importance of doing it just as they're doing it.

Actually, I think I do understand why they it. See the first paragraph of this reply.

Re:Comment your code

[Fulg]

Amen. The best practice I ever adopted was adding comments like "As per bug #1351", so that a year later I can remember why I picked took a seemingly screwy approach

I agree with the sentiment, but I prefer to be a bit more verbose than "bug #1351". Bug trackers change, projects change, access levels too. When bug #1351 in project ABC refers to a bug database that is no longer available, you'll still know what it refers to. No need for being excessively verbose, but just a simple explanation can help tremendously.. e.g. "bug #1351 - state ptr can be NULL when reloading". In my case we frequently re-use code from other projects, but seldom have access to their bug databases (and often they no longer exist).

Same thing for checkin comments - just telling me the bug number won't help me if I wasn't on your team.

Re:Comment your code

[TheoMurpse]

I indented the code to make it readable.

The joke's on you. IT'S PYTHON CODE! Now you removed the indentations and it won't run! Bwahahaha!!

Re:Comment your code

[rhyder128k]

The road to hell is paved with good indentions.

Re:Comment your code

[fast turtle]

If you need that level of commenting in the code, then I suggest you write a man page. That will provide a more concise explanation of what the program is supposed to do instead of what it actually does. It also serves the purpose of starting work on a manual that tells the end user what in hell your app does.

Re:Comment your code

[gringer]

My current annoyance with python:

1. Copy code from one section
2. Paste code to another section
3. Different indentation changes code function
4. ...
5. Bash head on keyboard

Re:Comment your code

[Anonymous Coward]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do.

A comment is a lie. Write self-documenting code.

Re:Comment your code

[cheater512]

I find 8 quite nice with widescreen monitors.
Might as well use the horizontal space and it does make things clearer slightly than 4.

Re:Comment your code

[WolfWithoutAClause]

My comments are stuff like: // look for needle in a haystack
for (a=stuff)
{ // it's got to be a needle if it's made of iron (everything else is straw!)
    if (a.material ==IRON)
    { //yay!
        return a;
    }
}

In other words you document the intent of every paragraph of code, and a high level explanation of how it does it

Re:Comment your code

[Anonymous Coward]

The code is the good perfect to document what the code does and is fine if the code does what the programmer intended and everyone who uses the code agrees on what the code does.
Comments describe intent and assumptions not implementation. // velocity: linear velocity in feet per second

Re:Comment your code

[MBGMorden]

That's what you think.

Some people take commenting to the extreme. I love code comments. I heavily comment my code WHERE NEEDED. It's a negative though when I start looking through a bit of code and it looks like // declare X as an integer
int x; // set x equal to 3
x = 3; // print the value of x
printf("%d\n", x); // return x
return x; // comment about end of function // end of function

Really people, comment the parts that need commenting. Too many comments makes it hard to find the comments that actually explain something that needs to be explained versus all the inane comments about essentially nothing.

Re:Comment your code

[owlstead]

Personally I describe *what* the code does. If it contains several (small) steps, I try to put little markers in as well. If there are certain uncommon techniques I am using, or when an API call is sufficiently unclear, I document that too, but normally the code itself should be self descriptive. I refactor quite a bit during coding, it happens only a few times that I am happy with the names of the identifiers at the first go. Fortunately the Eclipse Java IDE has good support for refactoring. Finally, whenever I still have to do implement, debug or test a method I will mark it using an identifier in a comment. Currently I am using Eclipse tasks for this, using NOTE: (this class should be refactored), WARNING: (this class is not thread safe), DEBUG: (remove these System.out.println statement) and of course the dreaded TODO: (freakin' implement the stuff already). TODO: is always accompanied by an exception (in Java: throw new IllegalStateException("Not implemented")) of course.

Normally my design don't go to method level. They do normally include a system overview and a module overview though, and a view per module of how the *main* classes are related to each other. Personally I don't design each and every method - that's for the implementation phase. I'll just run JavaDoc after I've finished; at least JavaDoc is more likely to be up to date than a separate design document. You *must* document each and every method that is publicly available or that can be overwritten by a child class (protected methods in Java).

I've described Java, but the same techniques should go for any programming language (although the tool support might be lacking for most alternatives).

Re:Comment your code

[HereIAmJH]

Hence, I usually have more commented code than actual code.

And I'll gladly avoid working on any of your projects. Commented code should be removed unless it is only being removed on a temporary basis.
IE: // v1.13 we need to turn off this feature until the next release.

I have worked on too many systems where there were comments like: // removed this to fix ticket #xxxx

I don't know if you have put something else in place of it. And I don't want to wade through a ton of comments why you added a certain bit of code to replace it either. I don't want to retrace your logic to see if you made the right choices. If there are still problems in that section I'll look at what it does compared to what it is supposed to do. I don't know if subsequent changes will be affected if this code is reverted, so I'm unlikely to do so. Use the damn CVS and label it. If we need that code in the future we can pull it out and see if it can be integrated in the current code base.

Re:Comment your code

[Anonymous Coward]

The road to hell is paved with good indentions.

s/road/code/

Re:Comment your code

[Just Some Guy]

Well, that was shorthand for longer comments like "Customer Foo sends poorly-formatted XML, so look at the 3rd word in the 4th line for their invoice number, as per bug 1351 from Jane."

Re:Comment your code

[wurp]

I write comments as a first step to implementing a method. I write out the flow, then go back and implement in between the comments.

  It helps me with thinking through the design, and I end up with documentation that I know reflects what the code says at an appropriate level.

Re:Comment your code

[Just Some Guy]

In other words you document the intent of every paragraph of code, and a high level explanation of how it does it.

No offense, but I despise that style of commenting. I can tell what your code is doing by reading the code. What I don't know is why you're doing it that way. Why are you looking for needles? Have you proven needles are the only thing in your haystack made out of iron so that the code won't detect nails, screws, and hatchets six months from now? Is there a reason your haystack is a list and not a binary tree? These are the things I need to know to maintain your code. I don't need to be told how a loop works.

Re:Comment your code

[GarryFre]

If You think you need to comment to help yourself and others understand the code years down the line but you don't know what to say, you're probably doing it wrong. The thing I wish I knew years ago, was that everything would go web. So many folks I know who are jobless because they didn't bother to learn web.

Re:Comment your code

[Twinbee]

And I'll gladly avoid working on any of your projects.

What, you think I actually enjoy working on code like this or that I really have a choice here?

Guess what. Visual C++ has something called 'code collapsing'. This can be applied to comments too. Great eh? Except it isn't, because Visual C++ has great fun expanding such code every so often randomly and at a whim. I value efficiency and flexibility, and code which might be 'dead' to you might very well come in handy later if it's better in a certain way.

For the record, I probably exaggerated when I said most of my code is commented code. But still, a good proportion is, and admittedly, some of it could be probably be wiped out. I try to get rid of any dead wood code every so often.

Guess what else would help me cut down on commented code? Yep, loop unswitching. Sigh.

I don't know if you have put something else in place of it. And I don't want to wade through a ton of comments why you added a certain bit of code to replace it either.

Yeah, I'd probably agree there, although I'm pretty sometimes bad at adding description comments, only to regret it later.

Use the damn CVS and label it. If we need that code in the future we can pull it out and see if it can be integrated in the current code base.

That's all fine and dandy, but then what happens if you want to not only incorporate bits and pieces from an older version (rather than the thing wholesale), but also where the latest version uses extra code, and so it becomes difficult to know where to insert the bits without worrying that doing so won't wreck the ordering of the code (x needs to execute before y before z etc.). So there's this messy 'interleaving' of code which can be hard to avoid.

Re:Comment your code

[Greyfox]

/* Don't touch this function unless you REALLY know what you're doing */ Funnily enough the only guy who ever touched it didn't know what he was doing. In the short time he worked there, he did enough damage to the code base that it took 6 months to fix it. Too bad the version control system we were using at the time didn't allow you to simply mass-revert a user. That would have actually saved us a lot of time.

Re:Comment your code

[Greyfox]

I prefer comments about business logic and what the function is supposed to do. I've found that whenever I try to document the parameters, the function call changes parameters the next time I have to look at it. If there's a description of what the function actually does, though, I can usually figure it out.

You can usually figure out what someone was trying to do by looking at code, but it helps if he actually told you. You can also tell when someone was confused by his requirements or the problem by reading his code. The more convoluted the code is, the less likely it was actually designed.

Re:Comment your code

[Coryoth]

But not so many that you (or others) will find it more work than it's worth to change the comments when the code changes.

I prefer code with no comments to code with actively misleading comments, and I hate code with no comments! :)

The trick is that if you are writing comments describing what the code is intended to do, you can write those comments in something like JML or Frama-C's ACSL. That was you can use ESC/Java2 and Junit, or Frama-C, to do your checking that the code does what you intended. You get two benefits: more rigorous checks on your code (including use of theorem provers from ESC/Java and Frama-C); if your documentation ever falls out of date with the code, you'll immediately get errors flagged.

Re:Comment your code

[rxan]

I recently began work in a team whose codebase is largely uncommented. Not only are there usually no comments within functions, but there aren't even function and class header comments. Needless to say, it's really annoying having to guess what functions do, what the return value means, etc. Often code alone isn't enough. Failing to write comments just adds more work time later on.

Re:Comment your code

[AuMatar]

Yes, because what's clear to you is going to be clear to everyone else who ever reads it. And it'll be perfectly clear to you in 6 months or 6 years when you try to alter it. Yup.

There is no such thing as self documenting code. If you ignore comments and/or documentation, you're a shit poor programmer and a blight on the profession.

Re:Comment your code

[Hooya]

I think you meant:

// the following comment is the truth

// the preceding comment is a lie

Re:Comment your code

[MikeFM]

The most important thing to comment is what your code doesn't do. If you didn't do it this mayor that way for some reason make a note. Otherwise some idiot, probably yourself, will come back to it later and get the bright idea to improve it in exactly the way you decided not to use for some specific reason. Don't know how many times I had to undo some brilliant fix because I forgot to write down why I hadn't done it that way.

Any idiot can see what your code does.

Re:Comment your code

[Hooya]

I have tried implementing a 'design document' process for the better part of 10 years that I've been with this group. It's never gotten done. We came close about a year ago. Here's why it's I still try (while knowing that it'll never get done):

There's a reason architects use blueprints.

Re:Comment your code

[Lord of Hyphens]

Tab to the block, space to line up individual statements (basically a line that's wrapped) and make things look cleaner. That way, no matter how your editor is configured to show tabs, everything will be spaced relative to the block they're supposed to live in. void foo() {
tab Some long statement
tab that should be indented like so.
}

Re:Comment your code

[Lord of Hyphens]

I hate /code and formatting code with HTML

Re:Comment your code

[russotto]

Hmm....if you're at all interested in your career, progress both in level and SALARY/BILL RATE, you need to resign yourself to the point that you will not be at the same job 10 years from now to look at code again. The only way to make it these days..is to job hop every 2-3 years.

Yeah, but the perversity of the universe is such that the company you're with in 10 years could buy up (or be bought by) the company with now, and you could end up responsible for the very same code...

Re:Comment your code

[russotto]

There's a reason architects use blueprints.

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

Re:Comment your code

[Khelder]

I agree except "five years"? More like 6 months, in my experience (sometimes even less). IMNSHO, any code you wrote and haven't looked at for 6 months or more was effectively written By Someone Else. Unless you're on a really short term with your current employer, writing good code with good comments isn't just good professional behavior and being considerate to the next guy, it's for you, too, so when you boss asks you next year to add feature X to that code you wrote you won't waste time flailing around trying to understand (again) code that *you wrote*.

Re:Comment your code

[WolfWithoutAClause]

No, the comments above would explain that we're trying to do, e.g. find a needle to sew with, and the comments around the data structure definition would explain that we use a hashtable to speed up searching for ... The point *isn't* simply to do

// add 4 to y
x = y + 4;

it's more like

// we need to calculate the x coordinate, it's equal to y to be on diagonal, but add 4 due to border width
x = y+4;

So you explain *what* you're doing in HIGH LEVEL terms, and set CONTEXT. And you don't comment each line, you comment each group of lines that do one thing.

And I also disagree with your 'despise' thing. It's very, very rare that code is over commented, and in most cases failing to correct comments helps create bugs, because it shows you weren't able to understand the code well enough to explain it to others, chances are there's a bug there goes way up. I also find that after I've written code, then's a good time to read it over and comment it. I find lots of bugs that way, and the code ends up much better. Not doing that wastes time in the long run because you'll find the bugs the hard way.

Re:Comment your code

[Kjella]

Actually I kinda like the first comment though I'd probably just shorten it to "// find needle". Why? Because when I go back to some old code it's probably to find something, I'm in search mode not looking to relearn everything. Even if it's a function that's only a screenful, if it has three lines of comment like

// find needle
...
// bend needle
...
// place needle in bin

More often than not, I know the problem I'm looking for. We're finding screws where there should have been needles. Some needles break in the bending. Now we're doing recycling and needles go in another bin. The comments allow me to at a glance discard 2/3rds of the code focusing on just what I need.

Here's my take on comments:
In-code: Short, short description of flow. If you're doing anything "clever", explain why.but otherwise leave it at that.
Functions: Good for API reference but remember most of the time I'm reading other code calling the function, not the function description.
Classes: Yes please.

I've found that good variable and function names are very important. It's really, really annoying to write because it becomes a typing job and after the n'th time you type neelde or needel instead of needle you'd love to just call it "n". I did that programming my T-81 as I was typing on the calculator, it was the greatest clusterfuck of no-comment, one-letter variable code ever. Even I couldn't understand anything a month later.

The worst possible sin you can make in my opinion is to create functions that also have secret side effects. For example a "load" function that also tries to fix some broken input formats or an output function that secretly is trying to rewrite the page to switch between A4 and US Letter size (first hand experience on that one). For the love of $deity, either put it in a separate function or if it really needs on-the-fly fixing in the loading, COMMENT in the API. Otherwise I'll just see the load() function being called and assume it does exactly that, nothing more. The same goes for changing state, if it's not obvious from the function name that this would or could happen don't do it there.

Re:Comment your code

[Fulg]

Then we are in violent agreement ;)

Some people at work haven't learned this yet and only put bug numbers in their code/checkin comments, so I didn't pick up on the shorthand.

Cheers!

Re:Comment your code

[Lotana]

Solution: Stop mindlessly copying and pasting.

I lost track of the amount of code I've seen that is a straight duplicate from some area that does vaguely the same thing. Then all the variable names don't match the new context, comments are not applicable and assumptions are not necessarily correct. Doesn't matter what language it is written in.

Of course the person that does it never bothered to look through it and only relied on compiler error messages to detect the obvious "not declared" errors. What ends up is a mess that is nearly guaranteed to have bugs.

If tabbing the new pasted code into place forces you to actually read it and think about what you just did: then it is well worth it!

Re:Comment your code

[OeLeWaPpErKe]

1) commenting assumptions

assert

2) give usage examples

"integration test"

3) files that are used

first, this should be configurable (as a general rule)
second, open(

4) conditions when module/script dies

assert ...

All of these could simply be part of the code, instead of mere comments. And really, who will check your assumptions, unless he's spent days looking for a bug without success ?

Re:Comment your code

[Anonymous Coward]

I actually laughed out loud at that one. Thanks!

Re:Comment your code

[Anonymous Coward]

I often write out a possible justification and comments step by step for a routine before I write the code..
That way it's often straight forwads how to fill it in.

Re:Comment your code

[DavidTC]

That's what the GP meant. Comment why, not how.

You only need how when you do something that people might miss, such as a switch fall-through, or not understand, like some tricky Boolean operation.

Everything else should be about what the code is attempting to do, and why it's trying to do it that way. When that's obvious, you don't need any comments at all. (Although function headings are still a good idea.)

Re:Comment your code

[DavidTC]

There is no such thing as self documenting code.

I really hope you mean 'There is no way to write all code as self documenting'.

There's plenty of self documenting code out there. No one needs to document what a sprintf() is doing or that you're declaring a variable. If we don't know that 'int result_count; // declares an integer to store the count of the results', we probably shouldn't be programming there in the first place.

What needs documenting is the goal of stuff and why you're doing it the way you're doing it, along with exceptionally confusing lines of code by themselves. (For example, any regexp beyond the absolute basic needs documenting.)

Re:Comment your code

[19thNervousBreakdown]

State your assumptions. If you fail at everything else but do that, your comments will be better than 90% of the comments out there.

Re:Comment your code

[SheeEttin]

Or just use Doxygen--the comments ARE the documentation.
In a good way, that is: The docs are generated from the comments.

Re:Comment your code

[SpaghettiPattern]

Put enough comments in your code...

And as soon as the comments turn inconsistent, read the code.

Re:Comment your code

[dasunt]

My comments are stuff like: // look for needle in a haystack

for (a=stuff)
{ // it's got to be a needle if it's made of iron (everything else is straw!)
if (a.material ==IRON)
{ //yay!
return a;
}
}

Isn't this better?

find_needle(haystack)
{
for ( i = 0; i < sizeof(haystack); i++ )
{
if is_needle(haystack[i]) return haystack[i];
}
return NULL;
}

is_needle(item)
{
// Everything but needles are non-iron
return item.material == IRON ? true : false;
}

Re:Comment your code

[complete loony]

People often compare building software to building physical structures. I'd say that building software is closer to drawing the blueprints for a structure.

Re:Comment your code

[Trane Francks]

Yeah, but the perversity of the universe is such that the company you're with in 10 years could buy up (or be bought by) the company with now, and you could end up responsible for the very same code...

I spent almost 14 years at my last job. I maintained the same code base for several applications that evolved over that entire time. And when somebody comes across a bug in a program you haven't updated in 2 or 3 years, you're bloody grateful for comments that let you know what the code is doing.

I also think that comments should not only describe intent, they should describe function. It's there where we can often see a discrepancy between what we want the code to do and what it actually does.

Re:Comment your code

[Anonymous Coward]

Obviously 8-character tab stops is the standard and pretending that a tab character means some non-standard number of spaces is an abomination.

Re:Comment your code

[IICV]

Is there a reason your haystack is a list and not a binary tree?

Clearly, this is why you need comments! Haystacks aren't lists - they're heaps.

Re:Comment your code

[flowwolf]

comments are for what the code is intended to do.

no one has ever put it that way for me before. you just blasted my mind sir

Re:Comment your code

[severoon]

This is true...I don't really comment my code that much—well, internally anyway. I comment copiously on the API, and if the API is designed properly and dependencies managed well, you shouldn't need a lot of comments in the code itself aside from the tricky cases you mention (switch fall-thrus...if you're using switches, which you probably shouldn't be).

Re:Comment your code

[Errol backfiring]

No. They are not compiled and therefore lag behind the state of the code in almost every software project I have encountered so far. If you can choose between naming and commenting, use naming. For example, a parameter called ZeroBasedIntegerIndex will be in your editor's autocompletion and therfore visible where you use it. A comment is only visible where you define it, and therefore gets lost in usage. And if the ZeroBasedIntegerIndex happens to be defined as a string, you'd change it immediately. The comment would probably stay wrong for a very long time.

Re:Comment your code

[MadKeithV]

There's a reason architects use blueprints.

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

Yes it does.
Consider all the useless implementation work you've done while your design was wrong (before you reworked it). It's just as important in software as in building.
Also, having blueprints / a design allows you to separate the work between many different teams with clear, agreed-on-beforehand interfaces between the sections that each team is working on. This is something that a lot of developers seem to miss (I know I did for the longest time) - perhaps because a lot of us tend to prefer working and thinking alone?

By the way, working out a design is not a solo 1-hour effort. It's just as much "development" as actually writing the code. Often you may actually want to write code while doing it just to make sure that what you are thinking of does work. But that's not the final software, it's still just the design. Don't give in to the temptation to "improve" that design prototype until it ships as finished software though...

Re:Comment your code

[Anonymous Coward]

Maintain an up to date user manual for all tools and apps. Add to it as you add features, update it as you update the features.

Completely impractical and superfluous for most systems I've worked on. The pace of development means it's difficult enough to get requirements pinned down, good unit test coverage, UAT and of course the actual development done. User manuals are the first things to slip by the wayside, becoming out-of-date and unused from almost the time it's written.

Plus, how many support calls are RTFM anyway, with a good manual?

Re:Comment your code

[MadKeithV]

The thing I wish I knew years ago, was that everything would go web. So many folks I know who are jobless because they didn't bother to learn web.

You may have forgotten the .com boom / bust - a lot of the early "everything will go web" people got seriously burned, because they got it mostly wrong.

Re:Comment your code

[leastsquares]

// the following code delivers cake to the subject

// the preceding comment is a lie!

// the above comment explains the joke

// TODO: I think the above comments are correct, but we need to check and correct them if necessary.

Re:Comment your code

[erixm]

Come to work knowing what you are doing. This may mean cramming in your off hours. Don't say that you don't know how to do something. Say that you do and then learn it!

Please ignore that. Honesty is always better - and if your team doesn't welcome honesty, try to change the team's attitude. If you trust each other your results will be better. If you don't know how to do something, say you don't and THEN learn it.

Re:Comment your code

[tibit]

The magic number avoidance is a double-edged sword. Some people take it for gospel even if it doesn't make sense.

Case in point: I have some code that talks to a piece of hardware. The code is really a library class. The hardware has a bunch of commands, that are invoked by using their numeric codes. I wrote the code like so:

Status Hardware::do_thingamabob() {
    return send_command(0x13);
}

This command code is not needed anywhere else. Now comes a magic number demagician and turns it into a header file with a bunch of defines, and the function becomes:

Status Hardware::do_thingamabob() {
    return send_command(THINGAMABOB_CMD);
}

No magic numbers? Check. Easier to understand? No, the locality has been lost. You need to refer to another file to see what is really sent. BTW, the code in question only cares about command codes within the functions that look like more or less like the above, and they are a perfect place to keep those numbers.

Re:Comment your code

[wvmarle]

Or you are mixing your indentation - spaces and tabs (try not to do that OK? ).

That hurt me once badly when programming in Python. Using tabs for the indentation, then for some reason one time I was using an editor that silently replaced those tabs with spaces (four spaces for a tab - which was also my default setting before so it looks the same), later back to my normal editor and when running it I suddenly got all kinds of indentation errors! Took me a bit of headscratching before I realised what was going on. Ouch.

Still occasionally running into this, especially when using bits of other people's code, but at least now I know what to look out for.

Re:Comment your code

[mrjb]

And while we're at it, move your is_needle function to above find_needle. If you include files or import modules, you do that at the start of your script, not at the end! This is a leftover from the one-pass compiler days- functions are defined (or at the very least declared) before they are used.

Failing to do this will at best cause your code to compile more slowly; but worse, you'll be slapped with bad puns from your fellow developers who will complain that finding your is_needle function is like looking for needles in a haystack. You definitely don't want that to happen!

Re:Comment your code

[cyclomedia]

MS's visual studio's XML comments (may not be original and ripped of from somewhere I've not used) are absolutely brought along with the autocompletion/debugging. It even generates compiler warnings if you change the parameters to a function and dont update the xml to suit and shows the info in a tooltip whenever you hover over it in the editor, brilliant and very handy.

You should absolutely still give your objects meaningful names though, it's not an either/or thing

Re:Comment your code

[tomtomtom777]

No. They are not compiled and therefore lag behind the state of the code in almost every software project I have encountered so far. If you can choose between naming and commenting, use naming. For example, a parameter called ZeroBasedIntegerIndex will be in your editor's autocompletion and therfore visible where you use it. A comment is only visible where you define it, and therefore gets lost in usage. And if the ZeroBasedIntegerIndex happens to be defined as a string, you'd change it immediately. The comment would probably stay wrong for a very long time.

So you are actually promoting the use of variables named like ZeroBasedIntegerIndex?

That name really sucks, dude. I understand that you want to clarify the advantage of good variable naming, but ZeroBasedIntegerIndex is NOT a good example. Names should describe what the variable means semantically.

Re:Comment your code

[EsbenMoseHansen]

Actually, what will have happened in 5 years is that the code will have changed, but the original comment remains, so that the code now does something slightly different. And it will be hours of debugging fun to find out which is right, if any.

Re:Comment your code

[BrokenHalo]

The perversity of the universe I live in is that in my first programming job back in the late '70s, one of my tasks was occasionally to maintain a bunch of commercial code for which the source (for some reason I don't recall) no longer existed, so I had to work directly on the binary. I actually didn't mind, since this was much more interesting than bashing out COBOL code by the bale.

Re:Comment your code

[BrokenHalo]

Names should describe what the variable means semantically.

Heh... We didn't always have the luxury of that. Back in the days when I was working on a Burroughs B3700, 6 characters was the maximum the OS (MCP IV) would allow, so our filenames were inevitably cryptic.

Re:Comment your code

[BrokenHalo]

Took me a bit of headscratching before I realised what was going on. Ouch.

That's one reason why I have a tendency to be suspicious of editors that offer a WYSIWYG interface. I much prefer YAFIYGI (You Asked For It, You Got it) editors like (of course) TECO or more recently, EMACS or (if you insist) VI.

Re:Comment your code

[Anonymous Coward]

On the contrary, nothing in the code mentions anything about needles or haystacks. The comments document what the code is doing (looking for a needle), and what assumptions are made (needles are iron, and only needles are iron). Neither of those are obvious from the code itself, which just looks for something iron in a bunch of stuff. Without the comments, someone could easily inadvertantly reuse the function to look for something else made of iron in some other pile of stuff. Then the assumptions change for one case, the function gets fixed, and the other cases break.

Granted, the method name should be something like Haystack.FindNeedle (or FindNeedleInHaystack, depending on your OO preferences), but that's not stated in the code snippet.

Re:Comment your code

[Anonymous Coward]

Commenting is good, but don't over-comment:

while 1:
        # Indentation
        d()

Re:Comment your code

[olau]

There's a reason architects use blueprints.

Yeah? They are building houses? Houses don't change a lot when they're first build, and the guys designing the houses need to explain their designs to the people who actually implement them, the builders.

While a high-level description might be a good idea for software too, a design document is at much higher risk than inline comments when it comes to bit rot. It's better to have the design documented inline in the code. For instance by writing legible code with class and function names that actually explain what's going on. :)

Re:Comment your code

[olau]

The trick is that if you are writing comments describing what the code is intended to do, you can write those comments in something like JML [ucf.edu] or Frama-C [frama-c.com]'s ACSL.

But most likely those comments will be pretty useless from a commenting perspective, namely to explain stuff which is not easily codified in a formal language.

Re:Comment your code

[Anonymous Coward]

Be very careful with descriptive variable names. Reading code where the variables have names that don't quite match up to my conceptual map of meanings is more confusing than ever especially if nouns and verbs get mangled in the process.

Use short variable names and explain what they do in comments.

int integerParsedFromLocalUserDatabaseOfPrimaryColoursNotIncludedInOtherDatabases = 0;

# urP = otherwise un-referenced primary colours
int urP = 0;

Re:Comment your code

[sourcerror]

Sorry, but you missed the point. Automatic doc generation is good, but we are speaking here about style. My point is that document your functions as an API even though they aren't exposed to the external world. And at harder parts it should be in the style of tutorials.

Re:Comment your code

[AmiMoJo]

The tick is to figure out what is worth designing and what isn't.

You need to get architectural stuff that can come back and bite you in the arse when it needs extending or scaling right from the start. Figuring out data validation tests that will prevent security exploits is also a good idea. You don't need too much detail, just enough to avoid the pitfalls.

Bothering to design a sort function or some other trivial bit of code is a waste of time. They are not hard to write and even if you did screw it up re-writing it later is no big deal.

Re:Comment your code

[deadcrow]

Then cake() always returns false

Re:Comment your code

[jc42]

Obviously 8-character tab stops is the standard ...

[citation needed]

Just out of curiosity, can you provide a reference to a standards doc that specifies exactly 8-char tab stops? The definitions of tab stop that I've seen have been along the lines of "whatever columns the user has listed in the tab-stops setting".

AFAIK, tab stops were invented by typewriter makers, who implemented the idea by providing a metal bar with a bunch of little "stop" widgets that you could slide to wherever you wanted them. There was no concept that the stops should be an any specific positions. In fact, the wire usually had little notches at ever char position, resting points for the "stop" widgets, further implying that the stops could and should be set at whatever positions you like.

(I suppose it's possible that this tab-stop mechanism could have predated the invention of the typewriter. Anyone know of an earlier machine that used something conceptually similar?)

Re:Comment your code

[kbielefe]

That's because people who don't know what they are doing don't know that they don't know what they're doing. Those types of comments should be accompanied by a clear competence or acceptance test. For example, the last such comment I wrote went something like: /* This might look like an unnecessary delay, but the timing has been carefully calibrated against a wide range of marginal real world conditions. If you touch this function, you must ensure it does not time out under these configurations... */

In other words, "knowing what you're doing" must be definable and transferable knowledge.

Re:Comment your code

[tomhudson]

No. They are not compiled and therefore lag behind the state of the code in almost every software project I have encountered so far

True - for compiled code. For scripts, they're loaded every script run. Sure, they're ignored, but it still takes time for the parser to skip over them.

But no, they never lag behind, because we never make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes make mistakes

(Yes, they almost always lag behind, because most of us are only human. And a wrong comment is worse than no comment when it comes to code maintenance and debugging - same as bad symbolic names)

Re:Comment your code

[tomhudson]

Sometimes the docs get it wrong - or we make invalid assumptions. I know of at least one system that I used that had such a limitation, but I found out that in reality it was that you could use much longer names, but only the first N characters were significant. As long as you didn't have any name dupes in the first N characters, your code was much more readable.

Always test assumptions :-)

Re:Comment your code

[tomhudson]

typically 8, far too large

If you think 8 is "far too large", when it was fine back in the days of 80-column amber monitors, you're doing it wrong.

Code that requires 4- or 2-space tabs is inevitably a mess - or written in JavaWithWayTooLongClassAndMethodNamesBecause.theyDontHaveAPreprocessorOrProperIncludes(), but I repeat myself.

Re:Comment your code

[Greyfox]

That's true. The function in question was actually quite well commented, and that last one was really more of a warning that all the comments should be read before messing with it. A set of unit tests would actually have been pretty nice and would have saved us a lot of trouble over the 5 years we worked on it, but it would have been a tough sell to management.

Re:Comment your code

[Thangodin]

Just define it right there, if that's the only place you use it. And no, THINGAMABOB_CMD is not easier to understand, because it's a poor choice of a name. Something like PRINTER_LINEFEED is what you want here.

Re:Comment your code

[Coryoth]

But most likely those comments will be pretty useless from a commenting perspective, namely to explain stuff which is not easily codified in a formal language.

Well no, it doesn't absolve you from writing comments detailing the "why" of what you are doing in plain english; we're talking about comments describing what the code is intended to do. If you are concerned that you won't be able to say anything interesting then I suggest you take some time to look at JML and ACSL because they are far more flexible and expressive than you might imagine. No, you can't always easily say everything you want to say, but far more often that not it is very easy particularly because of the syntactic sugar for expressing constraints that these languages provide.

Re:Comment your code

[Thangodin]

You're probably right, but I've seen whole projects done in the wrong language or using the wrong tools because someone there didn't know the right one, and wasn't willing to learn it. So the best answer would probably be, No, but I can learn it.

Re:Comment your code

[OolimPhon]

Put enough comments in your code so that five years from now you (and others) can remember what you indented the code to do.

I know, Python right?

Yes. Monty.

Re:Comment your code

[tehcyder]

There's a reason architects use blueprints.

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

I think it's more along the lines of "in twenty years time when someone wants to know why there is a big hole in the floor, we can check back to see who fucked up the original design, so we can sue them".

Re:Comment your code

[Anonymous Coward]

Don't repeat yourself.. this is important, don't repeat yourself.

Re:Comment your code

[GarryFre]

I didn't forget. Pick up a job listing and you'll be lucky to find any job not requiring web based skills. Businesses who went web too early got burned, but today its programmers without web skills that are in the fire.

Re:Comment your code

[tibit]

THINGAMABOB was meant to have an obfuscated meaning ;)

As for defines: first of all, since this is C++ code, an enum or a static const int would be more appropriate. But then -- why bother, it doesn't make code any more readable. It's less readable, since you add a layer of abstraction.

Methinks your line of thinking is a case in point for falling for "goto considered harmful" as an excuse not to think about what one is doing.

Re:Comment your code

[Doctor Memory]

ISTR that PL/I had a 128-character limit, but if you exceeded that limit it named the identifier by concatenating the first and last 64 characters. I always use something similar if I'm designing a language, albeit with smaller limits (I think that after a point, identifiers get harder to read as they get longer).

Re:Comment your code

[roju]

Nobody should be documenting sprintf calls because nobody should be writing them. Unless the comment is along the lines of /* since this program is a demonstration of buffer overflows, we're using sprintf instead of snprintf */

Re:Comment your code

[qwijibo]

PRINTER_LINEFEED would be a reasonable name for 013, but a horrible name for 0x13. Reminds me of this guy I used to work with who always knew the right thing, but always did the wrong thing for "job security". =)

Re:Comment your code

[DragonWriter]

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

I've seen studies which claimed that there was, IIRC, approximately an order of magnitude increase in cost to repair design defects in software identified once code has been built around it rather than in design, another order of magnitude for design defects identified in acceptance testing, and another order of magnitude for design defects caught once a system is in production.

So I'd like to see something more than an assertion that this saying does not apply to software.

Re:Comment your code

[jeff4747]

We should stop making design documents right after we stop making plans before building a house.

Re:Comment your code

[jeff4747]

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

It's so all the parts fit together. The only reason you have to "fix" a building is when the appropriate parts don't line up.

Re:Comment your code

[deander2]

i don't know those specific tools, but it sounds like your solution to poor commenting is to write the same code twice in two different languages. =P

Re:Comment your code

[jgrahn]

Obviously 8-character tab stops is the standard ...

[citation needed]

Just out of curiosity, can you provide a reference to a standards doc that specifies exactly 8-char tab stops? The definitions of tab stop that I've seen have been along the lines of "whatever columns the user has listed in the tab-stops setting".

The definition probably goes on "... or if he hasn't (like 99.99999% of all users) it's eight".

But do you seriously need a standards doc? Take any terminal emulator, real terminal, text editor or printer manufactured since the 1960s and give them a few test patterns. I can almost guarantee that all of them will be using 8-char TABs.

Re:Comment your code

[russotto]

Consider all the useless implementation work you've done while your design was wrong (before you reworked it). It's just as important in software as in building.

In building, "implementation" means leveling ground, pouring concrete, fabricating structural members, welding, etc.

In software, "implementation" means writing a bunch of code. It's not nearly the same. Rework the design, rework the code; not that big a difference. And by coding without a design document, I save the time required to produce a design document, and I discover more quickly if my design does not work.

Re:Comment your code

[jgrahn]

Commenting code isn't enough, it's just a small part of the design and documentation process. Comments are there to tie the code to the relevant part in your design document, which really is a part of programming people should put more effort into.

It's been said for years, but it is almost never done. When it is done, it's most often (IME) done _after the fact_ because of some requirement to produce the paperwork. Perhaps it's time to give up on it. Is there a real reason for insisting on a design document, or is it just some sort of self-flagellation on the part of programmers?

It might be flagellation, but not self-flagellation. Programmers hate writing useless documents. Management wants those documents *before* the code is written, but the programmer knows he'll rework the design because of stuff he learns while doing the coding. Management also wants the documents in some word processor format not compatible with the version control software, so it cannot be updated along with the code either.

If I could have it my way, I'd have these things:

• User reference documentation, like man pages. File and protocol format specifications. Standards. Those can be very helpful for the programmer too. They cover the requirements aspect of things.
• Good checkin comments from the version control system, i.e. "I did this because of that".
• A *brief* text about the general design and architectural decisions, glossary etc. Things like "this program revolves around a single select() loop" or "we try to make this part bloody fast because ..." or "we use this trick throughout the code to avoid table lookups". Maybe a not-too-detailed class diagram, with a note that it might be obsolete and if you want a recent one you're free to draw one.
• A clear design with clear naming and strong typing, and/or readable unit tests.
• Comments on the class or module level: "class Foo represents this thing, with these limitations". On the function level too, but only if it's needed.

Re:Comment your code

[jgrahn]

Think of comments as an API documentation*. If something complicated is going on, I usually include several usage example as well. Just look at the documantation 3rd party libraries, and try to follow that granularity and style.

I think that's a bad idea, because comments are usually *not* API documentation. Much of the code you write is aware of what code is calling it, and the reader is in control of both.

If you want a style guide, I can recommend Unix man pages. This one (adapted from Linux memcpy(3)) is both elegant, brief and clear:

Copy n bytes from memory area src to memory area dest, and return dest. The memory areas should not overlap. Use memmove(3) if the memory areas do overlap.

Re:Comment your code

[Anonymous Coward]

There's a reason architects use blueprints.

Which is? If it's along the lines of "It's easier and cheaper to fix a blueprint than a building", then it does not apply to software.

If the root cause lies at a higher level of abstraction than the implementation source code, then yes it does apply to software.

Re:Comment your code

[rhyder128k]

This demonstrates the advantage of open source one-liners.

Re:Comment your code

[marcosdumay]

"Come to work knowing what you are doing. This may mean cramming in your off hours. Don't say that you don't know how to do something. Say that you do and then learn it!"

How true! But make those off ours work for you, not just for your boss. Have a growth path planned.

"f you have to complicate a mathematical formula by breaking it into sections appropriate for inner and outer loops, put the formula in the comments. It may even be worth putting in an ASCII diagram if you are working with geometry."

Or, keep a small (one page, one paragraph, no bigger than needed) LaTeX paper explaining that comlicated mathematical formula at the doc directory. (Your projects do have a doc directory, don't they, there is where you put the emails or annotations you have explaining what your program should do, and is also a greath path for auto generated documentation, like javadoc or doxygen.) If you use a complex geometric construct, by all means, draw it, and put it on the docs too (ASCII art on the code, where possible, is a plus).

"Normalize your data and objects. Don't waste memory and time maintaining variables you don't need. Don't repeat yourself."

I don't completely agree with that. Sometimes it is way easier to maintain replicated data than to postpone calculations to the time their are required. Other times it is not, there seems to be no rule.

Re:Comment your code

[Coryoth]

I really think that means that you need to become better acquainted with those tools. specifying what you intend to happen is a lot easier than specifying how to do it. The first is a specification, the second is an implementation. Consider, I can specify a sqrt function by simply saying the result squared should be within some error tolerance of the input; that is a long way from writing an implementation of a square root function. Likewise, I can specify a sort function (the output list should contain exactly the same elements as the input list, and they should now be in order with respect to a comparison operator) without ever doing anything as complicated as actually implementing quicksort, or mergesort, or heapsort or ... Try looking at the tools and what they can do and you'll satrt to get the idea.

Re:Comment your code

[PCM2]

But "implementation," in the sense of "my customer needs this feature/this bug fixed," does not end with writing the code. If you've written the code -- and let's assume you've gone to the trouble to unit test your changes -- all you've done is update the blueprint. Before the features are actually available or the bugs are actually fixed, you still need to ship your changes to the appropriate servers and deploy them with a minimal amount of downtime. If you're talking high-volume, mission critical applications, then truly implementing the fixes can indeed be very costly and potentially very risky.

Maybe a blueprint is the wrong metaphor, but with software, "measure twice, cut once" still applies.

Re:Comment your code

[Reziac]

Which was actually how I interpreted your first remark... a shorthand for "I don't know it today, but I'll know it by tomorrow".

Re:Comment your code

[MadKeithV]

Actually I'm a 100% non-web programmer, and I find no shortage of work, but I do notice more and more what you are saying. Often, even if the project is completely off-line, people higher up will try to push for web-technologies "just in case".

Re:Comment your code

[MadKeithV]

In building, "implementation" means leveling ground, pouring concrete, fabricating structural members, welding, etc.

In software, "implementation" means writing a bunch of code. It's not nearly the same. Rework the design, rework the code; not that big a difference.

I feel there's a big difference between reworking the design and reworking the code.

Firstly, it has been shown empirically over and over in studies that the later in the process that you catch and fix bugs, the more expensive they are (for deeper analysis, see The Cost of Bugs.) Design bugs should be caught early, because they ripple through the whole project and can cost an enormous amount.

Secondly, unless you are working alone or in a tiny team there is a significant difference between design and implementation: you have a larger team working on implementation than on design. If you sign off too soon on a bad design, you'll have a lot of people writing a lot of code in a short time. Depending on how "off" the design was a lot of that code may have to be thrown away, and when you go into redesign, a lot of the team has to be idled because you can't all work on the design efficiently.

Re:Comment your code

[Anonymous Coward]

And heaps aren't searchable in sublinear time, which is why finding the needle is so hard

Re:Comment your code

[DavidTC]

You're assuming C and not, for example, PHP, where there's no chance of that. (Not that I'm asserting that PHP is secure, it just doesn't have any buffer overflows with sprintf.)

Re:Comment your code

[Thangodin]

Whatever... define it, comment it, but whatever you do, leave some note telling what the fuck 0x13 means! An enum is fine. I prefer to put all the magic numbers in one place so I can tweak the behaviour of the app in one place without having to hunt parameters down, a major time waster, and a great way to conceal an obvious bug till four builds (and two weeks of screaming clients and lawyers) later. But if you just leave a magic number and a bug shows up related to it, someone will be asking you, and they won't be asking politely if it's crashing the build.

And I said something like PRINTER_LINEFEED. It might be GO_TO_GOAL, or COPY, or ANIM_RUN. But whatever it is, say what it is.

Re:Comment your code

[tomhudson]

Then cake() always returns false

Cake does not return. It sticks to your waist-line like ... well, like cake.

Re:Comment your code

[lamapper]

LMAO, I have heard better reasons to NOT comment code than this.

This explains why most Object oriented code loads such large and unneeded libraries.

Do I need this? Don't know, better load it so nothing breaks...thanks for nothing.

Re:Comment your code

[gr8dude]

It does apply to software, but it depends on how complex the system is and how many people are involved in the development process.

To get a convincing answer to your question, I suggest that you read "Software requirements" by Karl Wiegers.

Re:Comment your code

[ps2os2]

I do not know you and from a person that was responsible for being first person to fix a problem at all hours of the night I can tell you that comments do not mean squat unless they actually make sense. I wish I had saved some of the comments over the years but I think you would cry over some of them I have seen. Also (this partly applies to the comments) labels should mean be descritptive not switch-1 or dog3-has-fleas or any of the other impossible names I have seen over the years.

I have witnessed several different coding techniques and while each has it positives and negatives the KISS methodolgy always wins my vote. The more complicated the program the longer it takes to debug, especially if you are not the writer. I have seen code that defies explaination. I have been in more than a few arguments with a programmer on how a machine instruction works. Also when you start programming and find a work around for a problem that either appears to be a bug in the compiler or someother really arcane issue. At the minimum put it in the comments of your code as you never know if down the road the people who created the compiler (subroutine) actually fixes the problem shooting the problem at 0xxx in the AM does not make friends of the person being called in.

I will only repeate #1 comment comment and comment some more. I do not care if you write a book in the comments as long as they are pertenient put them in.

The hard way is more fun

[msobkow]

The truth is that the "hard" way of doing things is often more fun, because you have the challenge of learning a new tool or API. Plus sometimes it's actually easier in the long run because you've engineered a solution for the outer bounds conditions of scalability, so if your application takes off, it can handle the load.

I guess the real issue is that you have to engineer a "good enough" solution rather than a "worst case" solution.

Re:The hard way is more fun

[pseudonomous]

I'd like to add to this that one of classic reasons why someone writes an appllication is because they need program to do something, on the other hand, if you always find and use existing solutions, you end up not writing any but short bash scripts. (I speak from personal experience)

Re:The hard way is more fun

[amaupin]

Also, doing something the "hard way" the first time often leads to a greater understanding and appreciation of the "easy way."

Reading the article, I fail to see why I should avoid PIL for ImageMagick. In neither case is Linux going to just "do it" for me. And in either case I have to tell PIL or ImageMagick how to process my images, right?

Re:The hard way is more fun

[Anonymous Coward]

Not only hard way is "fun", but the guy doesn't know what scalability is. You can't *buy your way out* as he writes regarding scalable systems. You can't throw hardware at a problem that does not scale.

Re:The hard way is more fun

[hedwards]

Eh, isn't that the basic difference between a programmer and an engineer? When I took my programming courses, there was a distinct focus on writing code with the smallest amount of new code possible, using preexisting libraries whenever possible and if there was any suspicion that the code might be useful later to split it off into a subroutine or method of some sort as done in the particular language. Sure it's a good idea to be able to write your own libraries and solutions, but it's a waste of time once you get to the point where you know how to do it. And generally using a library that's shared by other projects is much less likely that you're going to stumble on an undiscovered bugs with your program.

Re:The hard way is more fun

[Morth]

The problem is that you're far more likely to end up with crappy code. If you want to ask your user for a password, it's insanely much easier to do a system("stty -echo"); instead of learning the ioctls to accomplish the same thing, plus you get portability for free.

I recently replaced a SMTP protocol implementation with a pipe to /usr/sbin/sendmail. Guess what? It worked much better.

Re:The hard way is more fun

[jgrahn]

The truth is that the "hard" way of doing things is often more fun, because you have the challenge of learning a new tool or API.

People are different. I hate learning new tools and APIs (although I like learning the ones I already use better).

By extension, I distrust the people who're constantly looking for excuses to play with new toys. When they get tired and give up half-way through, people like me have to clean up after them.

I guess the real issue is that you have to engineer a "good enough" solution rather than a "worst case" solution.

If you mean you have to write shitty code so you don't risk being 70% done at deadline, I don't think that's true. But you probably should use techniques you know well.

Re:The hard way is more fun

[jellomizer]

Well if you are coding professionally. You are wasting time going the hard route unless there is a real good reason to do such. Newbees want to show off that they can do all these things. Mature programmers know they can but if there is an easier way then go for it as it gets your project done faster and you look better to your boss who can give a rats ass on how you did it. Just as long as there isn't any consequences in the future.

Re:The hard way is more fun

[daveime]

Obviously the author had ImageMagick pre-installed on his Linux box, and has never had to actually install the fucker.

What a chore, pages and pages of mumbo-jumbo flying up the shell, then just when you think your done, it turns out it needs all kinds of extra libraries for certain filetypes (installed in VERY specific locations) to handle popular formats like JPG. Why in God's name the supposed "number one image processing solution for Linux" can't even handle JPG out of the box is beyond me.

It's not like you have to pay for the additional libraries, just download them from somewhere else, make some extra incantations, and sacrifice one extra chicken.

Re:The hard way is more fun

[dodobh]

Which Linux distribution doesn't ship with ImageMagick? RedHat, Debian, Fedora, Ubuntu all ship with it.

Re:The hard way is more fun

[jlarocco]

Trolling?

apt-get install imagemagick

Was that so hard?

Re:The hard way is more fun

[msobkow]

Actually I volunteer for the projects that are going to expose me to something new, rather than only taking on projects where I already know how the solution will work. The latter are bread-n-butter to the company, the former are the future of the company.

For example, I've spent the past year on a Freeswitch project rather than on the older Asterisk based code. Freeswitch scales better, is better architected, and is more flexible. The downside was spending 3-6 months working with the Freeswitch team to resolve issues with the code.

In the end, Freeswitch is where we are going; Asterisk is where we were. At the time that the Asterisk code was started, Freeswitch hadn't even reached it's first release, so Freeswitch wasn't an option back then.

Next up is a rework of the database IO codebase so that it becomes feasible to plug-n-play different databases. We could do it with the existing code base, but it would be very painful, kludgy, and difficult to maintain. Instead we're going to make a clean break on our next release to a new architecture for the database code. Sure it'll take longer at first -- but by the time we're on to our third database we should be well ahead of the curve and saving time.

Re:The hard way is more fun

[Anonymous Coward]

I tried ImageMagick once. Its JPEG compression is, I'm sorry, awful. It produced much larger files with very poor image quality than anything else I tried.

Re:The hard way is more fun

[Richard Steiner]

I installed ImageMagick on some of the Solaris servers I work on for my own use (I stick stuff in ~/myaccount/bin) without having any root access (I didn't want to bother the sysadmins to install a formal package), and it wasn't that hard.

Re:The hard way is more fun

[Lunix Nutcase]

Yes, but if you never do anything the hard way because the problems are "already solved" how exactly is one supposed to learn anything at all?

Re:The hard way is more fun

[Anonymous Coward]

Well, as long as you can grok interfaces quickly and having a myriad of dependencies are not a problem in your project..

Maybe it's just me but I'd rather write 5 lines of actual obscure bit-twiddling code than 20 lines of glue.

Re:The hard way is more fun

[Richard Steiner]

Hee! I mean ~/bin. That's what I get for trying to quickly edit without sending ... I originally said /myaccount/bun and then thought it was silly.

Re:The hard way is more fun

[fat4eyes]

Unless of course the additional complexity causes everything to fail even with smaller data because you didn't have enough time or manpower to properly test it. Or that you spend so much time building the complicated system that your startup/project group runs out of money and folds/is disbanded. Starting small and engineering it so that it can be migrated easily to the next level works a lot better that designing a system for a worst case that may never come up in practice.

Re:The hard way is more fun

[Firehed]

Hardware can certainly be the correct answer to the problem. Are you disk-bound? Then you need faster disks. Can't fix that in software. Saturating your pipe? You need more bandwidth.

Yes, you can do things to try and cut back on your use of those resources, but that's ultimately going to produce much less effective results unless you're at massive scale (read: you have your own datacenter) where a 1% performance gain frees up the equivalent of several hundred servers.

Hardware is also a good way to buy yourself a bit more time while you're working on fixing the real bottleneck, such as reworking an expensive database query. Sometimes you just need a couple extra weeks.

Don't read this post as advice to write bad software and only focus on vertical scaling, as that's not what I'm saying at all. But solutions that take you from 1-10k users are very different than going from 10k-1m users, and different still from 1m-10m. Trying to code for 10m when you don't have any users a) is a waste of time, b) is expensive (they rely on the assumption that you have lots of hardware anyways), c) won't work at all, and d) won't help you get users. Don't do dumb things, but users don't give a damn how well your software can scale if it sucks, and if they think it sucks you'll never find out how well it scales.

Re:The hard way is more fun

[Anonymous Coward]

So... apt/yum/fink didn't work for you?

Re:The hard way is more fun

[syousef]

The truth is that the "hard" way of doing things is often more fun, because you have the challenge of learning a new tool or API. Plus sometimes it's actually easier in the long run because you've engineered a solution for the outer bounds conditions of scalability, so if your application takes off, it can handle the load.

I'll counter: There are plenty of technically hard problems to solve without reinventing the wheel or solving simple problems in a more difficult way. Wasting your effort on a solved problem when you could be contributing something new is not in anyone's best interests. As for scalability and bounds conditions, if you pick a well used product chances are it's been not only built with that in mind but had more effort put into testing before you picked it up than you'll ever be able to contribute. It's probably been used in a much wider set of scenarios than your particular use case too. (There are exceptions. If the existing solution is junk, rolling your own can be the way to go....but only after you've exhausted other avenues).

Re:The hard way is more fun

[Anonymous Coward]

Why in God's name the supposed "number one image processing solution for Linux" can't even handle JPG out of the box is beyond me.

Perhaps because the ImageMagick developers are professional enough to realise the best thing is to re-use the pre-existing, debugged, validated JPEG library from the Independent JPEG Group? Which is pretty much the approach this article is advocating.

Re:The hard way is more fun

[danieltdp]

Maybe people will see my post as a troll, but it is not my intention at all: in my experience, your reasoning goes along with people who wants to have fun coding, and not from people who are committed with the final result.

It is the NIH syndrome, a.k.a "I can do it better". Usually the guy looses a couple of weeks reinventing the wheel and turn up a result similar to what was available in first place (assuming it is a good software engineer).

Re:The hard way is more fun

[danieltdp]

One build bridges and the other one codes. Oh, wait!

Re:The hard way is more fun

[Anonymous Coward]

Solve *new* problems, instead of solving the old ones for the Nth time.

Re:The hard way is more fun

[Joe Tie.]

At some point at least, centos didn't. Or possibly it shipped with it, but in a useless state. I remember we used to get calls about it fairly frequently. That was a hell of a long time ago though.

Re:The hard way is more fun

[SL Baur]

I recently replaced a SMTP protocol implementation with a pipe to /usr/sbin/sendmail. Guess what? It worked much better.

It depends. Did you get all the quoting correct when starting sendmail? How did you handle error returns?

For any kind of enterprise code, I'd lean towards an SMTP implementation exactly because when something fails (and pagers are going off everywhere at 3am) you can make a log of exactly what went wrong.

Re:The hard way is more fun

[JanneM]

Yes, but if you never do anything the hard way because the problems are "already solved" how exactly is one supposed to learn anything at all?

You aren't supposed to, when you're on the job. You're supposed to get the product done, on time and under budget if at all possible. Learning stuff is time spent not working on the main goal, and should happen only if there is no other alternative.

Now, in practice I doubt there's many places that take it that far. But learning new stuff for a project _is_ an expensive way (in time) to get results - hence the lure of consultants that presumably already know. Allowing people to spend time learning new stuff that's not connected to the work is very much an indulgence and something to be grateful for rather than expect as a matter of course. Kind of like allowing employees to spend time with a game console or having a reading room with newspapers and fiction.

Re:The hard way is more fun

[cynyr]

but i have to write my image data to a temp file, process the temp file into a second, and finally send it somewhere. That sounds like a very bad plan if you are doing it from a django project, or for that matter from python in general.

Re:The hard way is more fun

[Required Snark]

This is an example of what I call the "engineer's disease." If you give an (electrical) engineer an unlimited budget they first thing they will buy is a bucket and a shovel. That's so they can go to the nearest beach and shovel the sand that they will refine to get their own silicon for their circuits that they design from scratch on their custom built CAD system...

Do you want to do a good job or do you want "way more fun." Most of the time it is one or the other. If you are in the "fun" camp then you are NOT A PROFESSIONAL, and you give the rest of us a bad name with your never quite debugged code that is over time and over budget. Grow up. Get a clue.

Every job I've ever had had plenty of hard problems to solve, and I didn't have to go looking for extra work. If you don't have that experience they either find a different area to program in, or get out of the field. Like I said before, get out of the way and let the grown ups handle it.

Re:The hard way is more fun

[Anonymous Coward]

Databases last longer than code.
The only reason to develop code to handle multiple databases is to support 'enterprise' clients who insist on a certain db, which means you should only do it if some corporate expense account is paying you a million bucks or more. Even then, it may not be worth it. The level of pain is huge.

Re:The hard way is more fun

[marcosdumay]

Do you have any idea of what the system administrators will do with your log? If 'yes', it is perfectly ok for you to bring all that extra complication for the gained tranparency, but that doesn't means it is suitable for other people.

Re:The hard way is more fun

[bbtom]

His point isn't that sometimes you need to write code that solves scalability issues: it is that a lot of Web 2.0 douchebag types think that they need to write their application using NoSQL and Node and all sorts of other fancy scalability shit, when in fact scalability is not their concern yet and that they should shut the fuck up, install a mature RDBMS (like Postgres or The Other Database) and use it until they have a scalability problem.

But they don't. Because geeks love to play with the new shiny toys and brag about it on their blogs.

Re:The hard way is more fun

[SL Baur]

Do you have any idea of what the system administrators will do with your log?

Oh yes, I do. They won't touch it. Their job is only to keep the server(s) running. If it's an application problem, the hammer comes down on me.

The admins only keep the machines running. Application support isn't in their job description.

Of course, we don't leave our logs in system directories. We have our disk where we keep those.

We're talking "Enterprise computing". It's a very different environment.

Lesson #8

[pedantic bore]

Don't ask for advice about programming on slashdot unless you have a pile of salt grains ready.

Re:Lesson #8

[thoughtsatthemoment]

That applies to other sites as well. So many people are trying to preach than genuinely teach. The internet is great for seeking specifics, but for insight on programming, it has to come from within yourself.

Re:Lesson #8

[interval1066]

"...but for insight on programming, it has to come from within yourself."

Uh, this isn't a job for people on a journey of self-discovery, I've found in my 20+ years of programming. For several years as a contractor my #1 task often ended up chasing down problems caused by horrible, horrible database design, and decoding mind-bendingly awful joins to work around the design flaws. And other coding problems that were obviously work of some one who was on some kind of journey of finding "insight from within ones self." There's no substitute for a few solid courses in theory and design.

Re:Lesson #8

[danieltdp]

The reason are two fold:

1) The amount of morons *always* exceeds the amount of experienced and reasonable people

2) Usually, morons don't have a clue that they... well... are morons!

I for one can assure you that I'm NOT a moron :-)

Re:Lesson #8

[thoughtsatthemoment]

Well I don't mean you get insight by just meditation. I was trying to convey a sense of independent thinking, which I believe is that makes a good programmer.

this isn't a job for people on a journey of self-discovery

That seems to have a religious connotation. But I think a great programmer has to wonder a lot about artificial intelligence. And I think that's the ultimate self-discovery for programmers.

Re:Lesson #8

[symbolset]

Ah, yes. There is much wisdom to be found in "The Tao of Programming". For example:

The Tao gave birth to machine language. Machine language gave birth to the assembler.

The assembler gave birth to the compiler. Now there are ten thousand languages.

Each language has its purpose, however humble. Each language expresses the Yin and Yang of software. Each language has its place within the Tao.

But do not program in COBOL if you can avoid it.

Re:Lesson #8

[SL Baur]

There's no substitute for a few solid courses in theory and design.

You must be new here. When I started programming, there were only a handful of colleges that had computer programming departments.

You came along late enough that you might have had decent instructors.

And I walked to and from college both ways, uphill, in the snow, barefooted, my first programming class used paper tape, my second programming class used punch cards, yadda yadda yadda.

Now, get off my lawn!

Re:Lesson #8

[fdrebin]

Ya know, I actually DID walk to classes, barefoot, in the snow. Yes, I was weird. (I growed up as a hillbilly. My mamma never could learn me to wear shoes much.) Now there were was only a very minor hill on the way, but the dorm was not 100 meters from the classroom building.

True, we only learned FORTRAN, using punch cards... then again I was a physics major, not a CS weenie. Those guys were too damned weird... imagine a physicist thinking another group was too weird to associate with.

We did have paper tape available on our KSR33s but it wasn't used all that much. We even had like 3 'glass teletypes' out of about 150 total terminals.

I don't even have a lawn any more.

/F

Re:Lesson #8

[SheeEttin]

The internet is great for seeking specifics, but for insight on programming, it has to come from within yourself.

Reminds me of a joke.
This guy's in New York or something, and he sees this Buddhist guy running a hot dog stand. "Hey," the guy thinks, "It's lunchtime, why not?"
So he goes up to the stand and decides to have a little fun with the guy. He says, "Make me one with everything." They both laugh, and the guy gets his hot dog and pays with a twenty.
He waits for his change, but the guy doesn't seem to be making change, so he asks. The Buddhist looks up and replies, "Change must come from within!"

Re:Lesson #8

[SL Baur]

Ya know, I actually DID walk to classes, barefoot, in the snow. Yes, I was weird. (I growed up as a hillbilly. My mamma never could learn me to wear shoes much.)

The paper tape and punch cards part I wrote was real.

Seeing how you have similar experience I don't whether to invite you inside to talk about the old days, order you off my lawn, or quietly step off of your lawn.

Re:Lesson #8

[Reziac]

[brandishing boxful of punch cards and Fortran manuals]

That's easy: "Hey kids, mow my lawn and I'll give you a quarter!!"

3 things I've learned.

[Anonymous Coward]

Sometimes it's easier and faster to code from scratch than it is to use off-the shelf software - especially in the age of "frameworks".

In that train of thought, its often better to toss and rewrite (or write new programs) than it is to extend existing programs.

It's easier to implement a whole new framework than it is to convince your boss that writing anew is actually faster.

Re:3 things I've learned.

[Trepidity]

Sometimes it's easier and faster to code from scratch than it is to use off-the shelf software - especially in the age of "frameworks".

I like this take on it: redundancy is bad, but the primary way of avoiding redundancy, factoring things out into libraries or frameworks, introduces dependencies, which are also bad. Which is worse? Depends, but I think dependencies are worse in more cases than people believe. I personally like to depend on something only when it's a big enough chunk of code to be worth adding a dependency for, has a clean API, and has an active and interested maintainer.

Re:3 things I've learned.

[danieltdp]

Another things that bites sometimes is that the practice can introduce of and external dependency that will limit the portability of your code. Note that I'm not against the practice, but one has to be careful and understand the consequences.

Re:3 things I've learned.

[Anonymous Coward]

I think part of the thing is don't be afraid to use well known CLI backends if you are programming in linux. You don't need a monolithic app with custom libraries. you can use a bunch of simple CLI tools to manipulate images (convert, mogrify), you can burn cds/dvds with growisofs, cdrecord, you can do video work with transcode, and it's considered ok to have dependencies. I once helped write a script to do measurements with a rs-232 interface, dump them to csv, format them, graph them, put them into latex reports, make a pdf report. Sure it depended on imagemagick, latex, pdflatex, gnuplot, and setserial, but who cares? Doing it all from scratch would have taken months, but it was working perfectly in a couple days.

Re:3 things I've learned.

[turbidostato]

"Sometimes it's easier and faster to code from scratch"

Sure. But they are about ten-fold less common than programers tend to think. They want to start from scratch because they think they'll do it better than their ancestors and they don't want to deep into other's code. Of course they'll fail into many of the same mistakes of the previous implementation and it will be as hard to read for a third party as the one they blamed. Oh! and will push forward time-to-market about twice as expected by the very programmer (with luck) which already is twice as long as if they start working now instead of whinning about how ugly is the inherited codebase.

"than it is to use off-the shelf software"

But we are not talking here about "off-the shelf" software. We are talking here about good in-the-unix-philosophy tools that are designed towards that very end: to be used in collaboration.

Re:3 things I've learned.

[MikeFM]

The hardest thing I've done in years was implementing interfaces to Amazon, eBay, UPS, FedEx, etc. Horribly designed. Badly documented. Constantly changing in small ways. Every time you think you have it some special case comes up or they change something and bang the shit hits the fan again. Very frustrating. I'm sure I could rewrite their code to be decent much quicker than dealing with the crap they throw out.

Give me pure computer science type problems any day where pure logic and good debugging will always lead to software that works.

Re:3 things I've learned.

[gateep]

Why not get the best of both worlds?

I will generally create an API which matches my needs as simply and concisely as possible, then implement that API using a 3rd party library.

This means that my main application code is not dependent on the 3rd party library, only on the API which I designed myself. I can then easily adjust the API as I see fit, or even re-implement it using a different library if a better option becomes apparent. The application code is protected from this, as there is no direct dependency on the libraries.

Re:3 things I've learned.

[Anonymous Coward]

For some of the larger frameworks out there today I'd say the first project can take longer to do with than without. That's assuming the framework is a good fit, if not it will of course be an unmitigated disaster.
You start seeing the returns on the second project, when everyone already know the ropes and can use the framework effectively.

The Kitchen Sink

[Anonymous Coward]

Python already has a library to do the details for you. Perl does too, but the documentation may or may not be as nice and may require a separate install.

xargs <-> mapreduce?

[Anonymous Coward]

I guess I'm an idiot, but.. err... did someone REALLY use mapreduce to solve an argument passing problem (the domain of xargs), or is the writing just shit?

Making use of a database

[sphealey]

A database is not a bitbucket. Re-building basic database functionality in an external app is not a good idea. Applications, frameworks, languages come and go; data remains forever [1]. Business logic is part of the database. If you find yourself adding more and more "application servers" to get performance than you have a fundamental problem with your architecture (and probably a fundamental misunderstanding of how databases work). While it is not impossible to learn and implement good data management/database development practices using Microsoft tools, such a result is seldom seen in the wild.

sPh

[1] Per Tom Kyte of Oracle, whose first database job at the Department of Agriculture involved working with datasets stretching back to 1790.

Re:Making use of a database

[WarwickRyan]

> Business logic is part of the database.

I used to think the same thing, but that was before I ever had to solve a no-trivial problem.

What do I think now? Basically that business logic belongs in a domain layer, but that shouldn't mean that the database is treated like a black box. You should make use of constraints within the database server to ensure a base level of data consistancy. Default values (especially on auditing columns), correct datatypes, relationships, versioning and not-null constraints all belong within the database. Basically anything which can be supported without resorting to triggers or stored procedures.

Databases are excellent at sorting and filtering data. A massive amount of engineering talent has been invested in shaving miliseconds off of their algorithms. So let your database handle that. Don't filter in the application layer and then start complaining about performance. Need more capacity? Cluster. Throw more databases at the problem. Not more application servers. Make sure they're running on properly specified physical hardware, built to best match the actual usage pattern of the database.

Re:Making use of a database

[Anonymous Coward]

Business logic is part of the database. If you find yourself adding more and more "application servers" to get performance than you have a fundamental problem with your architecture (and probably a fundamental misunderstanding of how databases work). While it is not impossible to learn and implement good data management/database development practices using Microsoft tools, such a result is seldom seen in the wild.

sPh

[1] Per Tom Kyte of Oracle, whose first database job at the Department of Agriculture involved working with datasets stretching back to 1790.

Tom Kyle is in the business of selling database servers.

It's interesting how this comment of his doesn't address or acknowledge the many proven capabilities of multi-tier architectures, choosing instead to engage in vague statements and handwaving, while also managing to land a low blow in on their competitor.

Sorry, but I'm going to have to call bullshit on this guy.

I don't want Business Logic in any of my databases; that would require Oracle to sell me the tools to extract it and do anything useful with it, as well as maintain it. I saw them try to push this with their "Designer 2000" crap and it was bottom heavy junk.

Business LOGIC has NO PLACE in a relational data store - this guy is talking out of is ass.

Re:Making use of a database

[Osty]

Basically anything which can be supported without resorting to triggers or stored procedures.

And what's so wrong with stored procedures? Trigger hate I get -- you don't want arbitrary, unknown-complexity code running on every insert. But stored procedures are not triggers. Why exactly do you believe stored procedures are bad? Hint: Any reference to anything MySQL folks have ever said about anything related to relational databases, you automatically lose.

Re:Making use of a database

[swilver]

What's wrong with stored procedures?

1) It means we need to support two languages instead of one, with all that that entails (a proper debugger, expert knowledge, etc)
2) Stored procedures cannot do ALL business logic -- there comes a point where the business logic requires something that is near impossible to do within a stored procedure -- better to just keep it all together in your LANGUAGE OF CHOICE, not in some crappy half-assed non-compiled script crap that database vendor X whipped up in an afternoon because it looks good on their feature list
3) Because you don't want to have to deal with Oracle's horrid error messages anymore than absolutely necessary

Re:Making use of a database

[WarwickRyan]

Main reason is that you're hiding logic in the data layer. You don't want to do that. You should encapsulate all business logic in one place. Which, for systems of a certain complexity, is a domain model / layer.

Stored procedures are generally hard to test, and, unless you're using something like SQL Server's CLR integration the available functionality is limited. Not being object oriented, it's harder to manage complexity.

I've built some pretty complex logic in the past with SPs, and they're extremely useful when you need to do some data-only task with maximum performance. In my experience you'll usually get the task done better and faster (in coding time) with your programming language (assuming that you're working with a domain model etc etc).

Triggers are a similar thing, though it's good to treat them as a case of "I'll only use them if forced to" or if the alternative involves a massive amount of work (and if it does, then the triggers are the least of your problems) OR big risks to data integrity (think database which doesn't support cascade deletes).

Re:Making use of a database

[Osty]

1) It means we need to support two languages instead of one, with all that that entails (a proper debugger, expert knowledge, etc)

If you're talking to a SQL database, you're going to have to write SQL eventually. You could scatter it around all of your code in strings here and there, hopefully at least using parameterized queries to help prevent SQL injection, or you could write stored procedures. The only way to get away from the "two language" requirement is not to use a SQL-based database in the first place.

2) Stored procedures cannot do ALL business logic -- there comes a point where the business logic requires something that is near impossible to do within a stored procedure -- better to just keep it all together in your LANGUAGE OF CHOICE, not in some crappy half-assed non-compiled script crap that database vendor X whipped up in an afternoon because it looks good on their feature list

This point is really two points, so I'll address it as such.

First, you're right, you will run into cases of business logic that you can't do entirely in stored procedures. There's nothing wrong with that, but it's also the exception rather than the rule. Even then, you're still better off doing as much work as you can in your stored procedures so that you can leverage the strengths of SQL (in particular, the fact that it's a set-based language and provides huge benefits if you can model your work as a series of set operations).

Second, if you think SQL is "half-assed non-compiled script crap" you've obviously not done much work with SQL. I'll cede that the major (and minor) db developers have done a piss poor job of implementing the ANSI SQL standard, but if you think stored procedures are "non-compiled" then you obviously don't understand RDBMS engines (or you're using a subpar system like MySQL). For example, Microsoft's SQL Server does in fact "compile" its stored procedures using a JIT process. The engine evaluates the stored procedure at runtime to determine a "plan" based on your data volume and then caches that plan for future calls. To be fair, parameterized queries also have their plans cached so you do get some stored procedure-like benefits from SQL soup in your code as long as you do it right.

Going back to your first point, as long as you're using a SQL database you're going to have to write SQL code somewhere and it's going to have to be database-specific because of your second point. Moving your business logic to app code isn't going to make you any more portable.

3) Because you don't want to have to deal with Oracle's horrid error messages anymore than absolutely necessary

And that explains it. For all of Oracle's power and scalability, their PL/SQL language just plain sucks. Try Microsoft's SQL Server or IBM's DB2 or even PostgreSQL. You might be pleasantly surprised, and learn that the prejudices you built from too-long exposure to Oracle just don't make sense elsewhere.

Re:Making use of a database

[ADRA]

Much of what I'm going to say is specific to my company, but I hope you see some of the patterns as a symptom of the practice besides just the anecdote. Our lead Architect decided that he didn't want Java programmers couldn't be trusted to perform simple business logic operations, so they hired an ever growing lot of trained Oracle DBA's (I refer to DBA meaning all roles related to databases, not just arch/admin) to support their ever growing codebase of stored procedures.

- Our company's DBA's never used change tracking tools to 'manage' their code. When a new version of database X was needed, they cloned an existing one. They had stored procedures in SVN, but they never used, branched, back-pushed, etc.. that storage in SVN may as well have been a backup step.
- They had a single DBA who was able to promote code to any single database (though some DBA's had their own schemas for testing). This meant long delays in projects with several DBA's on the team.
- Debugging in code Databases is a lot harder than firing up a debugger for high level languages (I can't say about MsSQL, but true for any of the others)
- Free/low cost database development software is almost unheard of outside of the few open source tools used for open source databases. There is a substantial cost to outfitting DB developers for the same or similar role you can fill with high level language developers
- Profiling in databases is a matter of getting the most skilled DBA to stare at a program to find the flaws. There are no runtime profilers (At least in Oracle) that I'm aware of beside the built-in cost estimators that is only really good at catching glaringly obvious faults
- Database languages are always more primitive than the high level languages interacting with them
- Database languages generally don't have the facilities to facilitate code reuse / encapsulation. Ok, I may as well say that Database development tools in general suck it in comparison to high level software. If they have ctrl-space completion, you're in the top shelf of DB development software
- Refactoring / runtime diff comparisons (though other runtimes have similar problems)
- Having two codebases and two realms of responsibility adds more friction to the diagnosis and resolution of problems

From my experience, DBA's are generally better at domain definition and structuring, but they're crappy programmers. I've seen consistently higher wages paid to mediocre DB programmers than to adequate programmers. If you take work out of the hands of some and put them into the hands of higher paid others, all things being equal, the cost of production goes up, which nobody wants.

Stored procedures and triggers definitely have their places and benefits throughout the system. I don't want to bash the concepts all together. But a little bit of salt can go a long way. Adding too much salt and things start tasting bitter. Stored procs are amazingly well for self encapsulated data operations that just need to be kicked off and left alone. They can be alright to speed up specifically painful highly filtered data sets that need large arrays of input criteria which span multiple domains
Triggers are great for performing small yet essential pieces of work. We had a trigger that updated an external table that the table contents had been updated. You can throw in calculated fields into triggers if the calculation is table specific or purely parent data driven. Adding too much into triggers or without consistency means risking ridiculously complicated data race conditions or painfully slow write performance.

Re:Making use of a database

[Osty]

Sounds like your problems are specific to bad DBAs. You make a great argument against letting DBAs touch code, not so much against using the full set of features of an RDBMS. When it comes to structure and code, a DBA is useful to help with data layout and optimization, but they shouldn't be writing code (and yes, SQL is code). Your developers should be doing that. If you don't have developers who are competent with SQL or capable of learning SQL, that's a different problem.

Where I work, our DBAs are part of our operations team. They're invaluable for keeping our databases up and running and performing well, and we keep them in the loop during design and development, but design and development is left up to the developers. The DBAs are not allowed to change functionality on running systems (they can tweak statistics and such, but even something like optimizing indexes requires dev contact). If they need to play around, their work is done in scratch databases off of replicas and the feedback gets incorporated into the product by development.

You've seen first-hand what happens if you let DBAs run the system. The same absolutely can happen if you let development run things as well, but at least most dev teams are better about using source control and not changing production systems on the fly. If you can build a good relationship between DBAs and developers, things will run much more smoothly.

Re:Making use of a database

[dcam]

A corollary to this: data structures tend to very long lived. Screw up your database structure and you will have to live with it for a long time. This is typically because it is hard to migrate from one data structure to another.

Re:Making use of a database

[turbidostato]

"What's wrong with stored procedures?
1) It means we need to support two languages instead of one, with all that that entails (a proper debugger, expert knowledge, etc)"

Wrong! *you*, not we. Repeat again: data is company's property, not the application's. You can bet your company data *will* be accessed by more than one application at more than one age if it's of any use (if it isn't the app shouldn't be programmed to start with). It will be the applications' side the one that will add supporting needs for more than one language, debugging, expert knowledge while the data will still use the same SQL language, the same tools and the same domain experts.

"2) Stored procedures cannot do ALL business logic"

True. Neither it's expected to do so. Data-bounded logic should be retained at the data engine level if only to be sure every accessor will comply to it. Your specific app logic can and should be developed within the application. Sometimes it takes some cleverness to find the boundaries, usually not.

"3) Because you don't want to have to deal with Oracle's horrid error messages anymore than absolutely necessary"

This is an argument *against* your position, not supporting it. The way you are free from Oracle's whatever is insuring that data managing functionality is as bound as possible to the data management engine (encapsulation, you know the concept). This way it will be the DBA's problem not yours.

Re:Making use of a database

[b4dc0d3r]

Stored procedures are not hard to test, they just require some preparation. You should be able to write a script with a few DELETE, INSERT, EXEC, and SELECT statements and check the results. Depending on your platform, you can SELECT the results of a proc that returns a data set into another table, checking the results either visually or with a script.

Automated unit testing for each procedure is a lot easier than unit testing statements scattered throughout the code. In fact, if you declare that this project will never use ad-hoc SQL, and everything is done through sp, you can see all of your data operations in one place. Make a table change, recompile all your procs, and see if any other logic needs to be updated.

In my experience you'll usually get the task done better and faster (in coding time) with your programming language

What if SQL is one of your programming languages? Would you concede that you can possibly get the task done faster in SQL in that case?

Also, I don't see the "hiding logic in the data layer" problem. If you limit results to a one week time period, you'd prefer that the same clock be used to generate the timestamps and calculate the date difference. So you could pass in the number of days and select something WHERE timestamp > sysdate()-N if you want. Or use a View, parameterized if necessary, to have the number of days in a single place.

In other words, the data arrangement is one layer, the stored procs and views are the low-level operations that may be repeated frequently. "Give me last week's revenues" is business logic, but it's best handled by the database. "Log in this user with this salted password" seems simple, but the database can be programmed to return 0 results or a specific error message if the user has wrong password, nonexistent user name, expired or banned or inactive account, whatever. Instead of selecting the relevant data and deciphering it application side, the database can return only the data you need. That is a much more scalable solution.

Think about it this way. If the database has to join things and select things and put it together, it can either return all of that up the pipe, or since it's already in memory just go ahead and act on it and return the results. You would never select values from the database and add them in the application layer unless you actually needed to show the values. So where do you draw the line? Let the database play with the data and filter and sort and you still have a layer you can point to as the business layer - the procs and views.

At work, I have 7 web servers and a single shared database cluster (2 nodes) for one of my projects. The web part of the app is mostly a dumb display - the only complicated part is validating security (because we interface with an external identity management tool and that's where the security data is, web side). The web app executes stored procs and puts the results into repeaters for display. It handles the business logic quite well, and I don't have to go searching for every reference to a table in the code if I need to make a change. Sure you can write your own data access layer that has everything in one place, but your webserver is doing both data and render work at that point.

It makes no sense. If I had to use Oracle, I probably would put business logic in the app because I hate working with Oracle*, but I get to use MS-SQL, and it gets the job done, efficiently. Of course, I'm the kind of guy who appreciates writing an analog clock in T-SQL, just because you can.

http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=33357

*(how can you get up to version 10g of a database server and still fail to issue "ambiguous column name" errors consistently, instead arbitrarily taking one or the other depending on how you feel?)

Re:Making use of a database

[b4dc0d3r]

I hate Oracle, but it's not as bad as you make it. I'm no DBA, I only have a few pass-through queries that go through Oracle, but I can diagnose problems and debug and profile and do all of that without having DBA-level access to the system. Knowing how to use a tool is vital for getting good results, and people who know how to use it do get good results.

In fact, I'm going to take issue with every statement you made, except for SQL being more primitive. It is, because it's already complicated enough dealing with low-level data. I won't take the time to deconstruct everything, because you work with Oracle and I try to avoid it so I don't want to get called out on technicalities when I am close but not perfect. But if you choose the right tool for the job, and learn how to use it, every other one of your complaints goes away. SQL will remain primitive, though (unless you move to MS-SQL with .NET CLR procs, which I'm certainly not suggesting).

You work with poor decision makers, that's hard to fix.

Re:Making use of a database

[quanticle]

The problem with stored procs. is that there aren't as many tools for testing and debugging as there are with more mainstream programming languages. I understand that this isn't a total deal breaker - stored procedures are more difficult to test, but testing isn't impossible. I do think, though, that ease of testing and debugging is one factor that needs to be taken into consideration when deciding whether to implement something in the application or in the database.

Re:Making use of a database

[GWBasic]

So basically, if you're dealing with a large set of persistant data, everyone has to know how to work with a database. Avoid programmers who don't know how to use a database, or isolate them to a GUI.

Re:Making use of a database

[Qzukk]

The way you are free from Oracle's whatever is insuring that data managing functionality is as bound as possible to the data management engine

And then checking all the constraints again in the application so that the user gets told that they need to select a thingamabobber, not DATA ERROR E0152 NON-NULL CONSTRAINT VIOLATED

Re:Making use of a database

[Anonymous Coward]

I'm a DBA and SQL is not the ultimate solution to all your problems.

Do you believe in silver bullets, herding mentality? Fine then SQL is for you, go for it!

Serious architects, on the other hand, starts by considering what structures will deliver what ever they are working on with the best outcome.

Is it a graph, a hash table, a tree, a [put your structure here] or perhaps a mix of this and that?

Second or perhaps first, what are the access characteristics. Is ACID needed?

Third or perhaps first, what are the performance requirements? Perhaps this or that data needs to be in continuous blocks of memory optimized to minimize cache misses. Who the fuck knows?

But, I'm glad...lame system designers who think it's a good idea to jam absolutely everything into RDBMS, claiming they are excellent at sorting and filtering data (all kinds of data right? ;-)) like you are after all what keeps me occupied :)

Keep at it.

Re:Making use of a database

[jadavis]

need to support two languages instead of one,

In PostgreSQL, you can write functions in a lot of languages, and if your favorite language doesn't happen to be supported, you can probably add support without even restarting.

So, none of your arguments apply to postgresql.

Re:Making use of a database

[WarwickRyan]

Sounds like we've got a similar background. I've been developing with SQL for the best part of 14 years, longer than I've been coding Java or C# (10 and 7 years respectivly).

MS T-SQL is really nice (MSSQL2005+). It allows fairly clean code, and can be very *very* fast in execution.

One of my previous large projects was implemented almost entirely in MS-TSQL with a very dumb web layer (which is purely display).

The project which I'm currently working on is the replacement for that. We've moved all of the logic from the database and implemented it in the application through use of a domain model.

Why? Because there's a point where the logic is simply unmanagably complex. Even with a lot of effort put into TSQL code reuse we've still got hundreds of scripts which can be replaced by a few dozen objects.

It's questions like "this customer must pay this rate on mondays, that rate on tuesdays unless they've got a blue shirt on. In which case they use a totally different rate." which are hard (or a lot more work) to implement in TSQL.

Sure, you can go a long way whilst implementing everything in TSQL. If you don't need complex logic then you'll likely outperform any object oriented application.

What I'm really saying really boils down to this: use the right tool for the job.

Don't know what that is because the job isn't clear enough? Then start with the easiest (and fastest) option, but be prepared to throw everything away and start again.

Re:Making use of a database

[WarwickRyan]

Just to add, when I said this:

In my experience you'll usually get the task done better and faster (in coding time) with your programming language

I was refering to a problem which involves *complex* logic.

Re:Making use of a database

[turbidostato]

"And then checking all the constraints again in the application so that the user gets told that they need to select a thingamabobber, not DATA ERROR E0152 NON-NULL CONSTRAINT VIOLATED"

Then again, an encapsulation problem: an app shouldn't allow for low tier errors to arise but should cope with them.

Re:Making use of a database

[michaelok]

Interesting you mention PostgreSQL. There was a recent presentation on InfoQ by Andres Kutt, one of the architects of Skype, and they use PostgreSQL quite heavily (and are also contributors), and also put business logic in the databases. He was initially against this, but it's worked out well. They have use some other interesting components from PostrgeSQL, such as their messaging system. Worth seeing:

Learnings from Five Years as a Skype Architect

Quite like a decent system architecture gets turned into a convoluted pile of spaghetti by mindless technical changes, the functionality of your application will be turned into a similar mess by mindless changes in functionality.

no MVC pattern...

[drolli]

unless tcl/tk wont do the job

Re:no MVC pattern...

[buchner.johannes]

Here is my advice for the day:

Don't build GUIs. They are dead ends.

The reason that they are dead ends for the developer is that the code is not reusable, and that web interfaces and desktop interfaces are not transferrable (even between desktops due to different UI guideline approaches). The reason that they are dead ends for the user is that he can't automate tasks with a GUI.
I know it's sad.

Re:no MVC pattern...

[drolli]

Excuse me *how* is the code not reusable by definition? And *why* does a GUI prevent automation?

A GUI should assist the User in entering exactly the information which can be entered in a structured way. Nobody forces you to force everything in a *bad* GUI.

2-port programs, Linux, PIL, expensive hardware

[tepples]

If you are writing a program that touches more than two persistent data stores, it is too complicated.

I disagree. Is a program too complicated if it has 1. input, 2. output, and 3. logging? Is a program to prepare images for an online store too complicated if it reads 1. raw source images and 2. an overlay image and writes 3. finished images?

If Linux can do it, you shouldn't.

That'd be fine if we all ran Linux. But in an organization that already has to run Microsoft Access for other reasons, we have to take Windows into consideration. And I don't think Ted Dziuba was talking about just using Windows as a shell to run Linux in VirtualBox OSE either.

Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.

Our programmer is far more experienced in Python than in bash, and if I felt like it, I could benchmark PIL against subprocess.Popen(['convert', ...]).

if the physical machine is not the bottleneck, do not split the work to multiple physical machines.

Yet PC game developers split a 4-player game across four PCs when one could do, and increasingly, PS3 and Xbox 360 game developers are following the same path.

It is far more efficient to buy your way out of a performance problem than it is to rewrite software. When running your app on commodity hardware, don't expect anything better than commodity performance.

If you are writing software to be used internally, sometimes springing for better hardware is worth it. But if you are writing software to distribute to the public, you can generally assume your customer has commodity hardware unless your software costs at least 1000 USD a seat.

Re:2-port programs, Linux, PIL, expensive hardware

[Peach Rings]

Funny how in the same post he bashes EC2 for selling brute-force massive scaling and then says it's better to buy more hardware than to fix your application...

Re:2-port programs, Linux, PIL, expensive hardware

[JamesP]

I disagree. Is a program too complicated if it has 1. input, 2. output, and 3. logging? Is a program to prepare images for an online store too complicated if it reads 1. raw source images and 2. an overlay image and writes 3. finished images?

Logging is write-only for the program. But yes, the definition is debatable (not the 'spirit' so much).

If Linux can do it, you shouldn't.

That'd be fine if we all ran Linux. But in an organization that already has to run Microsoft Access for other reasons, we have to take Windows into consideration. And I don't think Ted Dziuba was talking about just using Windows as a shell to run Linux in VirtualBox OSE either.

But windows and libraries does a lot as well. The problem here is reinventing the wheel.

Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.

Our programmer is far more experienced in Python than in bash, and if I felt like it, I could benchmark PIL against subprocess.Popen(['convert', ...]).

Yeah, debatable. But it's sometimes easier to do in ImageMagick, if you need only file->file ops.

Re:2-port programs, Linux, PIL, expensive hardware

[Anonymous Coward]

if the physical machine is not the bottleneck, do not split the work to multiple physical machines.

Yet PC game developers split a 4-player game across four PCs when one could do, and increasingly, PS3 and Xbox 360 game developers are following the same path.

Except in the case of a 4-player game, the physical machine is the bottleneck. You can't have 4 players simultaneously use the same set of input peripherals, display, speakers, &c.

Re:2-port programs, Linux, PIL, expensive hardware

[SanityInAnarchy]

Actually, he bashes EC2 for virtualizing away the hardware, meaning you don't know how long fsync takes.

But then he bashes people for not benchmarking. So... erm... benchmark EC2?

Re:2-port programs, Linux, PIL, expensive hardware

[kannibal_klown]

If you are writing a program that touches more than two persistent data stores, it is too complicated.

I disagree. Is a program too complicated if it has 1. input, 2. output, and 3. logging? Is a program to prepare images for an online store too complicated if it reads 1. raw source images and 2. an overlay image and writes 3. finished images?

If you program in the corporate world, 2+ data sources are common. Sometimes your program may need to access different centralized corporate data that isn't allowed to be replicated or you using database-links.

Your input/output/logging is also a good example.

Re:2-port programs, Linux, PIL, expensive hardware

[Anonymous Coward]

Way to miss the point of the post and just glom onto the trivialities. Congratulations on either missing the central thesis or choosing to ignore it purposefully so as to allow yourself to appear superior.

Re:2-port programs, Linux, PIL, expensive hardware

[Anonymous Coward]

Plus, ImageMagick offers a perfectly fine library interface. It's much easier to use since you don't have to juggle subprocesses and in my experience it is much faster.

Re:2-port programs, Linux, PIL, expensive hardware

[turbidostato]

"if you are writing software to distribute to the public, you can generally assume your customer has commodity hardware unless your software costs at least 1000 USD a seat."

Well, the biggest commodity software vendor, Microsoft, seems to disagree with you. Remember Vista?

Re:2-port programs, Linux, PIL, expensive hardware

[skids]

Yeah, debatable. But it's sometimes easier to do in ImageMagick, if you need only file->file ops. ...for a reasonably normal size of file, sure. But last time I looked, ImageMagick didn't stream its calculations and used intermediate temporary buffers containing the entire data, so its memory profile was awful. Not sure much of any of the tools do -- maybe gstreamer but that might be frame-by-frame only, I forget.

So things I wish I had known ahead of time: tools that already do what you want might not actually, when you push them past what most people use them for.

Re:2-port programs, Linux, PIL, expensive hardware

[Gverig]

Hear, hear!
Pretty much every point of the article can be prefaced with "if you have no clue what you are doing, ".

Overall, article seems to be pretty worthless. "don't just do for the hell of it, think first"- well, erm... a) duh; b) yeah, I know anybody actually doing this is a rarity but that does not change the fact that [see 'c']; c) duh!

I RTFA

[Jorl17]

I just RTFA. It isn't that good. These aren't many tips and they also don't seem to be too specialized. Most of them are already known or predictable. I can say that I didn't learn anything from TFA.

I think that anything that reads "___ things to know about ____" or similar gets instant hits on /.

-1, Boring from me; hope it helps others.

Re:I RTFA

[Anonymous Coward]

What would the Daily WTF do if it wasn't for these kind of mistakes?

Re:I RTFA

[Sarten-X]

I learned that the author has spent an hour working with NoSQL solutions and thinks he's an expert. But hey, let's go ahead and make more giant SQL databases for all situations. After all, those DBAs need jobs, too.

Re:I RTFA

[Jorl17]

oooh, he hit you right in the NoHeart, didn't he?

Re:I RTFA

[danieltdp]

Its your fault. Why in hell a slashdotter will spend its time reading the freaking article? Shame on you.
I for one didn't even think about clicking that link

Re:I RTFA

[ADRA]

What can you say? Its a blog. Is it slashdot worthy? No. Can you blame the 'article'? No, because its a BLOG. Its a subjective opinion piece like so many online by a single person's experience.

Re:I RTFA

[Bigjeff5]

I don't see any practical difference between a blog and an editorial, and editorials are newsworthy.

Rejecting the information/argument solely because of the delivery method is a classic logical fallacy.

Re:I RTFA

[Anonymous Coward]

Yeah, its a real bombshell. Dont write code when you can reuse something that is already freely available and stable.

It is how ever a continuation of learing about what is available on linux. I remember being told Dont use AWK if SED can do it, don't use SED is GREP can do it, dont use GREP if TR can do it.

Re:I RTFA

[thePowerOfGrayskull]

ANd a couple are just incorrect, or at the very best subjective. I'm not sure how this made it to the front page.

Re:I RTFA

[Jorl17]

I'm not sure how this made it to the front "page."

You must be new here.

Hehe.

Re:I RTFA

[Sarten-X]

I just don't understand the hatred of NoSQL. Apparently the use of cheap terabyte drives is supposed to take care of storage problems, and replication for reliability. What about datasets in the tens or hundreds of terabytes? Heck, my project at my company has already used 15 terabytes, and I've barely started. My project also has some write speed requirements that can't be met by any SQL system. After much research, a distributed NoSQL solution works best.

My project also touches a MySQL server, that handles a lot of the read-intensive processes. Again, this is the result of a long research phase. Relational databases enforce the structure and consistency that works well with our application development, and it's faster to read than the NoSQL source.

Both solutions have their benefits, and both are appropriate for different circumstances. This also blows a hole in the "only one persistent store" idea. Systems should use the resources they need, and no more. It seems obvious to me.

Re:I RTFA

[Jorl17]

"Both solutions have their benefits, and both are appropriate for different circumstances. This also blows a hole in the "only one persistent store" idea. Systems should use the resources they need, and no more. It seems obvious to me."

Now THAT is what should be posted all over Slashdot. It is true.

I wish I'd known that CMS's are really hard

[petes_PoV]

Sure, you can do all the superficial stuff with simple point and shoot. But I wish I'd known (before I started?) that to do anything beyond that meant diving deep and dirty into PHP, Javascript, CSS, DOM, and a whole lot more alphabetti spaghetti.

And that's before you get into the really difficult stuff (that very few have managed to master) of getting a website that is easy to navigate and intuitive to use

Re:I wish I'd known that CMS's are really hard

[Peach Rings]

That stuff is really easy though, and it's a lot better to have to muddle around in PHP and HTML/CSS/Javascript than to have a "platform within a platform" with pluggable modules and highly configurable styles.

Re:I wish I'd known that CMS's are really hard

[Anonymous Coward]

ALLLPHABETTI SPAGHETTI??!

Re:I wish I'd known that CMS's are really hard

[jimicus]

I don't consider most CMSs to be particularly mature, for exactly this reason.

Rationale: There are thousands, if not millions of small businesses worldwide that would benefit from a CMS-based website where they had to do precisely zero management. Where the business owner clicks "Sign Up", chooses a template (from a decent selection), messes with it as s/he likes and adds content without once having to write a single line of PHP - and preferably without even writing any HTML. Such businesses typically don't have the money to contract a web developer much beyond the initial setup, so usually wind up with one of the following:

1. A totally static site that never gets updated.
2. A CMS-driven site that gets compromised and is abused to sell viagra. (You seen how often a typical PHP CMS requires security updates?)
3. Cobbling something together themselves that looks like arse.

There are - to my knowledge - three or four companies offering such a service. AFAIK, one is a startup using Drupal version 7 (which I believe is still in beta) - the others seem to have built their own CMS.

They built their own CMS despite there being dozens out there. I wonder why that could be?

Re:I wish I'd known that CMS's are really hard

[HawaiianToast]

Yeah, having to use the technologies a piece of software is built on in order to extend it is just plain bad design. Uh huh.

Re:I wish I'd known that CMS's are really hard

[Xyrus]

That's because there is no true "Web API". You've got multiple standards with multiple languages and multiple frameworks and multiple browsers all trying in multiple ways to be THE solution to web development.

It's a mess. Comparitively speaking developing "normal" applications, even massively parallel computational models, seems easy by comparison.

Re:I wish I'd known that CMS's are really hard

[buchner.johannes]

There are thousands, if not millions of small businesses worldwide that would benefit from a CMS-based website where they had to do precisely zero management.

Here is a piece of wisdom:

The reason why CMS's, calendar software, TODO lists and PIM systems are not solved yet is not due to a technical reason.

Every company, in fact every person has a different approach on how a calendar and a content system should look like, how it should behave, and what tasks it should perform. Sometimes you can take one product and bash it until it roughly does what you want. Even if that works, it takes time and knowledge of existing products. That's why everyone makes their own CMS, the drawback being that there is no upgrading to new features.

It can't be solved because there is a conflict between "I just want it to do Y", "I don't want/have the time to learn X, or find out what can do Y" and simply that everyone works in a different way.

My advice...

[Anonymous Coward]

Don't listen to other people they are the biggest source of painful misconceptions.

Re:My advice...

[Anonymous Coward]

Listen, but don't take it as gospel. Verify.

until you find out why Python doesn't do the job

[tepples]

Don't program in C if Python will do the job.

But in a lot of cases, Python does not do the job. It doesn't do the job on iOS because Apple has explicitly banned everything but Objective-C++. It doesn't do the job on Xbox 360 or Windows Phone 7 because IronPython uses Reflection.Emit, and the version of .NET used by XNA doesn't support Reflection.Emit. And it doesn't do the job on Nintendo DS because the runtime uses up a lot of the available 4 MB of RAM.

What I want people to remember

[93 Escort Wagon]

Don't assume that, even six months from now, you're going to remember why you did things a certain way.

And the corollary: Don't assume you're going to be the one modifying the code a year or two from now.

Either way: Add comments liberally. Even if you're a conservative.

Re:What I want people to remember

[AnonymousClown]

I agree. Here's what I normally do when I code - regardless of language (For 'C' style languages):

/* (and a lot more asterisks)
* Function: FooBar
*
* Input: void.
* Output: Returns 'true' if there's a bar
*
* Purpose: This function checks to see if there's bar etc...etc...etc...
*
*
* (a lot more asterisks)*/

I tell you my employer LOVED it. It made it much easier for the Indians to understand my code when they took it over after the company canned me.

Re:What I want people to remember

[daveime]

please sir to be sending me an explanation of your code. I am reading your comments, but you have not explained HOW function FooBar works, only that it is doing something and getting this result. there is no input, so what bar are we checking ? please to be answering quickly sir, as my employer will be withholding my $2 an hour salary until I fix this.

Re:What I want people to remember

[Joebert]

This comment deserves a Yosemite Sam standing just off the edge of the cliff about to fall into the gourge laugh.

Re:What I want people to remember

[T.E.D.]

Always write code as though it will be maintained by a homicidal, axe-wielding maniac who knows where you live.

Re:The hard way is more fun^H^H^H educational

[petes_PoV]

You might learn something from doing things the hard way, but all you'll achieve is a version #1. As we all know (or will learn) version #1 of pretty much everything should be thrown away and should NEVER see the light of a production server. However, timescales being what they are as soon as an application gets close to functional it gets snatched away and put live - no matter how ugly it is. After that, all you ever have time for is to patch the worst parts. Doing a complete rewrite from the ground up, to do it right, is a luxury few of us experience.

Re:xargs mapreduce?

[tepples]

I guess I'm an idiot, but.. err... did someone REALLY use mapreduce to solve an argument passing problem (the domain of xargs), or is the writing just shit?

xargs and its successor GNU parallel implement the "map" part of MapReduce.

Re:HUH?????

[Anonymous Coward]

Agreed. The entire article is awful.

newfangled stuff is not worth the risk

That's why you test new stuff, idiot.

If you have a process running and you want it to be restarted automatically if it crashes, use Upstart.

Hang on a minute, new stuff is not worth the risk, apparently. Perhaps when he said "Upstart" he meant "mon", or "monit", or one of the other well established process monitoring tools?

Parallelize When You Have To, Not When You Want To

I'd like to see this wonderkid take his non parallel code and parallelize it in five years time when he realises his non-parallel implementation doesn't scale.

Crap, crap, crap, crap, crap.

Thing I wish _others_ would know

[melted]

Do not make things super-modular and generic unless they 100% have to be. In 99.9% of the projects no one, including yourself, will use your stupid dependency injection, and logging / access control can be done just fine without AOP. Don't layer patterns where there's no need. Aim for the simplest possible design that will work. Don't overemphasize extensibility and flexibility, unless you KNOW you will need it, follow the YAGNI principle (you ain't gonna need it).

Re:Thing I wish _others_ would know

[Anonymous Coward]

I'd combine this with the earlier replies about thorough commenting:

When you follow that advice and don't make things "super-modular and generic", for !#%!%! sake, document the limitations of your quick-and-dirty implementation in the code. If it does the job at hand, then there is nothing wrong with it at all. Get it done. But at least take a few seconds to put "NOTE: Does not handle case X" or "Depends on this being true", so that when someone tries to re-use your code later, you will realize whether or not it is up to the task. This holds even if you are only writing code for yourself, such as when you come back to the code months or years later after you've remembered "Oh yeah, I did something like this before", but forget the limitations. Writing reliable, generic, modular code is difficult and time-consuming. Writing "Beware of this limitation" is not, so do it!

I'm just re-writing some code now. The original code made the assumption that there would be only one return from a listing of files with a particular pattern -- so I said exactly that in the comments. I also said if the assumption was ever wrong it would have to be fixed. It sure made things a lot easier when I looked at that code months later because it was breaking in certain situations, and there the problem was, already documented as something that might someday need fixing. Even better is if you put a little trap in the code to pop up a "This should never happen, but if it does it needs to be fixed" message at key points. It saves the time of implementing a condition/fix you don't think will be necessary, but still provides useful information if it ever does.

Re:Thing I wish _others_ would know

[Anonymous Coward]

(too lazy to log in)

hear hear.

i'm working in an over-architected codebase now and all the Design Patterns thrown in haven't done squat for making it flexible or modular,
they've just made it obtuse.

Re:Thing I wish _others_ would know

[Anonymous Coward]

Agree. I've never worked as a software developer, but studied computer science.
Been programming since I'm 14 or something. Before I was thinking too much about reuse and 'perfection', I usually achieved my goals.
At some point I got into C++, became a C++ standard fanatic, new all quirks with templates etc., read a lot about design patterns, and successively I always wanted to write the 'best' ... whatever that is ... or most reusable code ever.
Spend sometimes weeks trying to satisfy my high expectations on my code, often throwing it away at some point anyway due to not being pleased with something I didn't think of in the beginning, instead of just trying to complete the job efficiently.

Re:Thing I wish _others_ would know

[Bill, Shooter of Bul]

Absolutely, I'm often the target of such advice, however the alternative solution to mine usually involves excessive code copying of non trivial business logic. I thik you should design something the most obvious way, then step back and figure out the simplest way to do it.

Re:Thing I wish _others_ would know

[Anonymous Coward]

It's a good thing to strive for perfect code but you have to remember that it won't work out that way. Do a good design, avoid messy code, use design patterns but if you need to have something working you'll have to accept problems in the design, let that nasty looking (but functionality wise, correct) code stay in the code base and break the rules of a design pattern to get it done. Just make sure you document the issues, make sure that even though you are not 100% happy, that the requirements are met and that the code is tested and working properly. Next time you see a similar problem you now have experience to use the improved design over the design you learned was wrong. You can then make less nasty code to work around the design (which will have new problems), you will know some of the pitfalls and avoid them and use better design patterns.

Writing good software takes experience and that some from learning what was wrong last time.

Re:Thing I wish _others_ would know

[hal2814]

Obligatory: "Kahn, I'm laughing at your superior intellect."

What frustrates me more than anything else as a programmer is to walk into an over-engineered situation needing to make what should only be a slight alteration. Not only are over-engineered solutions unnecessary, they're generally hard to maintain.

Re:Thing I wish _others_ would know

[Anonymous Coward]

"I've never worked... yaddah yaddah... I tend to do things in a particular way, then I fond a different one and I felt in love with it... yaddah yaddah... then I found it was not the perfect lover I thought... yaddah yaddah...

News to you: that's a known process called "learning". Leave with it.

Re:Thing I wish _others_ would know

[The Clockwork Troll]

Dependency injection is a real boon for being able to test components while holding their dependencies constant. Not using it, you're committing yourself to all kinds of fakery to simulate mock database connections, 3rd-party resources, etc. - or you're just committing to not being able to unit test a component in isolation.

Re:Thing I wish _others_ would know

[melted]

>> you're just committing to not being able to unit test a component in isolation

Horseshit. You can supply mocks to your component just fine, provided that it's properly designed. I don't see how dependency injection frees you from creating mocks for external dependencies.

This is te problem with Linux

[bogaboga]

"Modern Linux distributions are capable of a lot, and most hard problems are already solved for you. You just need to know where to look."

First off, I must say this piece says a lot about the Linux ecosystem. Specifically that this system's documentation is anemic at best. Why won't we have something like:

"What do you want to do?...with an associated answer...this kind of arrangement surely cannot hurt the Linux ecosystem.
 

Re:This is te problem with Linux

[jpate]

apropos

Re:This is te problem with Linux

[costing]

whatis apropos

Re:This is te problem with Linux

[danieltdp]

And if apropos don't help, you can alway pray

Re:This is te problem with Linux

[zippthorne]

Inefficient.

man -k

Also, that only works if you have the thing you need. You might need to do

aptitude search

to find what you're looking for...

Side note:

Why is the good coreutils documentation in info, which on linux distributions is so useless that no one ever thinks about using it?

Re:This is te problem with Linux

[Hinhule]

I thought it was some kind of acronym, turns out it was a word.
http://dictionary.reference.com/browse/apropos

Re:This is te problem with Linux

[jpate]

Is man -k different from apropos? man man says man -k and apropos -r are equivalent. And, on my system at least, apropos is just a symlink to whatis.
And actually, I'll need to do:

pacman -Qs

Not every distro uses aptitude ;)

Re:This is te problem with Linux

[Anonymous Coward]

This problem is solved by the package-manager of your linux-distribution.
Say that I'm looking for another editor (on gentoo):

eix app-editors/

shows me all available editors.
Perhaps I'm looking for an editor specifically to edit latex-files:

eix app-editors/ -S latex

shows me editors specifically designed for editing latex.
Most popular distros provides guis for their package manager, if you prefer that.

How does any other system handle this better? As far as I know, the best solution for other systems is "google it". A solution that works just as fine on linux.

Re:This is te problem with Linux

[El_Oscuro]

$ whatis apropos apropos (1) - search the manual page names and descriptions $

Re:This is te problem with Linux

[skila]

"What do you want to do?...with an associated answer...this kind of arrangement surely cannot hurt the Linux ecosystem.

What, like putting oft-used examples at the top of the man page or in the cruft? That's too creative man, next people will be asking for an exit button in vi, I mean emacs, oops, no I mean nano.

Re:This is te problem with Linux

[Anonymous Coward]

so... you want to port clippy to linux?

Re:This is te problem with Linux

[Anonymous Coward]

We do - it's called google. Extra tip: if you don't find what you're looking for on your first go, I find it helps to prepend your query with ubuntu.

Re:This is te problem with Linux

[Bratmon]

And actually, I'll need to do: pacman -Qs

He says for programs you don't have yet. You want pacman -Ss

Re:This is te problem with Linux

[jpate]

d'oh! honestly, I never use that command; I don't like installing things that I've not read up on so I usually discover new software by way of forum and wiki.

I wonder how long we can keep this going until the off-topic mods descend...

Re:This is te problem with Linux

[zippthorne]

man -k is one character fewer than apropos It's not really any better, unless you're a keystroke miser, or occasionally pretend to be one. Or you're searching for a *lot* of things you don't know of yet...

Re:This is te problem with Linux

[Anonymous Coward]

Yeah, cause then you'll find 5 noobs asking the same question in the ubuntu forums, with nobody who knows what they're doing around to answer them.

Seriously, I often find useful discussions in the Ubuntu forums when I'm trying to investigate the state of hardware support (and/or instructions for the less-automatic hardware support) for some random laptop or UMPC I'm looking into, but for any generic technical questions, it's ridiculously common to find one or several unanswered (or answered by a series of equally confused replies, with no solution) posts on there.

My theory: communities like this are, at least from one perspective, a knowledge pyramid -- you have a bunch of newbies who just about know how to install, if everything goes right, a good many intermediate users who can mostly keep their desktop up-to-date and running smooth, with some googling as needed, and a few experts, who just know what they're doing (and a handful of kernel hackers, distro-makers, and the like at the top). As newbs are recruited, everyone else moves up the pyramid, by a process like:

1. post newb questions at linuxquestions.org
2. get answers
3. learn more
4. answer noob questions while asking fewer and more advanced questions
5. repeat
6. ???
7. Profit!!!

But with Ubuntu being known as The Distro for noobs, it seems that the growth rate has exceeded the education rate, so the whole pyramid becomes too bottom-heavy, and now tons of noob questions go unanswered, lost in the noise. Those noobs don't learn (or learn more slowly, anyway), so they don't advance as more pour in. To make matters worse, since Ubuntu is supposed to be "easy for noobs", some users are starting with it, then after a month or two of noob questions, they try out some other distros, find one they like better, and switch to it -- not that they don't "give back" to the Linux community as a whole, but they added to the load on Ubuntu, then make their own contribution in another subcommunity.

I don't really know what the long-term effect is -- maybe Ubuntu caves in under its own weight and some other distro takes its place, maybe the noob influx slackens for awhile and a wave of noobs does move up, rebalancing things, maybe some useful non-denominational megaforum takes over, allowing other more mature subcommunities to give Ubuntu users a hand up. I certainly wish them the best -- Ubuntu's not for me, but for those who are trying it, asking questions and getting no reply has got to be discouraging.

Re:This is te problem with Linux

[jpate]

apro<tab> :)

Re:This is te problem with Linux

[Zaiff Urgulbunger]

There is a lot of noise on the Ubuntu forums, but I can't help thinking that it doesn't help that that forum (and soooo many others) have a useless search faciility -- if I search for "term1 term2" it'll bring up threads that mention either term. If I use Google instead, and search (say): "ubuntu 9.10" foobar it'll bring up pages for different versions of Ubuntu, because rather brilliantly, the pages on Ubuntu forums mention loads of other things on the same page, e.g. the Ubuntu version each user uses.

So I can kind of understand why people end up posting the same questions over and over!

Re:This is te problem with Linux

[PybusJ]

Only in some 1990s unix-like world where the manual pages are comprehensive, up to date and reasonably well organised.

As time goes by this applies to recent linux tools less and less. With many projects the only replacement seems to be googling for blog postings.

Re:This is te problem with Linux

[Gunstick]

site:ubuntuforums.org term1 term2

Why they insist of not simply putting a google search link on the forums is a mystery to me.

Re:This is te problem with Linux

[jgrahn]

Only in some 1990s unix-like world where the manual pages are comprehensive, up to date and reasonably well organised.

As time goes by this applies to recent linux tools less and less. With many projects the only replacement seems to be googling for blog postings.

The sucky GUI and Web 2.0 stuff might have that problem, but the good tools have (IME) excellent documentation. Or what tools did you think of, specifically? Perhaps I'm stuck in the 1990s; if so, I enjoy it here.

Re:This is te problem with Linux

[marcosdumay]

The oods of having the stuff you need are so low that you should default to aptitude search, not apropos. Unless you are doing a very common operation.

Re:This is te problem with Linux

[marcosdumay]

And actually, I'll need to do:

pacman -Qs

Not every distro uses aptitude ;)

You may not always have aptitude, but the odds of finding the right tool on one distro that doesn't use it are much lower, so you may want to search the debian packages...

Re:This is te problem with Linux

[zippthorne]

Depending on how common it is, you should default to info coreutils or man bash Two places where there is a plethora of knowledge that ought to have been divided up into smaller manpages.

Overengineering can be a good thing

[caywen]

You know, I find that as I get older, I am able to avoid overengineering things a lot better than when I was twenty something. There's nasty effect, though. I'm learning a lot less in depth about systems than I normally would.

Overengineering is terrible for a project, but it often is highly educational.

Re:Overengineering can be a good thing

[naasking]

You know, I find that as I get older, I am able to avoid overengineering things a lot better than when I was twenty something

The Evolution of a Haskell Programmer.

In Defense of Not-Invented-Here Syndrome

[satuon]

... When you're working on a really, really good team with great programmers, everybody else's code, frankly, is bug-infested garbage, and nobody else knows how to ship on time. When you're a cordon bleu chef and you need fresh lavender, you grow it yourself instead of buying it in the farmers' market, because sometimes they don't have fresh lavender or they have old lavender which they pass off as fresh.

I read this in Joel Spolsky's article (http://www.joelonsoftware.com/articles/fog0000000007.html) but I think he has a point.

One more tip

[Posting=!Working]

Visual Basic - Don't. Just don't.

It's always great to learn a language then have the company change it so drastically in the next version that all your knowledge of the language is useless. I don't believe it'll be the last time that happens either. I do know I will never bother to learn another MS programming language again.

Good luck to all you C# programmers when they switch to C#.NET, or whatever they call the next one. Hope you like reading!

Re:One more tip

[radish]

Right - but a real programmer realizes that languages always come and go, that the real skill is PROGRAMMING not C# or Java or C++ or Python or whatever. If you have a real understanding of the fundamentals then learning a new language is usually easy (and often kinda fun).

Re:One more tip

[Anonymous Coward]

Asinine. If you used VB you got what you deserved, it's a silly pointless language now and it was then. C#.NET...seriously?

Re:One more tip

[bertok]

Visual Basic - Don't. Just don't.

It's always great to learn a language then have the company change it so drastically in the next version that all your knowledge of the language is useless. I don't believe it'll be the last time that happens either. I do know I will never bother to learn another MS programming language again.

Good luck to all you C# programmers when they switch to C#.NET, or whatever they call the next one. Hope you like reading!

C# is already .NET, there has never been any other version.

And what about C++, hmm? That's a language that went from "C with classes" to a multi-paradigm language with a sprinkling of template mate-programming programming thrown in! The shift was so huge, in most compilers, the standard libraries have two versions, a legacy and a modern template version.

I had to re-learn C++ several times. Some of the new features are so advanced, that not even all compilers support it. Things like "partial template function specialization" are only turning up now, 10 years after standardization. It's a language where I was shocked to learn that there were entire language features I wasn't even aware of after having used the language in production code for 20 years! That kind of thing has never happened to me before or since with any other language.

Re:One more tip

[Dr_Barnowl]

VB6 and below just made it easy to screw up. If you knew what you were doing, you could crank out applications entirely fit for purpose, in a fraction of the time of a C equivalent.

VB.NET is an entirely different prospect though. I have deliberately avoided it for a number of reasons.

First amongst these is that C# is nicer, cleaner, and better paid for, so why learn VB.NET when it targets exactly the same libraries and runtime, and even interoperates with components written in C#.

Secondly, VB6 is the COBOL of the modern business in that vast numbers of existing applications are written in it - while I hate writing code in it, I'm very good at writing code in it that's much better engineered than average, and good at taking existing VB6 systems and unravelling the mess. In 10 years time, it might be the only thing that pays into my retirement fund, once all the cool kids have declined to learn it. VB.NET is close to VB6 syntax but differs in lots of subtle ways which would only muddle my potentially valuable VB6 knowledge base.

Not that I use either extensively at them moment, Java is my day job.

Re:One more tip

[Anonymous Coward]

VB needed a (really) major revision because otherwise it would be impossible to run it on .NET, and maintaining a legacy runtime was expensive (and it must've been complete SHIT, I wouldn't be surprised if they had to drop it because no one wanted to touch that crap).

I wouldn't be afraid of coding in C#. MS keeps investing LOTS of effort into it, it's officially standardized, and there are alternative implementations. I like the language itself too, I think it's the ideal middle ground between Java, C++ and Python.

Re:One more tip

[wild_berry]

It's the programming environment that works for dotNET developers (developers! developers!): Visual Studio plus ReSharper and the best of the platform flows from your fingers. Sadly, it's not the same as Vim or EMACS, where the best of the platform flows from your muscle memory...

Re:One more tip

[dbIII]

C# is already .NET, there has never been any other version.

Java :)

Re:One more tip

[Anonymous Coward]

Coming from a C++ programmer who knows a good deal of STL; Visual Basic is turning complete, and crazy-powerful. It also makes it very easy for a non/low algorithm experienced programmer to write applications that get the job done but at a cost many times (e.g., x1000) worst than if they had been written well. (i.e., Use an upfront quicksort and binary search a list rather than linear search it each time.)

Don't hate the language because it allows anyone to build apps.
Hate the lack of knowledge some VB developers persist in maintaining.

Re:One more tip

[quanticle]

Er, C# is .NET. Also, the problem you cite isn't just limited to "proprietary" programming languages: look at Python 2 vs Python 3.

Re:One more tip

[Anonymous Coward]

That's the beauty of C++. What you don't know cannot hurt you.

Re:One more tip

[Anonymous Coward]

Happens in all succesfull languages, in fact. C has C99. MSVC doesn't support it. FORTRAN has Fortran 2003 and Fortran 2008. GCC only has partial support.

Also, "partial template function specialization" ? I think you're confused; there's no such thing. It's not really necessary either. Functions (including instantations of function templates) overload, and thus you can have two functions Foo(). Partial specialization is necessary for class templates, as classes or class templates can't be overloaded.

Re:One more tip

[Posting=!Working]

C# is already .NET, there has never been any other version.

Which is why I said "or whatever they call the next one." VB 6 went to VB.NET, I am currently lacking the psychic abilities to tell you what MS will call the next version.

You need to learn new things when C++ evolves, but it would still compile your old code, unless they changed a specific function that you used. VB.NET will not run a VB6 version of "Hello world," let alone anything else.

Re:One more tip

[marcosdumay]

"languages always come and go"

What are those non Microsoft languages that go away? I can think only of COBOL, can you name others?

Re:One more tip

[marcosdumay]

I'm still waiting the ISO people actualy say that that "C++ was a scam" thing wasn't really a fools day joke... Anyway, ISO confirmation of things is mostly just bureocratic assurance with no real word consequences. For they to anounce something everybody must know it first, so the one person that still don't get it, please, raise your hand, so we can proced to clear everything up.

Re:One more tip

[PCM2]

You need to learn new things when C++ evolves, but it would still compile your old code, unless they changed a specific function that you used. VB.NET will not run a VB6 version of "Hello world," let alone anything else.

Well guess what? C++/CLI (the version of C++ that runs on the .Net CLR) might not compile your old code either. I never learned VB so I can't say just how different the VB.Net syntax is, but C++/CLI introduces the concept of "handles" -- not pointers -- to objects managed by the CLR. I imagine that's a feature most .Net programmers are going to want to use heavily, and it means a lot of old code will have to be changed. But that's life when you're switching to a completely different runtime target -- a virtual machine, rather than the bare hardware. It seems silly to expect anything else.

Re:One more tip

[Anonymous Coward]

I was shocked to learn that there were entire language features I wasn't even aware of after having used the language in production code for 20 years!

That's the problem with you old farts.

Time for some pedanting the pedants

[Anonymous Coward]

Most of that stuff isn't even part of "linux", strictly speaking. No, I can't resist pointing out this nitpickery after having been annoyed to death by "gn00/linex/debjan" and all the overlong designations that crowd of painfully annoying pedants felt on slapping onto systems that worked Just Fine without their oh-so-open goo. Like, just about all modern members of the BSD family. Thanks so much for that guis. Anyway. If we're to be properly open-minded about our tools, one could argue that "linux" is just as much part of "Unix" these days as the entire at&t Unix and BSD families and perhaps even minix and a bunch of others are. Why be so centric on a single mass of code?

Re:Time for some pedanting the pedants

[kiwix]

It's not only pedantry, it's just common sense.

Suggesting that ImageMagick is part of "Linux" is quite stupid, they have Windows and MacOS binary available for download...

Re:Time for some pedanting the pedants

[danieltdp]

Maybe the article meant to say that ImageMagick would be installed by default in your Linux box

Yes, and no

[drwho]

Some oversimplified philosphy, some good hints. Programmers and SysAdmins who do a lot of resource management eventually become managers. This isn't neccessarily a bad thing, as the world needs more managers with extensive experience with that which they are managing, and the respect of those people they are managing. It's true that it's silly to adopt some software, technology or process just because it's new. But Ted seems to be resistant to any change, which is not good either. The problem with "don't fix it if it ain't broke" reasoning is..what do you do when it eventually breaks? This is a mistake made by many in process control / automation eniveronments: failure of a part which is so obsolete that it has become difficult and expensive to obtain a replacement. Just try to find a new motherboard with an ISA bus these days. Or a composite monitor. The same thing can happen with software and the OS..where are you going to find a guy who knows enough about that old Kaypro which was running some COBOL software on CP/M, which controlled the electroplating machinery? This is why companies have lifecycle management, so that the pain of switching to newer software / hardware comes with predictable cost and timetables instead of sudden, possibly prolonged unavailability and expensive, awkward, band-aid fixes.

This flows into the idea of organizational amnesia, where important processes become lost. This is perhaps best illustrated by the US DoE forgetting how to make this secret substance called FOGBANK, which is a critical component of H-bombs. Upper management felt as though, because there was no need for additional H-bombs, the process was unimportant, and didn't take into account that H-bombs become (more) dangerous with advancing age, and eventually these needed to be replaced. It took considerable time and money to re-engineer FOGBANK.

These are both examples of failure to consider that all equipment wears out, and failure to plan for long-term needs.

Anonymous Coward

[Anonymous Coward]

"The purpose of software engineering is to manage complexity, not to create it."

Bill Catambay, Pascal Programmer on Macintosh and Open VMS

Re:until you find out why Python doesn't do the jo

[iammani]

But in a few cases, Python does not do the job.

FTFY.

Sounds like good advice

[Anonymous Coward]

I've been building a website. Mostly its just a big pile of bash scripts building everything. I have one script that calls about a dozen others. It builds Apache from source, PHP from source, MySQL from source, and my content manager Typo3 from source, plus a dozen support packages all from source, patches (hardens) PHP and adds in a security module, patches Apache (modsecurity2), configures everything (all scripts and configuration files, get configured from this one main script), and also builds an ODBC driver plus builds ODBC connectors for OpenOffice, all from the one main script. With nothing else running, my CoreI7-920 runs at 800% for about 20 minutes. I have a backup script mostly using rsync to back everything up once per week. I wrote the bash scripts, but I didn't write the web server, database engine, interpreted web language or content management system (although I have heavily configured all of them). If I ever needed functionality exceeding the bounds of what the pre-built packages can do, then I would write a program (likely in C) to provide the functionality. Its just a matter of efficiency. If I can get 90% or 95% or 99% of the functionality I need at 1% of the effort (or whatever the ratio is between writing a software package yourself versus just configuring a pre-built one), I get much more done in a smaller amount of time. Likewise, its easier to modify the Apache source code, rather than building my own web server from scratch. If I really wanted to build my own web server from scratch, I suppose I would. But alas, the desire is not within me.

Re:until you find out why Python doesn't do the jo

[TheLink]

Funny thing is sometimes you wait a year or two and:
1) The C guy still hasn't finished the job either (at least not got all the annoying bugs out).
2) Intel and friends now let you use Python to do the job ;).

Not usually true (hopefully ;) ) but, a few years ago I wrote some TCP/IP servers/services in Perl and they performed OK, it wasn't my stuff that was falling apart regularly due to load, unexpected/malicious input, etc.

Thanks go out to Intel, AMD and the DRAM bunch :).

The smart phones are already more powerful than many old PC desktops still creaking on merrily...

Re:Don't code in C

[PenisLands]

You're wrong. C is more like a swiss army knife which has a chainsaw, a sword, a lightsabre, a rod of destiny, and a diamond tipped blade. Python is like a playdoh toy, and that's why it's so popular with the Ubuntu developers.

What is this?

[MacGyver2210]

I was hoping for some actual programming tips or tricks.

This was completely irrelevant to me, and aside from the NoSQL review, not even that interesting.

Don't name your stack

[NotSoHeavyD3]

q since that makes no sense. (Yes, I've seen an open source project with a stack that had the name 'q'.)

Re:Don't name your stack

[21mhz]

q since that makes no sense

Will it help if I add just one more letter?

Pipe to a subprocess in C too

[WebManWalking]

I don't know if Linux has everything Unix has, but in C on Unix, you can call popen() to pipe to a subprocess from within compiled code. That allows you to use any utility that reads from stdin or writes to stdout. That goes along the "don't do stuff that Linux already does" and Python subprocess.Popen sentiments.

Re:Pipe to a subprocess in C too

[marcosdumay]

You can use pipes on C code in Linux too, and Perl. I don't know about Python. Anyway, most of the times you'd better oppening your pipes on Bash, and dealing only with the std streams on C.

python -c "import this"

[Anonymous Coward]

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Some tips from a C guy.

[Murdoch5]

Hey

I know these might sound odd but hear me out. Start by trying to rewrite the basic library's. make your own printf, strcpy. strlen etc..... Write copies of your own link list and tree storage methods and above all really really start to understand how memory works.

Another really really important thing that I have learned is stay FAR FAR away from OO programming until your really really comfortable in lower level languages. The reason is that to many students and beginners sit there trying to figure out why there variable started with value X and ended up with value Y only to find out that there object bashed some memory earlier on.

Basically just grab a good C compiler, I mean C COMPILER, not C++, not C#, not F# and start to learn how all the functions you use on a dailiy basis work, it will give you new insight into why and how you can quickly avoid and fix problems when and before they happen. It's also important to get a really really good handle on using a CLI over a GUI, Stay away from Visual Studio and other simular compilers. Use GCC and CC and make sure you look at how LD is working and understand how compilers do optimizations and improvements to your own code. This post is taking about grabbing tool to do Image processing and preform functions that have working solutions. However taking the time to see how the solutions work and why they work will give you good insight into not only great code design but great programming methods. It might seem odd for me to suggest to a beginner to try and rewrite strcpy or strcmp, but once you see how they really work you'll be far less likely to make the simple mistakes that can ground your program / project from working. It's the same say with with a beginner figuring out how malloc works and where memory gets taken from and put to, all of these suggestions are coming from the way I learned to program in C and other languages.

Feel free to throw any of these away or take any of them into your own programming adventure, but one thing is for sure. When you can figure out how the basic functions you use every day work it will save you hours and days of trouble shooting and leave you with a greater pallet of tools to use in the class room and on the job. I welcome and one who wants to add ideas to this post and attack it with there own view points.

Re:Some tips from a C guy.

[spiffmastercow]

That's essentially what the K&R C book steps you through, and I'd say it's the best programming book ever written (or at least the best I've read). I don't get to do much in C these days, but the stuff I learned 10-15 years ago in C has made me a much better programmer. It's sad how programmers these days give you a blank stare when you ask if you passed something by value or reference.

Re:Some tips from a C guy.

[synthesizerpatel]

This is a cogent argument but to say that staying far away from OO until you're really comfortable in lower level languages is somewhat biased. C++ is a lower-level language and when programmed correctly provides most of the speed that well written C does. I completely agree with the assertion that one should know whats going on under the hood. High level virtual machine languages don't spend much if any time worrying about it beyond trying to clean up after you.

The argument about understanding your memory usage only comes from C is a bit misleading as well. In cases such as kernels or kernel drivers, C is the obvious choice when assembly isn't necessary.

But there's a reason why C++ has classes like auto_ptr, and boost has shared_ptr/smart_ptr, and why RIAA is a great programming paradigm. Granted, there's 'overhead' in using a vector versus a raw C array -- but unless you need edge-of-bare-metal speed, the benefits of C++ outweigh the dangers of being on the edge of bare metal.

I'm not a great C++ guy, but I've seen great C guys make the same off-by-one array errors time and time again that respond to C-programmer mantra of "I like to know what my code is doing" with the rhetorical core dump that says "And yet, you don't.". Not always. But enough that it's noticable.

Is spending two weeks tracking down a corrupted stack trace worth the extra handful of bytes saved by not using boost shared_ptr's? In some cases, yes. In most cases, probably not.

When you consider that the other end of this spectrum is PHP and Ruby programmers who don't understand threads or forking or.. really anything beyond their templates -- the "bloat" that C programmers accuse C++ of bringing seems laughably trivial.

I will admit, C++ is ugly as sin, but inheritence and code-reuse make code so much easier.. C is the most beautiful to program if you have the time to do it right, no arguments.. The teeth of the gears are interlocked with no gaps when you do it right. But there's so much opportunity for graceful design with OO that strict C will never enjoy.

Re:Some tips from a C guy.

[azmodean 1]

I'd understand your response if the GP had said, "stay away from object-oriented languages FOREVER", but he didn't, he said, "stay away just until you are comfortable with lower-level languages".

The point isn't that C++ and similar languages are BAD, but that they hide things from you. When you are learning to program, that is BAD, but when you are writing production code, you will generally want to eliminate points of failure, and a great way to do that is to use a higher-level language that hides things like memory management from you.

One-liner version: C is dangerous to program in, even for great C programmers, but great C programmers make better programmers no matter what language they use, because they understand how the underlying system works.

Re:Some tips from a C guy.

[phantomfive]

It might seem odd for me to suggest to a beginner to try and rewrite strcpy or strcmp,

If that is really true, that's the saddest thing I've read all day

Re:Some tips from a C guy.

[fyngyrz]

But there's so much opportunity for graceful design with OO that strict C will never enjoy.

Just thought I'd throw this in. I write some pretty high performance code. I write in C. Not C++, but C. I use a *lot* of OO techniques. And I do that by writing in those capabilities from the bottom up. I have also, over the course of some four decades of programming, come up with my own libraries to do the things I've needed to do.

The benefits: I know exactly how and why everything works. Everything. List handling, memory management, local and global functions, constructors, destructors, objects, etc. If a bug is found, I can fix it, because it'll be in my code. My final executable is also typically a *lot* smaller, and a *lot* faster, and a *lot* better at memory management, than anything out there designed to do similar jobs. And my stuff is extremely portable.

The downside: It took me decades to get where I am.

In any case, I've simply not found a need to use C++, or C[whatever]. C is *very* powerful.

When I want a scripting language, I prefer Python. Between the two... there is very little I cannot accomplish. HTML, CSS, XML, SQL... these have their places too, but inevitably, I handle them, as much as I find they need handling, with C or Python, the choice generally being made depending on the ultimate performance required.

Re:Some tips from a C guy.

[Murdoch5]

Hey Well I hear what your saying about using C++, on a personal note I stay clear of all OO programming. I'm a bare metal guy, I have never been in a situation that an object couldn't be programed with strict data types. C++ is great when you want to program a sloppy rough function to add objects or to overload the signs + and -, but if you can stick with C and other lower level languages.

As for the off by 1 error, that truely means you don't understand arrays and pointers.

Re:Some tips from a C guy.

[ADRA]

"It's sad how programmers these days give you a blank stare when you ask if you passed something by value or reference."
Isn't that what progress is all about? The only way to get more productive in programming is to peel away at the low level complexity and boiler plates in the language / technology. Sure its great have an educational foundation of why things are they way they are, but when it comes down to it, but not every programmer studies comp sci. at an ivy league institution and we all have to learn to live with it.

Re:Some tips from a C guy.

[Murdoch5]

Take away the complexity? Thats what OO programming is about and that is how so many programmers can't handle real memory, they can only relate that there object can handle there memory.

Re:Some tips from a C guy.

[synthesizerpatel]

As for the off by 1 error, that truely means you don't understand arrays and pointers.

If you use a coding style that forces you to be aware at all times what you're doing -- you need to be aware at all times.

To put it in obligatory car analogy form: I think everyone with experience who knows how to drive a manual clutch 'understands' how it works, but that doesn't mean you'll never grind a gear by accident.

Re:Some tips from a C guy.

[spiffmastercow]

Unless you're programming in a purely functional language like Haskell, pass by value/reference is something that will be important to you on a daily basis. If you don't understand the difference, you're going to get unexpected behavior qnd not know why. And I sincerely hope that's not something o ly taught at the top schools these days.

Re:Some tips from a C guy.

[Anonymous Coward]

I object to your use of "there".

Re:Some tips from a C guy.

[synthesizerpatel]

If a bug is found, I can fix it, because it'll be in my code.

Thats something I've run into enough with C guys that it makes me shudder. I know you can find a bug in your code. But can anyone else? In a perfect world of software development any team member can thumb through source code (which will be pleasantly documented) and track down a problem.

In the real world without clear oversight, coders inflect their own personal style and way of doing things and while they may conform to a general code-style, only do-so on the highest API level without thinking much for 'the next guy' who has to maintain their code. I'm sure you're not that type of guy, but the hubris of 'I know how to do it right' strikes fear in

Also, while you may not be a fan of C++.. if you ever have to interface Python and C++, I urge you to look at boost::python.. It makes wrapping C++ libraries so effortlessly easy and fun it shouldn't be legal. After having written my own pyrex, ctypes and Python C-API wrappers I'd never do it any other way.

Re:Some tips from a C guy.

[turbidostato]

"I have also, over the course of some four decades of programming, come up with my own libraries to do the things I've needed to do.
  The benefits: I know exactly how and why everything works.
[...]
The downside: It took me decades to get where I am."

It is not the only downside. The biggest downside is that even if you know in and out how your libraries work, you are the only one.

Re:Some tips from a C guy.

[Murdoch5]

Agreed but making that error once in blue moon is very different from making it once every hour, which I know many C programmers who do. How ever there really is such thing as off by 1, infact C is actually correct in it's implementation. C is modeled after Assembler and because of that how C handles memory is "more correct" then how other languages handle it.

Take a C++ programmer and put him in assembler and laugh, put a bad C programmer in assembler and feel the need to smack him, put a good C programmer in assembler and notice how there is very little effort in the switch.

Anyway, I agree with your arguments, don't get me wrong, I think your points are good and I tend to agree with them, but C doesn't have an off by 1 issue, it's that other languages have a poor implementation of indexing.

Re:Some tips from a C guy.

[Anonymous Coward]

The thing is, you're advocating learning the details of an extremely small subset of things (C, gcc), and letting that form you're understanding of programming. But once all that's ingrained, and you move onto another platform/environment, all that changes, and perhaps even becomes a hindrance (old habits die hard :) ), especially when working nowadays with all this hypervisorvirtualizedistributedcloud 2.0 technology where you rarely have direct access to your own bare metal.

While I do agree with the need to know what the hell is happening at a low level, I think it can be taught more theoretically, as in what SHOULD be happening, and that can be then applied to whatever compiler/language/environment is being used. The concept of a pointer is the same whether it's in Java or C - memory is such a fundamental part of CS in general, that I don't think it should be tied specifically to C.

Re:Some tips from a C guy.

[Khelder]

If you have infinite time or are really interested in how these things work, I think it's unnecessary. Sure, some programmers do really need to know exactly how malloc and free work, or how string functions are implemented. But the vast majority do not. What you need to know about details like that depend massively on the type of programming you do.

Re:Some tips from a C guy.

[Khelder]

K&R is a great book. If you want to learn C, you need look no further.

However, I wouldn't say that if someone doesn't know what pass by value vs. reference is that the solution is to learn C. The solution is to learn more about programming language theory. Then you can apply it to all languages you learn.

Re:Some tips from a C guy.

[eddy the lip]

I make my money these days writing (god help me) PHP, and my entire professional career has been web development, but what I learned from grabbing a linux distro and working through K&R was such a fundamental experience for me, it's hard to overstate it. As the GP alluded to, learning how the tools you work with were made makes you more effective at using those tools.

I moved recently, and had to get rid of my books. I kept eight. That included the Bible, Ulysses and K&R.

Re:Some tips from a C guy.

[quanticle]

Hear hear. I think that a large part of the reason that programmers don't have any idea whether something is passed by value or reference is that these days, all the new programming languages pass by reference. If you're programming in Java, C#, Python, or Perl, the number of times a user defined type will be passed by value can usually be counted on one hand.

Re:Some tips from a C guy.

[CrazyJim1]

Another really really important thing that I have learned is stay FAR FAR away from OO programming until your really really comfortable in lower level languages.

I also say,"When at all possible, don't do something with pointers that you can do with an array even if you lose some unnoticeable performance"
Pointers in C++ are good for shooting your leg off if you don't know exactly what you're doing.
The things I typically use pointers for are,"Merge Sort and GUIs(which OO was designed for)"

From staying as much on the array side of things as possible, I have much more reliable code. If you use a mix of both arrays and pointers, repeatable code is less common too. And the bugs that pop up are of a wider variety than if you're just arrays as your memory structure.

Re:Some tips from a C guy.

[CrazyJim1]

An analogy would be like playing WOW as a mage.

The Frost Tree is arrays. The Fire Tree is pointer based memory management.

Theres times in battle where you may want a little frost for slowing, and a little fire for extra damage, but for the most part you want to focus on one or the other. With enough training, you can be proficient in both, but you can only use Frost or Fire one at a time so its good to specialize.

I'll say what I said in my above post again: If you stick with arrays instead of mixing arrays and pointers, your bugs will be easier to debug. Also your code will be more reusable. I think if there is anything a coder appreciates it is less bugs, easier to fix bugs, reusable code, and your boss attacks frost shield is nice too.

Re:Some tips from a C guy.

[Anonymous Coward]

Surely this can be fixed with a good IDE. Coding in Java enables me to attach the source to my IntelliJ IDEA project and debug through each step to determine why my code fails. This has given me a much greater understanding of how the Java libraries and 3rd party Open Source libraries work in my domain.

Re:Some tips from a C guy.

[wvmarle]

I'm happy knowing my way around Python, and doing OO programming to boot. I don't know C or any of it's relatives, wouldn't know about memory management, and honestly couldn't care less.

Now when I would start making a career out of programming large projects where performance is of utmost importance... or seriously programming embedded devices... then I may start to follow your approach. As I do agree that a compiled language like C is the way to go for those situations.

Re:Some tips from a C guy.

[fyngyrz]

Why do you think so? My libs are very well documented and designed to be both easy to use and strongly standalone ... Are you saying that only programmers who have prior knowledge of standard libs are effective? Because that's pretty silly. From my pov, if you can't wrap your head around any one of my lib functions in five minutes, you're no more than a script kiddie. That's the point of a lib - to make common tasks easy.

Re:Some tips from a C guy.

[fyngyrz]

Thanks for the tip.

Re:Some tips from a C guy.

[marcosdumay]

Hum... I've seen people that can't fix code in Java because they were passing some parameters by value, but expected it to be by reference. They didn't even had any idea of what the difference was. It is way more common than you said it was, and not knowing it will come later to bite you.

Also, as far as I know, Perl functions don't have parameters the same way the other languages do. And what resembles paramaters there could be accessed in a way that resembles being passed by reference or in a way that resembles being passed by value. Both ways have some problems you'll have to deal with, and there is no automatic better way (altouh there will probably be one that fits your code better).

Re:Some tips from a C guy.

[rinka]

I'd say it's the best programming book ever written (or at least the best I've read).

Some books that went a very long way for me: Code Complete http://en.wikipedia.org/wiki/Code_Complete Programming pearls: http://www.cs.bell-labs.com/cm/cs/pearls/ I won't add Knuth here as that was (kind of) a textbook but...

Re:Some tips from a C guy.

[Anonymous Coward]

The reason is that to many students and beginners sit there trying to figure out why there variable started with value X and ended up with value Y only to find out that there object bashed some memory earlier on.

If your object is bashing memory somewhere, then you haven't learned about encapsulation.

Ok, I will bite

[Zangief]

How does xargs replace hadoop, even in trivial examples?

Unless you are talking about running hadoop in a single machine, which is just waaay too trivial an example.

Re:Ok, I will bite

[codepunk]

I had the same thought myself since a hadoop cluster running a map reduce job has a very specific problem space that it solves. The processing of a singular large dataset in a short period of time across a large number of nodes. It is a simple matter to fire up a processing job across say one thousand nodes in parallel. The problem hadoop solves is the distribution of the data as well as accumulation of those results in a efficient manner.

Re:Ok, I will bite

[Urkki]

How does xargs replace hadoop, even in trivial examples?

Unless you are talking about running hadoop in a single machine, which is just waaay too trivial an example.

I think it was meant the other way around. You can (probably) replace xargs with hadoop, ie. just use hadoop if you don't know about xargs, but the result will probably be a minor WTF.

Don't program!

[Anonymous Coward]

I have a shock collar around my neck hooked to my computer, it activates if I attempt to use any IDE, compiler, or script interpreter.

Re:Don't code in C

[topham]

No. Sorry, I strongly disagree.

C is like a swiss army knife that can be used to assemble a chainsaw, a sword a lightsaber, etc. You'll have to carve out all the pieces, file them down and put them together yourself, but the swiss army knife will help you a long the way.

But don't forget the bandaids.

Oh if you find yourself repeating some code

[NotSoHeavyD3]

Consider making it function, especially when you've repeated the same 8 damn lines over 70 times. (Yes, I've seen that one too. Yes it bit the guy writing it in the ass and no it wasn't me, I had to code review it.)

Re:Oh if you find yourself repeating some code

[Haeleth]

Sadly, if you're working in Java then you will frequently be forced to write the same 8 damn lines over 70 times, and there will be literally no way to avoid it unless you resort to automatic code generation, which is a fancy name for using a separate program to do the copying-and-pasting for you.

Re:Oh if you find yourself repeating some code

[daseinw]

Sadly, if you're working in Java then you will frequently be forced to write the same 8 damn lines over 70 times, and there will be literally no way to avoid it unless you resort to automatic code generation, which is a fancy name for using a separate program to do the copying-and-pasting for you.

Color me confused. I'm genuinely not understanding you here. If a team member told me he was writing the same 8 lines over 70 times in Java, it would probably a hint to me that he needed to brush up on his understanding of basic object oriented programming. If the code in question is a cohesive unit, there's no reason you can't put it in a class, and re-use that class for all eternity. What did I miss?

Re:Oh if you find yourself repeating some code

[ADRA]

I assume you're referring to hashCode/equals since that can be the most repetitive piece of code there is in the language. If you hate it so much, why not just write static methods to encapsulate the code? Performance will suffer, but it means you won't have to write as much boiler plate. Still not good enough? You could always have pre-compilation macros and suffer with a world without good static analysis. Personally, I'd rather have a little bit more verbiage than the alternatives.

Re:Oh if you find yourself repeating some code

[selven]

The code in question (9 lines actually) is:

private int variable;
public void setVariable (int val)
{
  variable = val;
}
public int getVariable()
{
  return variable;
}

Ironically, making your code less object oriented (ie. screw encapsulation) fixes this problem.

Re:Oh if you find yourself repeating some code

[pjt33]

That isn't encapsulation. It's just wrapping some sugar round a non-encapsulated design.

Re:Oh if you find yourself repeating some code

[gangien]

i always wished java had a getter/setter keyword.
private get set int value;

or something similiar. maybe annona

Re:Oh if you find yourself repeating some code

[selven]

Then what is real encapsulation, and what are its benefits? Not very experienced programmer here, honestly wondering.

Re:Oh if you find yourself repeating some code

[Fallingcow]

I can't think of a reason that one would expose a "set" method that just sets a private variable to the passed value.

If you're going to do that, simply expose the variable itself.

If you use a "set" method, it should be because you need to do something to the incoming data--sanitize it, convert it to a different unit, change some other value based on the fact that this one has been modified, whatever--before assigning the variable that value.

In other words, if you're using a "set" method you should need the full method declaration.

Re:Oh if you find yourself repeating some code

[gangien]

well think harder. :P

it provides a consistent API to use across objects. Am I allowed to set a field or not? It also has the nice benefit with intilisense i can type ".set" and see all that I can change. And if there is sanitizing/converting that needs to be, it's transparent. So i'm not looking for .value i just look for .setValue. It also provides some control for who can read/write the field. You could make the setter protected and getter public.

probably other reasons. IDK, i don't care much either way. but it's certainly nice when i'm dealing with 10 different objects i'm unfamiliar with, that I can use certain prefixes(set,get,add,is) and probably get pretty close to the desired outcome.

Re:Oh if you find yourself repeating some code

[DragonWriter]

I can't think of a reason that one would expose a "set" method that just sets a private variable to the passed value.

It future-proofs the API a bit (this may be overengineering, but its a common enough evolution that assignment would develop side effects in the future that it may be worth considering.)

Also, while this isn't what the GP was doing (since it also had a getter) its the only way to create a write-only attribute, which is sometimes useful (since if you expose the variable, its a read/write attribute.)

Re:Oh if you find yourself repeating some code

[pjt33]

This and this do a reasonable job of explaining. The key phrases respectively, and they express the same thing, are

Don't ask for the information you need to do the work; ask the object that has the information to do the work for you.

and

Tell, Don't Ask

Re:Oh if you find yourself repeating some code

[marcosdumay]

No, that is making your code compatible with the available libraries. That 70*9 copied lines of code will make it possible to reduce the overall size of your program by some hundreds of thousands of lines.

But they are still some hellish 70*9 copy-and-pasted lines.

Re:until you find out why Python doesn't do the jo

[daveime]

Look, the PHB usually wants the code to run on HIS server for HIS use only. That's what he pays you for. Not to code it in the most cross-platform friendly language-du-jour and take 2 years to iron out all the bugs.

He doesn't give a shit if it'll run on the Xbox 360 or a Linux-ready Dead Badger.

And neither should you, unless you are some kind of anal retentive who spends all day arguing the merits of absolute versus relative positioning, fixed vs percentage tables, and worrying whether your code will run on every machine conceived in the next 50 years.

I have news for you, it won't.

Hell, I got an N900 6 months ago, and it's already EOL'd as far as updates to the OS are concerned.

I suppose I could wait till there's a port of Android or something, because the three coders on the project are doing a fine job, in less than a year I'll have the same functionality as a 3210.

So whats the argument against PIL?

[synthesizerpatel]

I read his page and the comments here and I can't seem to find any arguments on this?

Comment your data too!

[LihTox]

I'm in a different boat from most commenters here, I think, because I am a scientist writing simulations; some simluations run a long time and create a lot of data which would be costly to reproduce, and what I wish someone had told me early on was that I should comment my *data files*, not just my code. Each file should include the exact parameters used to create it, an explanation of what each column represents, and preferably there should be a way of knowing what version of your simulation code was used to create it. A couple of times in grad school I had toss out months of data after I discovered a bug in my code, and didn't know when the bug showed up and which data was affected by it.

(I'd welcome other advice from simulationists too; I've never had an advisor who was particularly programming-savvy, even though programming was always a large part of my research, and so I always had to make it up as I went along.)

Re:Comment your data too!

[dch24]

I had the same experience. As the data collection evolved, I used the revision number (from the source tree) compiled into the code and embedded in each data file.

The boss wanted everything in XML, since that was extensible, but then went halfway because raw images don't encode well in XML. So we maintained the dataset in XML and binary.

But as a result, I was able to keep around all versions of the binary-to-xml converter in the current code base. With some unit tests, and some comments, it really helped explain ancient data.

I enjoyed reading your comment. Thanks.

Re:Comment your data too!

[SplashMyBandit]

Yep. I learnt that lesson too when recording astronomical observations for research. You must design your software to *automatically* put in *all* the relevant information - just in case you need it. Rely on a human to put it in and you get mistakes that take a huge amount of time to untangle (if at all possible, sometimes you just don't have enough information to deconflict bad human-recorded values). My general rule is, "Never lose information". Once information has been thrown away it becomes a huge effort to try reconstruct it. Better to always keep as much as you can around to begin with. This is where using database to record 'state' stuff makes sense, and then have that point to the data files that are too big to go in the database. Then you can slice-and-dice the database tables as you need to. As a researcher I avoided writing software to existing databases (I now love Postgresql) but I wished I'd used Postgresql a lot earlier.

Re:Comment your data too!

[pz]

I'm in a different boat from most commenters here, I think, because I am a scientist writing simulations; some simluations run a long time and create a lot of data which would be costly to reproduce, and what I wish someone had told me early on was that I should comment my *data files*, not just my code. Each file should include the exact parameters used to create it, an explanation of what each column represents, and preferably there should be a way of knowing what version of your simulation code was used to create it. A couple of times in grad school I had toss out months of data after I discovered a bug in my code, and didn't know when the bug showed up and which data was affected by it.

(I'd welcome other advice from simulationists too; I've never had an advisor who was particularly programming-savvy, even though programming was always a large part of my research, and so I always had to make it up as I went along.)

Yes, yes, yes. This is why when I collect data (I'm an experimentalist), I save a COMPLETE copy of the code used to run the experiment along with each day's data. Hard drives (and CD/DVDs) are cheap compared to the potential time loss from not knowing exactly which data sets are subject to which bugs you will (not might, but *will*) find in your code.

I'd go one farther: unless space is a serious constraint, store your data files in ASCII. Including and especially the associated collection parameters. Everything in my lab gets stored as ASCII except a handful of data streams that would be excessively large if not stored in binary. I use the Windows world INI-style format, since there are many libraries available for parsing it (and it's easy to write your own, too), and it's very easy to read in an editor.

Re:Comment your data too!

[owlstead]

Yes, and if you use units (and generally you do) then make it clear what units each parameter expects. I am in crypto and I am always guessing if things are in bits or bytes. In physics it is even more important - fortunately students of physics will probably be more inclined to describe the units they expect. Also, the names of the parameters are even more important than those in the source files. Basically, if you have e.g. a configuration file, it should be thought of as part of the user interface.

Re:Comment your data too!

[B1ackDragon]

Very much agreed, though I'm not a simulationist per se anymore (these days I just do all kinds of crazy things to large data sets, and need to remember what crazy things I did, and when, and as you said, what the output means!)

Here's another tip, which I also thought about last time someone asked slashdot about scientific data organization: keep a wiki, and write down all the things you do (at least everything that isn't trivial to reproduce) there. Commands, paramaters used, input and output files created, etc. I organize chronologically. Having a digital "lab notebook" can be invaluable, and it makes the problem of organizing things much easier, since everything important is indexed in the wiki and can be looked up based on the timeframe of the project.

Re:Comment your data too!

[Anonymous Coward]

I am also a scientist, and had the same problem with uncommented data. My rather inelegant, brute force, approach was to set up a repository and add a line to every one of my scripts that auto-commits all scripts and parameter files. Then, I set up bash to log every command with a time stamp. So, to trace back a file, I can grep my bash log, get the time stamp, and recreate the exact status of the code. This is rather orthogonal to the main topic, but I hope it helps someone.

Re:Comment your data too!

[turbidostato]

"Yes, yes, yes. This is why when I collect data (I'm an experimentalist), I save a COMPLETE copy of the code used to run the experiment along with each day's data."

Do yourself a favour: use any SCM tool (Subversion, Git, heck even CVS) to manage your source code and make your program put its release version (and other interesting data, like parameters used, date, etc.) within the data files. It will not only save you space but will easy management and will bring to you the ability to easily find why the heck data from release 127 doesn't look like data from release 128.

"I'd go one farther: unless space is a serious constraint, store your data files in ASCII."

What's the problem with ASCII output and then compressing the files?

Re:Comment your data too!

[Anonymous Coward]

You're right. I never really gave it much thought until reading your reply, but I've been doing that more and more. Practically all the code I write to change data formats, do calculations, etc. has a standard "comment" indicator (same as UNIX shells: "#" as the first character of the line), and all the data-reading routines skip over or optionally echo those comments to STDERR (verbose mode) as the data is processed. That means I can chuck all sorts of comments in there and nothing gets messed up in the rest of the data. The first few lines of the file ordinarily have the date, source of the file, labels for the columns, units, problems/limitations etc., and those get preserved and sometimes augmented by more information from programs as the data gets passed down the processing pipeline (e.g., like program arguments). It helps a lot.

It is sloppier than having a proper data format where that sort of metadata is encoded in separate fields (e.g., netCDF files), but it's also very simple and much better than unlabeled, flat ASCII text files full of cryptic numbers. I've also pretty much standardized on tab as column delimiter and newline as record delimiter, letting me easily use standard UNIX tools like head, tail, cut, paste, sort, grep, etc. -- why reinvent the wheel for such simple operations? It's not a real database, of course, but it's amazing how versatile and quick these tools can be for batch processing massive quantities of data.

Re:Comment your data too!

[highacnumber]

That is too true man. I have recently been in a world of pain from not properly documenting output from simulations.

Re:Comment your data too!

[quanticle]

some simluations run a long time and create a lot of data which would be costly to reproduce, and what I wish someone had told me early on was that I should comment my *data files*, not just my code. Each file should include the exact parameters used to create it, an explanation of what each column represents, and preferably there should be a way of knowing what version of your simulation code was used to create it.

Well done. I've worked on some simulation code too, and I can say that its an absolute bear to try to reproduce a simulation when you don't know what the original parameters were. Sometimes you can try to guess the parameters from the data, but when you do, you're never sure whether your guess was correct, or whether your implementation of the model is subtly different and happened to produce the same output for different input parameters.

Having said that, I don't think commenting data is as big of an issue in the business side of things, simply because business data tends to be stored in databases, rather than files. Databases have a lot more "natural" documentation around them, since their columns are named (the names may not make sense, but at least they exist), and the relationships between data tables are captured with foreign keys.

Re:Comment your data too!

[GWBasic]

(I'd welcome other advice from simulationists too; I've never had an advisor who was particularly programming-savvy, even though programming was always a large part of my research, and so I always had to make it up as I went along.)

One of my tasks in my first job out of school was to load an XML definition of long-running chemistry experiments. These experiments, I learned, could run for many years. Needless to say, I wrote the strictest XML loader I could. It drove a few people nuts, but then they thanked me after I saved them months of debugging time.

Today, now that storage is cheap, the best advice is to never throw any data away. If you use an inefficent schema; so be it, you can always improve the schema later. But, if you never collected the data, then it's gone forever. You might want to consider some form of a database that allows for a flexible schema, like Mongo, because it can easily adapt to minute changes that occur as your program and research evolve.

Re:Comment your data too!

[Anonymous Coward]

I, too, write scientific simulations and data analysis software that generate gobs of data. It's very common during development (and when does development ever end?) to produce a lot of data files from different versions of the software. Long ago I adopted the following practice: A simulation writes multiple data files having different formats. One data file is essentially metadata: it contains all the physical, numerical, and software parameter values; the input script(s) used in the simulation; and software version identification. At times I have contemplated (and I think it really is the right thing to do) simply saving all the source code to this metadata file, too. Why not? The simulation data files are many GB big, so why not save O(1 MB) of program text? That way one has absolutely everything necessary to know what went into the simulation.

Of course there are other (complementary) approaches. For example, netcdf and similar APIs generate self-documenting data files (provided one uses the API as it is intended).

Re:Comment your data too!

[Anonymous Coward]

I'd just like to say thanks - I'm a starting PhD student, with a not-programming-savvy advisor, and that sounds like really good advice.

Re:Comment your data too!

[Anonymous Coward]

I'd welcome other advice from simulationists too; I've never had an advisor who was particularly programming-savvy, even though programming was always a large part of my research, and so I always had to make it up as I went along.

Use subversion/CVS/git/whatever. Do so *frequently* and not every 10 months.

Keep the manual updated - too often people have left our group rather suddenly without really knowing beforehand, and the only documentation left was in the code.

Use LAPACK/BLAS wherever possible. 20 years from now these libraries will still be around and *much* faster than your own code. And they will still compile (not true for some of the code I am working with - needs special old compilers because of some "clever" hacks by people long gone)

Do not only write "programming" comments but also describe what the physical background/model is. Provide citations of the relevant articles.

If you produce lots of data then make use of unformatted files (or whatever C calls them). Saves terabytes in the long run, and converting them to ASCII when necessary is trivial.

Do sanity checks wherever possible.

Do integrity checks wherever possible.

Write meaningful error messages. They *must* include where the error happened (subroutine e.g.), what type of error and *expected behaviour*

Provide some sort of debug mode where people can step through the program and review intermediate results (*You* can use a debugger. The next guy maybe not).

Make sure that you/users can include comments in the input files.

Save early, save often. That means write the program as modular as possible. If possible, split into subprograms chained by shell scripts and write the intermediate output to disk.

That's all I can think of for know...

Re:Comment your data too!

[pz]

"Yes, yes, yes. This is why when I collect data (I'm an experimentalist), I save a COMPLETE copy of the code used to run the experiment along with each day's data."

Do yourself a favour: use any SCM tool (Subversion, Git, heck even CVS) to manage your source code and make your program put its release version (and other interesting data, like parameters used, date, etc.) within the data files. It will not only save you space but will easy management and will bring to you the ability to easily find why the heck data from release 127 doesn't look like data from release 128.

Using a source control system is orthogonal to keeping a copy of the source code with the data. We use CVS. We also copy the entire source tree (it's under 50 MB with all of the associated image files ... each day's data is about 1 GB so 50 MB doesn't matter) and archive it. We don't use CVS to answer questions about which data set is subject to which bug, but that's really only because I hadn't thought of it, thanks for the good idea!

Keeping the source code in a complete copy means we can change source control systems and not have to worry about whether we'll still be able to track the code and data. Since we aim to keep data for decades, that's important.

"I'd go one farther: unless space is a serious constraint, store your data files in ASCII."

What's the problem with ASCII output and then compressing the files?

The problem is that you are then dependent on the uncompressing algorithm being available, and if you don't have complete specs on the compression algorithm (who actually does?), the data are lost. Uncompressed ASCII is universal. I've got data files from almost 20 years ago that are perfectly readable and since they're verbose and self-documenting (ie, variable, value pairs) it's easy to create a new parser if the one I currently use is lost, becomes unavailable, etc. And don't think that data files from that long ago aren't important, as just last week, I received a request for a copy from another researcher for whom the data are relevant.

Re:Comment your data too!

[LihTox]

When I was a grad student, my advisor (who was not really a computer person) wanted me to do hydrodynamic simulations for my dissertation. Now, there has been 50 years of research in how to model fluids on a computer, and there are many pitfalls, but I didn't know that and neither did he, so we started from scratch: basic forward-Euler equations, simplest thing you could think of. And the result was a mess: mass would disappear, velocities would increase exponentially at times, and the data was untrustworthy. It was only by talking to other professors, who had more experience with simulation, that I learned about Runge-Kutta and other things which improved my code at least a little bit, and managed to graduate.

The moral of the story: don't rely entirely on your advisor. Talk to other people in your department and outside your department (Comp. Sci. or Math, for instance) about what you're doing and get their advice. And if the simulations you're doing are nontrivial, learn how to write those simulations: take classes, read papers, have people look over your code from time to time. But don't try to figure it out as you go, lest you end up with months or years of worthless data and wasted time.

Re:Comment your data too!

[turbidostato]

"Using a source control system is orthogonal to keeping a copy of the source code with the data."

Yes, in that one thing doesn't disturb the other. No in that you make a copy of your source code as a means to manage your source code history. Obviously that's a thing for your source code management tool.

"Keeping the source code in a complete copy means we can change source control systems"

You always can either keep along your older SCM or migrate from the old to the new. Or at least you can do it if you use open source tools.

"and not have to worry about whether we'll still be able to track the code and data."

Not because of that but because now you have a "backdoor" to access your code history. Given sources are about 5% of overall data it seems a reasonable decision.

"The problem is that you are then dependent on the uncompressing algorithm being available, and if you don't have complete specs on the compression algorithm (who actually does?), the data are lost."

That, and your mention about "windows-like ini files", makes me think you are a bit "fooled" by proprietary software vendors. Compress/uncompress algorithms are published along with its current implementation. It's just a matter or choicing the proper tools (I don't expect gzip going anywhere in the next decades and even if it went away, both the algorithm and sources are published). Given what you say I wouldn't be surprised if your daily data sets came from 1GB to 100MB (a tenfold reduction) by using tar/gzip/bzip2.

"Uncompressed ASCII is universal."

Big endian or little endian? It's not as universal as you may think.

"And don't think that data files from that long ago aren't important"

I'd never make such a mistake: I was in that bussiness in a past life. But that's another strong reason to strictly stay with open source tools.

Re:Comment your data too!

[Crispy Critters]

Version control! If I know what date I created a certain data set or performed a bit of analysis, I know how to recreate it (even if I know it has an error).

One of the hardest things is to figure out which bits of programming will actually aid in your work and which are needless complications. Will making my code super modular or OOP or whatever let me build new tools faster or will it make any new development too complicated?

Re:Comment your data too!

[LihTox]

How do you manage version control? Do you use a program for it, or do you do it by hand? And are you working alone, or in a group? (Most systems are geared for multiple users and some check-in server, as I understand it, which would be severe overkill for me.)

Re:Comment your data too!

[nica]

After working in labs in academic and corporate environments for a few years and coming to terms with the limits of my memory, I advise people to keep a lab notebook mindset. First off, I recommend lab notebooks. Log what you are working on at a high level day to day. It gets you in the habit of writing about what you are doing. When you save large data files it begins to feel natural to add a little text file saying what the data is, how you made the data, and other comments. Sometimes it is not practical to have all this information saved automatically by the test software itself because you may have a comment like, "Due to time constraints I didn't let the system warm up the standard 60minutes before gathering data, but starting the testing immediately. This will likely add 2-5% error according to the equipment manual."

Keeping notes on paper, computer, "in the cloud" all works, as long as you actually make it a habit. It's not old-fashioned. It's not inefficient. Don't let people rush you into skimping on taking notes.

Re:Comment your data too!

[tomtomtom777]

"Uncompressed ASCII is universal."

Big endian or little endian? It's not as universal as you may think.

Endianness is irrelevant for ASCII. It only determines the order of bytes within multi-byte words. ASCII is 7-bit and therefore, there is no difference between "big endian ASCII" or "little endian ASCII".

"Easy" solutions vs time

[gmuslera]

Sometimes you have a command, an app, something, that makes things trivial or are already installed and are easy to use instead of installing a complex heavy app to do that. With a good amount of pipes and installed by default command line software you can do complex processing in a lot of data, or do a somewhat trivial python/perl script for that, But sometimes you don't know exactly what, or learning how to do it would take more time than the "non optimal" way. The priority is to solve problems, if takes too long to learn how to do it in the "right" way you first must solve it. And then learn how to do it right for the next time

Emotional Things I Wish I Knew Earlier

[daseinw]

we could not figure out whether the author was an incredibly elaborate troll or just a run-of-the-mill idiot.

Reading this comment of his reminds me of something I read recently:
Physicists stand on each other's shoulders. Engineers dig each other's graves.

I've never understood why so many software developers feel the need to disparage one another in an attempt to prove their intelligence/superiority. There are plenty of tough problems out there and we all can learn something from one another, no? I've definitely been guilty of this in my tech career but lately I'm wondering more and more, why does the person who has a different solution always have to be an "idiot?" Why isn't he/she just someone who has a different take on solving this particular problem?

Now, I'm not saying that engineers do this more than any other group but out of all of my friends (some of whom are doctors, lawyers, teachers, etc.) it certainly seems like a more common event among software developers.

Re:Emotional Things I Wish I Knew Earlier

[azmodean 1]

I think there are several factors that contribute to this:
1. Programming is a very popular and easy to enter field.
2. It's actually pretty easy to get by as a programmer without really understanding what you are doing.
3. Regardless of how much you hear about it, modularity, reusability, and highly structured programming do not have good penetration in Software Engineering.
4. Because of #3, it is all to easy for otherwise competent programmers to paint themselves into a corner and generate software with really messy architecture and/or implementation.
5. Programmers OFTEN have to clean up after other programmers.

So, due to #1 and #2, there actually are quite a number of really bad programmers running around.
Due to #3 and #4, there are quite a number of otherwise decent programmers who produce working but unmaintainable code.
Due to #5, most programmers have ample opportunity to experience a great deal of pain from other programmer's incompetence.
Due to human nature, programmers tend to assume that all that bad code comes from #1 and #2 rather than #3 and #4.

And more speculatively and unrelated to the above:
6. Lots of programmers tend to hang out on Usenet, internet fora, mailing lists, and IRC, where harsh criticism is de rigeur, and internalize the habit of harsh criticism in their professional lives.

Re:Emotional Things I Wish I Knew Earlier

[Astatine]

Because of the forced collaboration. Doctors, lawyers and teachers are often producing individual work, or working in very small groups. In software particularly, the individual work of one engineer is irrelevant unless it integrates successfully with the work of dozens or sometimes hundreds of other engineers.

When the success of your own piece of work depends, in an immediate and pressing manner, on the success of the pieces of work of many other people, frustration and bitching quickly ensue if one of those other pieces of work is causing problems in yours!

Re:Emotional Things I Wish I Knew Earlier

[laura42]

I agree with all of this, but also... Doctors, lawyers, and most (non-software) engineers have reasons not to present their work as being radically different from others in the same profession. Doctors who are sued for malpractice generally want to argue that what they did is the same as what any other doctor would do, lawyers want to argue that their positions are based on precedent, and civil engineers want to convince people that their suspension bridge designs are based on the same principles that make other suspension bridges safe.

Re:Emotional Things I Wish I Knew Earlier

[vitaflo]

Doctors, lawyers and teachers don't build things. Engineers do and often have to pick up the past work of others. Ask a carpenter, plumber or electrician what they think of other people's past work. You'll find similar calls of idiocy.

Re:Emotional Things I Wish I Knew Earlier

[ADRA]

Subjectivity. With Physics, its hard to be subjective about things that are usually proven factual, but its stupidly easy to make subjective opinions about software. It gets even more complicated when we have many facets of value that we put upon software. Who's right when one person only cares about performance, another only loves beautiful code, and another makes everything as modular as possible, etc.. There are too many valid differing opinions over what is 'right' and that is why we have disagreements.

Re:Emotional Things I Wish I Knew Earlier

[Anonymous Coward]

What both OPs of this thread do not understand is that 'programmers' are not engineers.

And engineers do typically tend to stand on the shoulders of their predecessors - when not beaten up by the company lawyer or marketing dweeb.

Re:Emotional Things I Wish I Knew Earlier

[Anonymous Coward]

Here's an anecdote in support of this observation: Some years ago I was an an NSF "how to win grants" conference for university faculty. One presenter said roughly: Computer scientists seem to think grants are a zero-sum affair, but that's not the way it works. Astronomers tend to praise each other's work, with the result that many proposals end up with high ratings, and money is moved to support them. Computer Scientists seem to like to disparage each other's work, with the result that proportionally more proposals end up with low ratings, and so money is moved away from what's been self-evaluated as a large pool of poor proposals.
I was astonished to hear this unvarnished comment from a senior NSF proposal officer. But it is strikingly descriptive of the attitude cited above.

Re:Emotional Things I Wish I Knew Earlier

[metaforest]

As a test engineer I often experienced the fallout from #1 & 2# and suffered serious cognitive dissonance from #3 & #4. I gave up software testing after 17 years.

Don't do it unless it's absolutely necessary

[Draco_es]

Resist the urge to start coding and think hard about the problem your program will solve. If it can't be solved without coding, resist again, and design every piece of the program carefully, and, paraphrasing Dijikstra think about LOC's as lines spent, not lines produced.

social advice, especially applicable to geeks

[FuckingNickName]

You're not that great.

Even if you think you're the best person in your department, there are other departments.

Even if you think you're in the best department, there are other firms.

Even if your organisation's top of its league, empires rise and fall, and so will yours.

There is no silver bullet. You are no Superman. You're not going to change the world.

So shut up, listen and chill. Feel free to do your best, but remember to be nice. Money buys you hookers, but love gives you peace.

Re:social advice, especially applicable to geeks

[russotto]

There is no silver bullet. You are no Superman. You're not going to change the world.

I'm not, no. But some people have, and they may be reading this right now. How do you know you're not addressing the next Bill Gates?

Anyway, your advice is way off. The people with the most _social_ success are raging egotists and shameless self-promoters. There's right ways and wrong ways to do it, and the wrong ways lead to you being written off as a pompous ass -- but not being such will never lead to social success.

Re:social advice, especially applicable to geeks

[FuckingNickName]

How do you know you're not addressing the next Bill Gates?

The same would apply to Gates on his own. Even if you consider Microsoft as a whole (which is already not "Gates"), it hasn't "changed the world" except in some superficial detail.

The people with the most _social_ success are raging egotists and shameless self-promoters.

Define, in a way neither shallow, fleeting nor insincere, "social success". Explain how it would make you feel successful in the world.

Re:social advice, especially applicable to geeks

[russotto]

The same would apply to Gates on his own. Even if you consider Microsoft as a whole (which is already not "Gates"), it hasn't "changed the world" except in some superficial detail.

What's your bar for changing the world, the Great Oxygenation Event?

Re:social advice, especially applicable to geeks

[FuckingNickName]

My arguments are that no-one has changed the world on his own, and that no single change makes much difference.

The only significant changes in the human world are those which change man's nature. The rest are about finding a different rock to bash other rocks or other men with. Primates have evolved to think they're just so great when they've found a more powerful rock.

Man's nature occasionally changes, but not by much, and usually not for long.

You're just not that great.

Re:social advice, especially applicable to geeks

[russotto]

Ah, I see. You have no accomplishments of your own, so you set a high and arbitrary standard of what constitutes changing the world, so you can bring down the accomplishments of others as well. I hope a primate hits you with a rock.

Re:social advice, especially applicable to geeks

[FuckingNickName]

Perhaps English isn't your first language. I said that no individual changes the world, not that I have no accomplishments.

I try to have a sense of proportion about the power of a single human (including myself) and the ability of humanity in general (of which I am part) to act contrary to its nature. You appear unable to grow out of seeing the world in terms of superhero comics...

I hope a primate hits you with a rock.

...which is a fairly typical teenage insecurity, and manifests itself accompanied by anger at the suggestion of impotence.

Re:HUH?????

[Anonymous Coward]

I'd like to see this wonderkid take his non parallel code and parallelize it in five years time when he realises his non-parallel implementation doesn't scale.

From reading his other articles, you deal with the lack of scaling once you have a working product that people actually use. The huge majority of the time the load levels never appear to make scalability remotely relevant.

But I'm sure you're built lots of apps with millions of users, right? That's why you're hanging out on slashdot...

flock() works just fine?

[Rogerborg]

flock() isn't POSIX, doesn't have a standard behaviour across Linux/Unix/BSD platforms, can and will deadlock a process against itself, doesn't play well with either fcntl() or lockf(), doesn't work on NFS mounts, but does work on o2cb-based ocfs2 clusters, and God alone knows what the version(s) of Python / Perl / Java on your target system(s) are using to do locking under the hood - and He sure won't have documented it for you.

Doing it the easy way works 95% of the time, but if you don't take the time to find out what's inside that handy black box, you'll have a hard time putting it all back together (often on a customer site, and always at 11pm at night) when it does finally explode on you.

Re:until you find out why Python doesn't do the jo

[MikeBabcock]

And the percentage of lines of code written on each of those platforms is what exactly compared to the total?

Python frequently does do the job. So does PERL, or AWK or other interpretive languages.

For the record, and I always find this interesting, Jak & Daxter for the PS2 was written in LISP.

Version Control

[BluePeppers]

Put everything in version control. Everything. EVERYTHING!

Well. You could skip /home, but I know a roll back of /etc has saved me a couple of times on config upgrades.

Remember that once code is deleted, you can't get it back. However, version control changes that. Version control is one of the most vital tools for anyone developing/working with a computer.

Oh and git rocks and stuff :)

Re:Version Control

[pclminion]

And if you're going to use vc, then for gods sake learn how to branch. I've seen some truly suicide-inducing codebase arrangements due to people not knowing how the hell to branch.

Re:Version Control

[Anonymous Coward]

Yes, and, if you are developing software commercially, consider hooking your VCS into your bug/feature/whatever tracking software.

Force each commit message to begin 'issue xxx:', where xxx is an *open* issue number, then ensure that a new comment is attached to issue 'xxx' referencing the commit. If you do this methodically, you can later use your VCS to track every single line of your source code back to the original business requirement/bug that prompted it being written.

Useful stuff.

Re:Version Control

[l0b0]

Not everything, not by far. The DRY principle still applies - Don't version control stuff that is already in another repository or files that can be generated from your own or others' files automatically (i.e., binaries, but not personal settings). Otherwise, you'll be wasting lots of disk + network resources (when others' files are updated), you'll get a much more complicated repository, and you might end up with legal liabilities for distributing without permission.

Local multiplayer: what I/O bottleneck?

[tepples]

in the case of a 4-player game, the physical machine is the bottleneck. You can't have 4 players simultaneously use the same set of input peripherals

For over a decade, PCs have had USB ports into which four gamepads can be plugged through a hub.

display

Bomberman series, Tetris series, Mario Kart series, GoldenEye 007, Mario Party series, and Smash Bros. series put four players on one display. Sometimes it's split; sometimes all players' characters are in the same room. All six of those have appeared on Nintendo 64, whose GPU is noticeably less powerful than even the Voodoo3-class GMA in a netbook hooked up to a TV. So what bottleneck were you talking about?

Re:Local multiplayer: what I/O bottleneck?

[Anonymous Coward]

He was probably talking about one of the following:

1. The screen-space bottleneck where cramming four viewpoints through a display leaves each one with 1/4 the pixels and physical size. (Applies to split view only.)
2. The geometrical bottleneck of fitting 4 people around a monitor, which reduces the angular size for some or all players. (Applies to split or shared view.)
3. The input device bottleneck where, by and large, PC OSes easily treat multiple gamepad inputs separately, but merge all mouse inputs and all keyboard inputs. (Applies mainly to FPSes and other twitch games designed around mouse/keyboard, and also to complex RPGes which need a keyboard to control numerous actions.)
4. The geographical bottleneck where people who aren't sitting in the same room might sometimes want to play a game together.

2. is generally not an issue for TV games, because TVs are ordinarily set up to be viewed by multiple people, thus have bigger screens and easily fit 4 people in front at comfortable viewing distance.

1., in terms of physical display size, can also be somewhat alleviated by the larger size of TVs. The pixel loss, while still fully applicable, becomes mainly a cosmetic rather than functional issue because TV games (by virtue of the commonly available input devices) tend to use autoaim and emphasize button timing and sequences, which don't rely on resolution; PC games tend rely on pixel-critical aiming and mouse twitches, which require significantly better visual feedback.

And yes, those all drop to console+TV levels, if you hook up gamepads and a projector or big TV to your PC, and play games designed for that combination, but the original discussion was about PC game programmers, and most indie developers and all commercial game houses write PC games that play well on typical PC setups, not ones that require a possible but (for now) eminently atypical rig. And you still have the cosmetic pixel-reduction issue for split-screens, and the geographical issue for all games, which are already pushing console gaming towards the PC model as internet access becomes ubiquitous.

(Not to mention there's hordes of PC gamers who know that mouse-and-keyboard games are inherently better than gamepad games, and will never concede that they simply emphasize different skills, and it's a matter of preference... about the same response you'd get if you told the console crowd to hook up 4 mice and 4 keyboards and play PC-style games. Making a product that the intended audience will hate you for is not a good business plan.)

Re:Local multiplayer: what I/O bottleneck?

[aiht]

So what bottleneck were you talking about?

The bottleneck's not the video hardware - it's the screen surface area. That, and lots of people like having their own chair.
Yes, there are exceptions (I'd add Jump'n'Bump to that list). That does not change the situation for most games.

Re:Local multiplayer: what I/O bottleneck?

[tepples]

the original discussion was about PC game programmers, and most indie developers and all commercial game houses write PC games that play well on typical PC setups, not ones that require a possible but (for now) eminently atypical rig.

Home theater PCs are likely to become less and less atypical as HDTV adoption increases. And right now, indies have little or no way to legitimately get their game on a Nintendo or Sony console, so they have to code for PCs, even if their games are console-style.

and the geographical issue for all games

At family reunions, I've run into the reverse geographical issue: the gamers are here but their PCs are at home in another state. Parallelizing also increases the software licensing cost: compare one copy of a $60 console game to four copies of a $30 PC game.

Re:Local multiplayer: what I/O bottleneck?

[tepples]

The bottleneck's not the video hardware - it's the screen surface area.

A 32" PC monitor has been cheap for the last couple years. I answered in more detail in a reply to Anonymous Coward.

LISP/Scheme

[turgid]

I wish I'd know about LISP 25 years ago. Stupid people told me it was "for processing lists." If only I'd known better. Functional programming gives you wings and a jet engine.

I wish I hadn't paid too much attention to people with limited imaginations. Just because they're older, have more money and shout louder doesn't mean they are clever or wise.

C++ is way over-rated but it's worth knowing because it's so widely used. Don't let it detract you from mastering C and learning scripting languages. Understanding object-oriented design is more important than knowing the latest trendy language.

Objective-C.

Just because software is Free/Open doesn't mean it's "cheap" and poor quality. I could have saved myself 2-3 years there.

Ignore Windows and it will ignore you.

Re:LISP/Scheme

[pz]

I wish I'd know about LISP 25 years ago. Stupid people told me it was "for processing lists." If only I'd known better. Functional programming gives you wings and a jet engine.

Almost everything we're doing now in modern OSes (except for eyecandy) was done 25-30 years ago for the Lisp Machine.

Re:LISP/Scheme

[Khelder]

Everyone who programs for a living should learn LISP or Scheme or similar. Unless that's all they know, then they should learn C or Java or something else with a big imperative component.

Knowing different programming paradigms makes you a better programmer in whatever language you use, because different probems map better (sometimes much, much better) to different approaches.

Gee, way to take all the fun out of Linux coding

[w0mprat]

Reinventing the wheel uncessisarily and many times over, then forking the project is part of the fun. This is why Linux as a whole is not wanting for labour hours, but despite this brute force army of coders in thousands of open source projects all over world, the year of the desktop linux is always a few years away. Because when it's finished there will be nothing to do.

Re:HUH?????

[Anonymous Coward]

I'm sure you're built lots of apps with millions of users, right?

I've built systems that can handle hundreds of thousands of users, yes thanks.

That's why you're hanging out on slashdot...

Yes, because obviously with an intellect as massive as mine it's impossible to ever finish work for the day and sit around posting on the internet. What a truly stunning insight you have had.

Don't force Linux on users because "It's easy."

[jbeach]

Because Linux is NOT "just as easy as Windows".

I don't like M$ products and I don't like Apple's attitude. But, they are both easier to use for non-technical users than Linux. They are even easier to use for technical users, in many cases - for instance, if one wants to use an audio or video peripheral, finding a Linux driver for it often means hoping another Linux user wrote it, or you writing it yourself.

Linux is great and does a great many things, but it's not ready to replace Windows or OS X for the every day business user or at-home consumer.

Then what's the product?

[tepples]

Look, the PHB usually wants the code to run on HIS server for HIS use only.

Then what's the product? For example, in the case of an online game that runs entirely on the PHB's servers, graphics and all, I don't think every one of the PHB's customers has an Internet connection fast enough to run something like OnLive with acceptable picture quality and latency. And certainly, mobile devices in the United States market don't have the bandwidth, with the typical cellular data plan capped in the low single digit GB per month.

Re:Then what's the product?

[daveime]

I'm sorry, I was thinking business programming, you were thinking Facebook games. My bad.

Re:Then what's the product?

[Anonymous Coward]

And then there is always the case of future employment. If all you know how to do is what the PHB tells you then you are not a programmer, let alone an engineer. You are a trained monkey, and will be first on the asphalt when trouble rolls along.

What is a good boss going to want more. A trained monkey that only does what he is told, and makes/lets the boss make all the technology choices on his own. Or an engineering who can see what is coming down the road and can think outside of the box, and picks/suggest the best technology for the problem at hand.

Personally in my career, I've done what I suspect daveime thinks of as business programming (software to run on a server, at the time Alta-Vista size connections/performance). I've also done coding for released software (think OS-X on top of BSD, but not on that scale) and coding for user software (think Photoshop). Quite frankly there is more money to make out of that EOL'ed N900 and the software written for it than 99% of the business servers on the planet. What, you don't think Nokia re-uses code?

Stick to one software tool choice, and you career will go exactly one place. No-where. Good on you Tepples for encouraging people to think outside of their narrow experiences.

I'd log in, but basically I just wanted to call daveime an asshole and let him know with his thinking, 'business programming' is what he will be doing for the rest of his career. And I didn't want to burn the karma.

Re:Then what's the product?

[Bigjeff5]

This is Slashdot, you should have known better man.

Re:until you find out why Python doesn't do the jo

[Jedi Alec]

I like to use an interpreted language to make a proof of concept and play around with that for a bit. Then once I feel I have a solid grasp of the problem I'm trying to fix and the solution I want to use I'll do a rewrite in C if performance or other reasons warrant it.

Meaningfull messages.

[fahrbot-bot]

I don't know if this is still around, but a long, long time ago, I got this, and nothing else, from the shell when it died:

Assertion botch: This can't happen!

Nice.

Re:Meaningfull messages.

[marcosdumay]

I still miss the "YAAAAAAHHHHHHHHHHHHH!!!!!!!!!!! - Kernel Panic" (not a literal copy, I can't remember it exactly) message of Linux I used to get so often when messing with device drivers.

Also, there was a "goto's are funny" single line comment on sched.c I think on Linux 2.4.18. Those goto's were really funy (I needed to print the code to understand them).

The Truth

[TheCount22]

A few rules of thumb for a startup environment:

1. Don't overengineer! Overengineering wastes time on things that may never be used. Features should be customer driven.

2. Functions and methods should be as small as possible. You should make it an obsession to split methods and functions into the smallest possible components. Only then can you have good code reuse. Don't start thinking I will split it when I need it, you never will!

3. Never ever reinvent the wheel. Reinventing things that exist is overengineering.

4. Don't optimize ahead of time. When I say that I don't mean don't use a hash table instead of an array where it makes sense. I mean don't try to avoid exception handling or function calls or other minor optimizations. If it has an impact on readability don't do it. Optimization always comes last. Often you'll find there are only 1 or 2 "hotspots" in your code. If you spend time optimizing these "hotspots" after your application is built thats when you'll get the best return on your investment. Another gotcha with optimization is using technologies that can't deliver the level of performance you expect. You should test to make sure the underlying components you plan to use will perform as expected before you start coding.

5. Don't cram as much code in a single statement as possible. Every compiler I know about today will produce identical code whether it's one statement or 5 statements. It makes it hard to read so don't do it!

6. Allocate time for testing. No one writes perfect code.You want to give a good impression to your customer so don't skip this step.

7. Make unit testing an obsession. Always add unit tests for new code, it reveals errors in your code. When you find a bug in your code add a unit test to test for it. If in the future someone decides to rewrite some function or method you wrote because it's not elegant enough they will not reintroduce old bugs.

8. Don't rewrite code if possible. Refectoring is almost always easier and less error prone.

Re:The Truth

[crimsontime]

Great list. I'd add 1) Don't build things into libraries that are convenient for you but inconvenient for others. 2) Never muck with low level code except in dire need.

Re:The Truth

[Anonymous Coward]

See now, that there, that is what I was expecting in the article. About programming. "Don't do it if Linux can do it?" What? the only programming being done is under Linux now? Windows sucks as a development environment, but I suspect you can still get paid to write software for it. NoSQL sucks? Really, so you programming tip title is about a specific development tool?

The parents points apply to coding for any type of program (user, system, enterprise) and any OS. The article on the other hand was narrow to the point of uselessness beyond the guy who wrote it.

Re:The Truth

[Anonymous Coward]

A few rules of thumb for a startup environment:

1. Don't overengineer! Overengineering wastes time on things that may never be used. Features should be customer driven.

2. Functions and methods should be as small as possible. You should make it an obsession to split methods and functions into the smallest possible components. Only then can you have good code reuse. Don't start thinking I will split it when I need it, you never will!

3. Never ever reinvent the wheel. Reinventing things that exist is overengineering.

4. Don't optimize ahead of time. When I say that I don't mean don't use a hash table instead of an array where it makes sense. I mean don't try to avoid exception handling or function calls or other minor optimizations. If it has an impact on readability don't do it. Optimization always comes last. Often you'll find there are only 1 or 2 "hotspots" in your code. If you spend time optimizing these "hotspots" after your application is built thats when you'll get the best return on your investment. Another gotcha with optimization is using technologies that can't deliver the level of performance you expect. You should test to make sure the underlying components you plan to use will perform as expected before you start coding.

5. Don't cram as much code in a single statement as possible. Every compiler I know about today will produce identical code whether it's one statement or 5 statements. It makes it hard to read so don't do it!

6. Allocate time for testing. No one writes perfect code.You want to give a good impression to your customer so don't skip this step.

7. Make unit testing an obsession. Always add unit tests for new code, it reveals errors in your code. When you find a bug in your code add a unit test to test for it. If in the future someone decides to rewrite some function or method you wrote because it's not elegant enough they will not reintroduce old bugs.

8. Don't rewrite code if possible. Refectoring is almost always easier and less error prone.

I meant refactoring.

Re:The Truth

[ljw1004]

I strongly disagree with "2. Functions and methods should be as small as possible. You should make it an obsession to split methods and functions into the smallest possible components."

I figure that an extra STATEMENT adds a bit of extra work for the person who maintains your code to understand;

an extra FUNCTION adds about five times as much work since the control-flows into and out of it are now harder to understand;

and an extra CLASS adds four times as much work again to understand what it abstracts and how lifetime &c. work.

So, every time you use a function for what could have been done with a statement, you're making your code harder to maintain.

Re:The Truth

[Anonymous Coward]

I really don't like it when I hear someone say to avoid overengineering. Of course by-definition over-engineering is bad, but this is generally bad advice. Back when I started doing software engineering out of college I admit I did some over-engineering. When you have someone inexperienced but passionate about software this will happen on occasion. My advice is, don't worry about it, it will pass in time.

However, despite running into over-engineered solutions on occasion, I don't really mind them. The worst code I have seen was done by people in the name of avoiding overengineering. Although I have seen them, overengineered solutions are so much less frequent than underengineered solutions. Also they generally work. I can handle maintaining them as I can slowly simplify it as I work. Usually there is a bit of a learning curve but once handled it can be easy to work with.

It is much harder when the design is too simple to handle the requirements. I have to add full error handing and support usage scenarios that the software wasn't designed for but was expected that it had been. It usually needs to be almost entirely rewritten because it wasn't modular or flexible at all. This type of code also tends to be "optimized" by using normally bad coding practices which the author thinks will make the code faster but didn't actually bother to check.

So sure, avoid overengineering but don't avoid it too much. It is like economics with inflation and deflation. Both can be bad but a little inflation to avoid deflation isn't a big deal.

Re:The Truth

[sog_abq]

I concur. I find that code is very infrequently reused. Often, it will take longer to understand some horrible api that has too many classes, parameter overloading (a personal pet peeve), and other unnecessary over-engineering. Instead, a modest, and mostly functional, design will be easily refactorable when you find that reuse is needed. I find that it is faster to refactor code when I need to reuse it than it is to engineer it for all possible reuse cases.

Re:The Truth

[wvmarle]

Your point 3) is very true, but so is a point made in the summary: "most hard problems are already solved for you. You just need to know where to look." This knowing where to look part is hard in itself.

For me it boils down to searching for some existing library whenever I realise that I need a complex function to do something, that sounds reasonably generic at the same time. E.g. image resizing. Sorting stuff. Building up and sending out an e-mail from software. Logging to syslog or your own log file. Python has a lot of that built in, and well documented, but lots of stuff is harder to find, such as creating a .pdf file as output.

Re:The Truth

[TheCount22]

I strongly disagree with "2. Functions and methods should be as small as possible. You should make it an obsession to split methods and functions into the smallest possible components."

I figure that an extra STATEMENT adds a bit of extra work for the person who maintains your code to understand;

an extra FUNCTION adds about five times as much work since the control-flows into and out of it are now harder to understand;

and an extra CLASS adds four times as much work again to understand what it abstracts and how lifetime &c. work.

So, every time you use a function for what could have been done with a statement, you're making your code harder to maintain.

In time you will learn that the alternative is worst. Even if it doesn't look like it at first.

Re:The Truth

[TheCount22]

I really don't like it when I hear someone say to avoid overengineering. Of course by-definition over-engineering is bad, but this is generally bad advice. Back when I started doing software engineering out of college I admit I did some over-engineering. When you have someone inexperienced but passionate about software this will happen on occasion. My advice is, don't worry about it, it will pass in time.

However, despite running into over-engineered solutions on occasion, I don't really mind them. The worst code I have seen was done by people in the name of avoiding overengineering. Although I have seen them, overengineered solutions are so much less frequent than underengineered solutions. Also they generally work. I can handle maintaining them as I can slowly simplify it as I work. Usually there is a bit of a learning curve but once handled it can be easy to work with.

It is much harder when the design is too simple to handle the requirements. I have to add full error handing and support usage scenarios that the software wasn't designed for but was expected that it had been. It usually needs to be almost entirely rewritten because it wasn't modular or flexible at all. This type of code also tends to be "optimized" by using normally bad coding practices which the author thinks will make the code faster but didn't actually bother to check.

So sure, avoid overengineering but don't avoid it too much. It is like economics with inflation and deflation. Both can be bad but a little inflation to avoid deflation isn't a big deal.

Avoid over-engineering and avoid no engineering is what it comes down to.

Re:The Truth

[QuzarDC]

5. Don't cram as much code in a single statement as possible. Every compiler I know about today will produce identical code whether it's one statement or 5 statements. It makes it hard to read so don't do it!

While it can make code harder to read, it can also help show certain specifics and at least with my experience with gcc will produce faster code (due to better register usage, and superscalar optimization) in many applications.

Take the following simple example:

a = x(); a *= y(); a += z(); vs a = (x()*y())+z();

This may look trivial, but it can make a difference to someone showing up later, as well as to the compiler now. Of course, depending on x,y,z the two may give different results, but I write assuming that both demonstrate the 'correct' outcome. The two show different rules as to the order in which operations must be applied. The second uses language much better suited to explaining a relative order which allows the compiler to decide what is best depending on what resources are free, not only that, but it will show whoever is reading it more explicitly the required order.

If I were to come along the first one at a later time, it would take some digging to ensure that x,y, and z are not dependent on one another and the order they run in does not matter to their results. With the second, it's clear from the start.

This is in many ways an extension of an adage I was barraged with in many high-level-complexity, low-level-code classes "NO MAGIC NUMBERS!". The idea was to use defines or variables instead of fixed numbers whenever possible. It worked to make code more readable, and more flexible with minimal extra effort.

DS Lite and iPod touch don't require data plan

[tepples]

The smart phones are already more powerful than many old PC desktops still creaking on merrily

A lot of people carry a DS Lite or iPod touch and a cheapo feature phone instead of an Android phone precisely to avoid the 70 USD/mo voice and data plan associated with a smartphone.

Re:DS Lite and iPod touch don't require data plan

[Anonymous Coward]

A lot of people apparently don't know what they're talking about because they can use the exact same plan on a smartphone -- they just will have the same (lack of) internet connectivity they do with their DS lite or iPod Touch.

OTOH, maybe they carry the separate gadget because it's cheaper, because they want assurance that they'll have juice to make phone calls even if they get carried away gaming or browsing, or because they work in a secure facility that forbids cameraphones (pet peeve: though it doesn't affect me, it seems there's a real dearth of smartphones without cameras for those guys -- I know a number of geeks who are forced to settle with a device they don't really want, or with the expense of two phones and the hassle of switching a SIM twice a day. That sucks.)

In any event, even the new dumbphones have some horsepower these days, and as a rule featurephones are no more than a year behind smartphones under the hood.

3rd party code is nice and all, until...

[J-F Mammet]

Like everyone else I went ahead and used third party code and libraries to accelerate the development of my various sites. Everything went as well as possible I guess, until one of my sites was defaced one day. Not because of my code (though by no means I'm pretending my code is perfect, far from it), but because one of these libraries had a security vulnerability and they didn't even have a security mailing list. That vulnerability was big enough that it would show in about every single site that would use this particular library, and was of course exploited very quickly by all the script kiddies in the world (I still see it being scanned automatically from time to time, years later).
Lesson learned, I don't ever use any 3rd party code unless there is a announcement security mailing list anywhere on their site, and even then I'd rather do the code myself if possible. Not because it will be perfect, but because at least my site won't be vulnerable to an automated attack targeting a 3rd party thing I put in their and totally forgot about.

And of course, don't get me started on phpBB and stuff like that, using such apps a few years ago was either having open doors for hackers, or a nightmare of patching.

I disagree with nearly every point...

[SanityInAnarchy]

If you are writing a program that touches more than two persistent data stores, it is too complicated.

Others have already mentioned cases where multiple datastores make sense. A trivial example: One database to handle user data, another to handle blobs (image conversions, etc) -- bonus if the second store can do its own conversions; a third to handle logging -- that's already three, and that's before we start considering things like RESTful services, which can function as intelligent datastores of their own...

If Linux can do it, you shouldn't.

Unless you're not on Linux. And, specifically:

Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.

If you're doing something that truly works as a shell script, and isn't part of a larger app, I agree. However, PIL likely performs better, and it removes the shell as an issue -- if you thought SQL injection was bad, wait till you have people exploiting your shell commands. You can do it safely, but why would you bother, when you've got libraries that accept Python (or Perl, or Ruby) native arguments, rather than forcing you to deal with commandline arguments? Why do you want to check return values, when you can have these native libraries throw exceptions?

Parallelize When You Have To, Not When You Want To

If you don't at least think about parallelization in the planning stage, it's going to be painful later on. It's easy to build a shared-nothing, stateless architecture and run it in a single-threaded way. It's hard to build a stateful web service with huge, heavyweight sessions, and then make it run on even two application servers in the future. Possible, but awkward, to say the least.

For example, if you are doing web crawling, and you have not saturated the pipe to the internet, then it is not worth your time to use more servers.

...unless, maybe, it's CPU-bound? And this is odd to mention in a section about parallelization -- wouldn't slow servers be a prime candidate for some sort of parallelization, even on a single machine, even if it's evented?

If you have a process running and you want it to be restarted automatically if it crashes, use Upstart.

Cool, but it looks like Upstart is becoming a Maslow's Hammer for this guy. Tools like Nagios, Monit, and God exist for a reason -- one such reason is knowing when and why your processes are dying even if they're spread across a cluster.

NoSQL is NotWorthIt

People who have read my other posts likely know where I stand on this, but...

Redis, even though it's an in-memory database, has a virtual memory feature, where you can cap the amount of RAM it uses and have it spill the data over to disk. So, I threw 75GB of data at it, giving it a healthy amount of physical memory to keep hot keys in...

So you found out an in-memory database wasn't suitable when you have far more data than physical memory? Great test, there.

Redis was an unknown quantity...

Maybe so, but that wasn't terribly hard to guess.

Yes, maybe things could have been different if I used Cassandra or MongoDB...

So maybe you should've benchmarked a NoSQL database which is actually designed to solve the problem you're trying to solve? Just a thought.

especially if something like PostgreSQL can do the same job.

If PostgreSQL could do the same job, the current generation of NoSQL databases wouldn't have been invented. Unless something's changed, PostgreSQL can't scale beyond a single machine for writes, unless you deliberately shard at the application layer, which would violate his rule about multiple datastores, wouldn't it?

It seems like the attitude is to no

Re:I disagree with nearly every point...

[codepunk]

I disagree with nearly every single response that you made.

Re:I disagree with nearly every point...

[Bigjeff5]

Why?

Re:I disagree with nearly every point...

[evilviper]

If you're doing something that truly works as a shell script, and isn't part of a larger app, I agree. However, PIL likely performs better, and it removes the shell as an issue -- if you thought SQL injection was bad, wait till you have people exploiting your shell commands. You can do it safely, but why would you bother, when you've got libraries that accept Python (or Perl, or Ruby) native arguments, rather than forcing you to deal with commandline arguments? Why do you want to check return values, when you can have these native libraries throw exceptions?

I have to disagree with you there, for a couple reasons:

The command-line version came first, has been widely tested and debugged, can be used no matter what the language of your application, and will always be the standard and won't ever disappear or be unmaintained even if the libraries for language X are.

And I've seen no end of (mainly Perl) libraries that re-impliment whatever command, poorly. I've probably wasted 3 months of my life on a stupid perl script because my predicessor used Net::SFTP, and I've had to debug endless problems on different sites where their peculiar SSH/SFTP server works with ssh/scp/sftp but NOT with Net::SFTP, or worse, is terribly intermittent, perhaps working for 10 minutes then dropping the connection. Using batch mode SFTP would have taken a little longer to start with, but would have avoided all these problems.

In short, if the command-line program doesn't work, EVERYONE is going to care. If the hacked-up reimplementation for language X doesn't work, you and the 10 people who use it are on your own.

Didn't understand what you were reading...

[IBitOBear]

eh... "input, output, and logging" as you describe is _one_ data store in the original post, namely the disk cum filesystem. That's why the original article said (paraphrased) "the disk being one data store." If you intelligently apply the rule then disk + database is okay, disk + net is okay, database + net is okay, but disk + database + net is probably unnecessarily complex.

eh (further)... the article presumed linux, and I will skip the *NIX vs MS debate here. The rule espoused is better stated If you have a wheel already don't try to invent it. *NIX vs MS is, in this context, only a difference of how many wheels you have and how well they work together.

eh (further still)... benchmarking subprocess.Popen(['convert', ...]) is a myopic response, foremost because if you are CPU bound then you may have already surpassed the predicate condition on the choice. Since ImageMagick is also a _library_ there is no need to call it through fork/exec anyway. So since you didn't investigate the tools maybe you shouldn't be critiquing their use or offering benchmark advice.

eh (yet further still)... games, at the level you are speaking of, are always CPU bound for rendering. By definition the machine (and the network latency of transferring pre-rendered frames etc) makes the machine the bottleneck in the game if it were monolithic. Your example supports the need for the predicate "if" rather than counters it. That is, you have provided an example of when you _do_ need to distribute the processing load rather than engage in the argument about when you _shouldn't_.

eh (yet again)... you give an example of where you _cannot_ buy your way out of a performance problem rather than discuss a situation where you _could_ buy your way out and then discussing the merits of performing the purchase rather than rewriting the code. That is, you have brought everyone and their car as a citation into a discussion of trans-Atlantic travel. The OP is discussing the use of upgrading to heavy haulers to cross an ocean and you yell "but most people only drive cars" as if it had the slightest bearing on the conversation.

So, having failed utterly to understand the articles points about using existing technology and keeping things manageable, you have made several observations of the "but since sometimes you cannot keep things simple, you should never try to" or some such nonsense. Citing already accounted-for corner-cases in apparent inability to understand the scope of the general case doesn't really add much to the discussion of the general case.

There is a reason the original article was filled with predicates like "if" and "when". Giving examples outside those predicates is, in no way, applicable to the discussion within those predicates.

[snark]Please post you CV so that I can make sure I never hire you to do a requirements analysis...[/snark]

Re:Didn't understand what you were reading...

[Gverig]

You missed both the point of the article and the point of the reply.
E.g. "If you intelligently apply the rule"- what rule? What are you talking about "net"? Since when is net a persistent store? Problem original article is talking about is maintaining integrity when state is spread across multiple stores. Unfortunately a) the article does not just come out and say that and instead goes into some weird rules about inputs and outputs and b) there are (and pretty common) very legit cases where violation of this rule is required. And in those cases you should deal with any edge conditions explicitly. And no "database and log file and images and whatever else crap is the single store because it's on disk" shows that you did not understand the point of the article. "single store" is wherever you have a clear contract between yourself and the final state of the store (aka transactionality or even ACID if you are lucky). BTW, most "persistent stores" other than databases make no guarantees so even if you are using only one output you still need to enforce your own failure checks.

Rest of your response shows as much understanding. Note, that tepples is *arguing against* blunt points made in the article, not making his own. Take that in context as well.

Cheers,
---

Be weary of toolkits and APIs

[GWBasic]

Be weary of toolkits and APIs that claim to solve difficult problems. They might solve it in the best way for small-scale deployments, but won't scale to fit your needs. ORM is a good example of being careful when choosing an API that claims to solve a difficult problem. Hibernate can quickly complicate traversing foreign-key relationships if it's used as a replacement for knowing how to use a database. In contrast, a team that knows how to do database programming might be happier with a much simpler ORM API.

Following up on the ORM example: A popular ORM technology is no replacement for good database design and a team member (or members) who know how to program a database. Even if you're using an embedded database that'll grow to 40 gigabytes, someone on the team needs to be comfortable programming with it.

Stick with tools that "do one thing and do it well." If the tool does many things, there should be a decent ecosystem that developed the "many" things. jQuery is a great example. It nicely abstracts browser differences and gives a helpful wrapper for dynamic HTML. There's also a healthy ecosystem of community plugins.

Avoid tools that get in the way of how you normally program, except in confined areas that can be refactored without starting from scratch. Spring.Net's aspect-oriented programming plugins for C# can leave lots of layers of auto-generated code on the stack trace, and require lots of additional work outside of normal C# in order to get it to work. Node.js's asynchronous approach is getting a lot of attention, but if your program crashes, you won't have a usable stack trace, and it's difficult to do a try-catch-finally around a non-blocking IO call. In both cases, when new technology brings akwardness, limit its use to one-off utilities or a small part of a larger whole. This way if the new technology proves too immature, your risk is minimized.

Code generators can be useful timesavers, but their scope should be limited to one or two layers of a program. Code generators that hit all layers of a program can become too inflexible to handle changing needs outside of what were originally anticipated. Likewise, code generators can become so complicated that it's easier to avoid using the generator altogether.

Runtime code generators need to be simple and well-tested. (These typically implement .Net or Java interfaces at runtime, or inherit from classes at runtime.) Bugs in code generators can be difficult to find and fix, because you'll have an incomprehensible stack trace that doesn't lead back to the bytecode generator, and because bytecode programming is quite time consuming.

Don't be afraid to write one or two utilities yourself, even if there are pre-existing libraries. As long as you have a good justification, it's a helpful learning experience.

There's a reason why many different APIs exist that appear to do the same thing. .Net has three different XML handling APIs; each is optimized for something different and introduces valid tradeoffs. Choosing between APIs, utilities, or libraries often isn't a matter of which one is best, but instead is a matter of which will meet the design needs of your program.

Re:Be weary of toolkits and APIs

[bigdavex]

Be weary of toolkits and APIs that claim to solve difficult problems.

I think the word you want here is wary.

Re:Be weary of toolkits and APIs

[Builder]

No, no... I think he meant weary. After a couple of months of working on a simple inventory product that was built using Spring where spring added more confusion than clarity, I was very, very weary!

Oh that's like a

[NotSoHeavyD3]

Macro in C/C++. (That auto code generation.) That sounds like it kind of sucks in Java. (The guy I'm talking about did it in C/C++. To make matters worse, he got his code wrong so he had to fix it 70 times and then retest.)

Re:Oh that's like a

[ADRA]

People who copy/paste code always get something wrong! That should be lesson #1!

The tech writers are friends not enemies

[Anonymous Coward]

They are there to be aggressive advocates for the user. /They can also save your arse because guess who is the one person in any meeting who took minutes and a manic depressive Project Manager is blaming you for not following through on an order that he failed to giv.

Simplicity can get you fired

[spectro]

I just finished a project where the other developer was a noob that thought he was uber and my code was crap.

He lobbied the boss behind my back and they started pushing me away, not including me in design meetings and reducing my coding responsibilities meanwhile the "uber" dev overcomplicated the code by throwing factories and inheritance all over the place without any regard to design patterns and coupling....

So yeah, keep it simple but make sure the boss understand your design decisions.

Re:Simplicity can get you fired

[Anonymous Coward]

Sounds like something you may want to be distanced from anyway!

"You just need to know where to look."

[grikdog]

Thank you, thank you, Mr. Ultra Obviousman. Open source usually means closed, archaic, obsolete or entirely missing FM-for-which-it-is-impossible-to-RT.

Has ImageMagick improved?

[SL Baur]

Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job.

I think the worst mistake I made as Mr. XEmacs was attempting to unify our graphics support to call ImageMagick libraries instead of the custom stuff we were using (and later restored when ImageMagick was backed out).

Does it work any better now? The last time I looked at display(1) a couple of years ago, it still wasn't close to long lost and patent challenged xv(1) that got shut down by the GIF patent war.

Re:Has ImageMagick improved?

[dbIII]

I always thought XV was shut down for being the shareware everyone had but nobody ever paid for. I still have about ten users that use it frequently.

Re:Has ImageMagick improved?

[Just Some Guy]

Does it work any better now?

Yep - mostly. I do a lot of image processing at work and discovered that all libraries suck most heinously if you work outside their assumptions. I eventually wrote my own, and started with a base class that uses calls to ImageMagick's "convert" command for every operation that our other in-house code would ever need. Then I subclassed it with format-specific classes that use better tools for specific operations.

For instance, we very often have to split multi-page images into individual pages. The base class implements that with convert's "+adjoin" operator which pretty much always works for every image format. Unfortunately, it's slow as hell on large TIFF images. I overloaded TiffImage.split() to call the "tiffsplit" utility instead. On a sample image I just now tested, "convert +adjoin foo.tif" took 19.7 seconds, while "tiffsplit foo.tif" took 3.0. Similarly, joining them back together with "convert -adjoin" took 18.4 seconds, while "tiffcp" took 12.5 seconds. Based on that, TiffImage.combine() calls "tiffcp".

The resulting library is a set of the most optimized methods I know of for performing all those operations, backed up with benchmarks. Each subclass is actually pretty simple since it only contains those few functions, but the payoff is a huge win.

Still, the best optimization for ImageMagick is to replace it with GraphicsMagick. I've yet to have a problem with that upgrade.

Re:Has ImageMagick improved?

[terremoto]

I always thought XV was shut down for being the shareware everyone had but nobody ever paid for.

Hey! I paid for it!

The total charge was $25.00 plus $0.00 tax.
...
Contact: John Bradley
Email: xvcc@trilon.com
Receipt: [elided]

One of us did at least

[tepples]

"input, output, and logging" as you describe is _one_ data store in the original post, namely the disk cum filesystem. That's why the original article said (paraphrased) "the disk being one data store."

The database is stored on the disk.

disk + database + net is probably unnecessarily complex.

I have one program that scrapes information from a web site into a database. (I have another that makes reports on the disk.) Does this mean the logging should also be to the database?

Since ImageMagick is also a _library_ there is no need to call it through fork/exec anyway.

In that case, PIL vs. PMW may be a wash. But in context, I understood the gist as the following: if shell is good enough, don't use Python.

games, at the level you are speaking of, are always CPU bound for rendering.

I thought they were GPU bound. But if they're fill-rate bound (as opposed to vertex shader bound), putting four 960x540 pixel windows on one screen won't tax the pixel shaders any more than one 1920x1080 pixel window. And some game styles don't even need four different views: look at Bomberman.

and the network latency of transferring pre-rendered frames

Why would you need to transfer frames over a network? You just display all four players' frames in windows on one monitor. Goldeneye 007 for Nintendo 64 does this. Or you render one frame that's suitable for all four players and again display that on one monitor. Smash Bros. does this.

you give an example of where you _cannot_ buy your way out of a performance problem

I took the article to mean that buying your way out is the preferred solution, and I gave a counterexample.

"but since sometimes you cannot keep things simple, you should never try to"

That is not what I meant. I meant only this: "but since sometimes you cannot keep things simple, you should never force yourself to."

There is a reason the original article was filled with predicates like "if" and "when". Giving examples outside those predicates is, in no way, applicable to the discussion within those predicates.

The problem comes when people read "if A then B", either gloss over the "if A" or forget the "if A" months later, and then blindly apply B without considering A. For example, consider Dijkstra's "Go To Statement Considered Harmful", which refers to "the unbridled use" of GOTO, but people forgot "unbridled". So if !A is a common occurrence, of course discussing how !A relates to B is on-topic.

Re:One of us did at least

[Anonymous Coward]

The database is stored on the disk.

If you're using a database where that matters in the API (other than as a URI), then you need to find another database.

Re:One of us did at least

[tepples]

If you're using a database where that matters in the API (other than as a URI), then you need to find another database.

I take it you're not a fan of SQLite, which implements a good chunk of SQL using a file in the file system as the underlying data store. It's not for highly concurrent applications, but it's perfect for a batch job that has only a couple programs occasionally hitting the DB.

Early optimisation

[Anonymous Coward]

It's good to design with some speed in mind. Structure your database for the kinds of queries you'll be running.

I've seen people waste effort producing unreadable code to shave nanoseconds off a system that then spends seconds or so waiting for a database to catch up. They worry about the cost of memory allocation or iterating over collections and go through hoops to avoid them, but this is nothing in comparison with the network and disk latency the job they are doing involves. Best optimise code later when you can get a profiler to work out what's really making it slow. Chances are it's your database queries.

lord duh

[Anonymous Coward]

Don't use PIL if you have imagemagick? If I'm using python anyway, why add another dependency? If I'm not using python than obviously i'd use IM. Does this giy actually roll his own for no good reason? Lulz what a geek.

Re:until you find out why Python doesn't do the jo

[Anonymous Coward]

Sure, it was written in LISP, and then the program got bottlenecked on the only programmer in the world who could understand the code, and they had to deal with 15 minute long garbage collection interrupting development and two games later they had to scrap the whole engine because the one guy in the world who understood it doesn't work for them anymore.

Re:HUH?????

[mabinogi]

if it takes five years for your solution to stop scaling, then you've probably done a pretty good job.

Re:Don't code in C

[danieltdp]

Hey, you can do everything in assembly!

Recent convert, so apologies...

[wild_berry]

I wish someone had suggested I and the team I'm in work in a flexible and agile way a long time ago. I'm a recent convert, so excuse my fanaticism.

The manifesto recommends:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

in light of the following principles:

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity--the art of maximizing the amount of work not done--is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

I never commented code until

[gagol]

I had to deal with massively asynchronous code. My head would have blown out without proper comments and a flow chart for the code.

Overusing a database is fun too!

[RulerOf]

While it is not impossible to learn and implement good data management/database development practices using Microsoft tools, such a result is seldom seen in the wild.

I've got a funny story that might give you a heart attack.

I peeked at the helpdesk one afternoon to see that there was a ticket from one of the programmers. One of the web pages they built was throwing a script error and it was likely a "permissions issue." I noticed the browser had a UNC path in it (to the web server, no less) and figured that had to be the problem. So I hop on the web server and drill down to the folder and find the URL for it, pop it in my browser... still not working. I crack the HTML file and get no help there, so I start tracing through it with Chrome... where I find the SQL connection string in the included javascript file and nearly have an aneurysm.

After the programmers assured me that "that's just how it's done there" and not knowing enough VB/ASP nor it being my job, I hopped on the SQL server and added the user group to the stored procedures necessary to get the pages to work.

At least it was using Windows Authentication and not "sa" right in the javascript file :-P

Re:Here's MY list

[Tubal-Cain]

by .Bruce Perens

Someone didn't pay much attention to Ender's Game. Period belongs at the end of the name. Of course, and underscore might have been better here.

2. Bathe periodically.

I bathe all the time.

5. Parents will eventually charge rent.

So do landlords, incidentally.

7. CTL-C, CTL-V. Took me 15 years until someone showed me that shortcut.

This ought to blow your mind then: CTL-X

8. The sun, it burns.

Funny, I figured that out long before I owned a computer.

9. As far as your family is concerned, "network admin," "programming," "database management," ... to them, it's all "computers."

It's good practice for talking to Management later.

Re:until you find out why Python doesn't do the jo

[Anonymous Coward]

I wish parent had learned about if statements.

Sk

[Anonymous Coward]

Next up is a rework of the database IO codebase so that it becomes feasible to plug-n-play different databases.

Why? Pick a widely use database that works and stick with it. Less work, simper code, easier to test and a shorter route to maturity.

Tao of Programming

[SL Baur]

That was a truly a forgettable book. I had a copy and even tried to read it through several times. I cannot recall anything about it other than the title.

That makes me sad in a way, because I usually can read something end to end.

Regarding the quote: It's not that COBOL was such a bad language for its time. It wasn't. Of the three languages COBOL, FORTRAN and Lisp, only Lisp survives (in new code) in anything resembling its ancestor. COBOL survives in legacy code that will probably never be retired. FORTRAN has mutated into something unrecognizable and arguably didn't last.

We can be grateful for FORTRAN because it buried the notion that compilers could never beat hand coded assembly.

We can be grateful for Lisp because it later spawned The One True Editor.

We can be grateful for COBOL because it made so mistakes that were so glaringly obvious, no one ever made them again.

Re:Tao of Programming

[symbolset]

I think the point of that particular line was that if you're choosing a language to program in, one that's more unpleasant than is needful probably should be your first choice. We're pretty spoiled these days.

We have databases that suck...

[Estanislao Mart�nez]

Business logic is part of the database.

I used to think the same thing, but that was before I ever had to solve a no-trivial problem. What do I think now? Basically that business logic belongs in a domain layer, but that shouldn't mean that the database is treated like a black box.

Well, you're both right. Business logic belongs in the database--but our databases are too weak when it comes to integrating with general-purpose programming languages, which is the whole reason why we have domain layers. Our relational databases come from the world of COBOL, and it shows in their built-in languages (e.g., SQL, PL/SQL).

We'd be better served by a relational database system that embedded its features inside a good programming language. I'll be bold and sneak in the suggestion that this ought to be a functional language; after all, the relational algebra is a simple kind of functional language.

Re:We have databases that suck...

[WarwickRyan]

> We'd be better served by a relational database system that embedded
> its features inside a good programming language. I'll be bold and sneak
> in the suggestion that this ought to be a functional language; after all,
> the relational algebra is a simple kind of functional language.

It's a problem which will be solved. Whoever solves it is going to be a very rich man. How do you combine the best features of the relation model (high performance when working with sets, fantastic domain specific language in SQL and data integrity) with those of an object database (better scalability curve and if done right the logic's coupled to the data).

I've not mentioned reporting, mainly because you quickly hit the limits of a normalised relation model and have to implement OLAP cubes to handle the data. Which, although clever, is a hack.... and doesn't really need all of the features of relation databases (locking, for example, is often not required).

Smart object-db providers would ship a SQL-fascade for their datastores to allow complex querying, but the performance and specifically integrity model of current implementations need a lot of work. Which is why ORM is still used so often..

Re:HUH?????

[skyride]

I've built systems that can handle hundreds of thousands of users, yes thanks.

No. He asked if you've built systems that are actually used by millions of users. The first user system I ever wrote could theoretically handle hundreds of thousands of users, but it was only ever used by 7.

Re:until you find out why Python doesn't do the jo

[Anonymous Coward]

Hell, I got an N900 6 months ago, and it's already EOL'd as far as updates to the OS are concerned.

I suppose I could wait till there's a port of Android or something, because the three coders on the project are doing a fine job, in less than a year I'll have the same functionality as a 3210.

NITdroid already provides voicecalls over gsm so it's coming along nicely.

Maemo seems EOLed because nokia and intel are focusing on Meego which will be released for N900 aswell.

Atleast with N900 you get platform which is open enough to modify to your liking, rather than being locked in a format which is force-fed from 'central command'

Rephrased

[tepples]

I wish parent had learned about if statements.

Perhaps I phrased it wrong. Let me try again: "I agree that C++ isn't always the right tool for the job, but then neither is Python. It is still wise to learn C++ because you're likely to need it sometime in your career." Is that phrased better?

Re:HUH?????

[Anonymous Coward]

I've built systems that can handle hundreds of thousands of users, yes thanks.

Which doesn't change the fact that you're the only one using it, making the scalability redundant. If you read again he said with millions of users, not just able to support them. Some in-house software that will only ever need to support up to 200 users shouldn't care if it can run for 2 million or not. If that suddenly becomes a requirement then there's much more than scalability to worry about.

Not that I agree that scalability shouldn't be thought about, it should but in the background, unless it's in the clients top 3 requirements. Good, decoupled code will always allow you to come back and optimise for scalability later.

Re:Here's MY list

[Anonymous Coward]

You are a strangely literal person. And your comeback sucked, I hope you don't think you were at all witty.

Who the fuck is Ted Dziuba?

[Anonymous Coward]

My main question is, Who the fuck is Ted Dziuba? My next question is, Why should I listen to what he has to say?

Cow-orker

[dbIII]

A co-worker proudly demonstrated his brilliance to myself and the CEO - spending two days writing a program on linux that swapped bytes to read data from old tapes. That is all it did. His input was piped from dd.
What could I say apart from "that's nice", instead of "why didn't you spend two minutes reading the dd manual and get dd to do it all for you instead of two days on a very simple task - or just ask me instead".

in

[The Clockwork Troll]

In B4 Persai

Its called learning

[saiha]

There is a reason that every write "hello world". There is a reason that it is actually ok to reinvent the wheel, if only to figure out how the wheel works so you can use it better.

I gotta say I cannot believe this hit /. frontpage, are there no more "do it yourself" programmers around?

Re:until you find out why Python doesn't do the jo

[dbIII]

Hell, I got an N900 6 months ago, and it's already EOL'd as far as updates to the OS are concerned.

So, where exactly did you get that information from and why do you think it is real?

Re:Here's MY list

[Tubal-Cain]

No, just bored.

Re:until you find out why Python doesn't do the jo

[Kludge]

Hell, I got an N900 6 months ago, and it's already EOL'd as far as updates to the OS are concerned.

I suppose I could wait till there's a port of Android or something, because the three coders on the project are doing a fine job, in less than a year I'll have the same functionality as a 3210.

Or you can read a little farther and see that upgrades are already available: http://en.wikipedia.org/wiki/MeeGo

Anonymous Coward

[Anonymous Coward]

perl wisdom.pl

Don't put your file system under version control

[Anonymous Coward]

That's the wrong tool for the job. Just use a modern file system with snapshotting capabilities, like btrfs.

Observations from a curmudgeon

[mnemotronic]

These are the rules. They suck. Get over it.

• There's never enough time to do it right, but there is time enough to do it again. That translates to: just get it done; fast and cheap. Give your manager something to crow about. Let someone else clean up the turds.
• Image is more important than value; Presentation is more important than content, especially if you're anywhere near DC or Manhattan. Nothing succeeds like spin.
• Winning is everything. Nobody remembers who came in second, and winners can get away with anything.
• People are expendable resources. Employers know that a better, cheaper, faster "you" will come along soon enough. Until then, you're just taking up space and consuming valuable resources.
• The HR department exists to protect the company from employee lawsuits. They are not your friends, they are not your representatives. Their only purpose is to collect a record of your activities that can be used against you if you dare speak ill of the company.

WHAT!?

[Anonymous Coward]

This is insanity! Bloat that code! Abstract those abstractions even more! Pile on the software! More! More! Then blame the hardware when it's slow!

Otherwise how can you keep creating jobs?

Software = planetary cancer. It will keep consuming resources until the host dies.

Awfulness...

[Ironpoint]

There is tons of awful, awful people working in programming. Fakers, posers, kiss-asses, sociopaths, narcissists, liars, and otherwise generally crappy people.

Good advice

[Profane MuthaFucka]

Snoop around in the shell histories of any account on the system you have access to.

Look at how other people do things. Some of them are smarter than you. Some of them are more experienced than you. Some of them even have some good porn you can snag.

Re:xargs mapreduce?

[Anonymous Coward]

Yes, obviouslly. But still, some one would use a distributed multi machine solution to do something that xargs could easily do on one machine??? That seems like extreme over engineering. If some one is really using hadoop to grep through a list of files... They have serious problems that learning about linux commands wil not solve

What you code, not how you code

[Anonymous Coward]

1) get requirements from those that will actually use the software (not their boss, not your boss, not the bean counters)
2) specify the requirements
3) document the specification
4) code to the documentation
5) test to the documentation
6) deliver software that works and does what the users want and can be compared to human (user) readable documentation
7) goto 1

My tips

[cowdung]

1. Don't over-engineer.
2. If the code is no longer used remove it.
3. Write defensive code (assertions, check arguments, etc..)
4. Think of how you're going to debug it in the field (use logs)
5. Write unit tests (Unfortunately I don't do this enough)
6. Think about error handling right from the beginning
7. Think about security right from the beginning

Three Laws of Software Development

[elmartinos]

Some years ago I had these three laws of software development:

• A developer must write code that creates value.
• A developer must make their code easy to maintain, except where such expenditure will conflict with the first law.
• A developer must reduce their code to the smallest size possible, as long as such reduction does not conflict with the first two laws.

I still believe that these rules are the essence of good code.

Don't write a program called "test"

[Walles]

Don't write a program called "test" and then try to run it by typing "test" at the console.

Doing this can cost you about half a day of printf() debugging because the program will work when you try running through GDB.

At least I did this only once :-).

Re:until you find out why Python doesn't do the jo

[daveime]

because nokia and intel are focusing on Meego which will be released for N900 aswell.

Citation please, for my piece of mind.

Design Documentation: Find The ESSENCE.

[Anonymous Coward]

( concept somewhat stolen from "Corps Business: the 30 MANAGEMENT PRINCIPLES of the US Marines" by David H. Freedman, Forbes senior-editor )

If you don't know the INTENT of the work, how are you going to know if
a) it's the right intent
b) you're solving it the wrong way, or
c) it's overwrought?

This *doesn't* mean one has to over-detail one's documentation...

but documenting the Intent is part of test-first, isn't it?

E.G.
This (whatever) is for PRESENTING the INFO needed FOR PURCHASING,
when UPDATING AN ORDER
and NOT at any OTHER TIMES,
while PASSING-THROUGH their UPDATES/amendations, to the current-orders (whatever).

Documents the Intent, isn't over-detailed *for its level*, and means if someone's reading just these comments, then they can get a GOOD overview, *quick*.

If people won't solve what the intent is,
then I'll bet they are producing buggier-than-necessary code.

Steve McGuire, was it? "Code Complete"...
Don't let bugs IN!

Wait, what?

[thePowerOfGrayskull]

I read the one quoted in the summary, and while it is definitely applicable to programming -- none of the examples were programming examples. Then when I saw this one, I started questioning...

For example, if you are doing web crawling, and you have not saturated the pipe to the internet, then it is not worth your time to use more servers. This guy got me thinking about it, he's doing "Large-scale HTTP fetching" in Clojure. He talks about parallelizing with some queueing silliness, but never mentions how much data is moving down the pipe on any one machine. If you have a 100 megabit connection to the internet, and your fetcher is using 700 kilobits, then figure out why your fetcher sucks. )

Erm -- if your process is not taking up all of the bandwidth, AND you have available CPU, grabbing more sites concurrently would use some of that additional bandwidth. I don't know about this specific example, but generally speaking you might be able to max out your CPU without maxing out your bandwidth depending on the type of processing you were doing.

Finishing the article solidified my conclusion. The author ranted about how nosql sucks (not really a programming thing), how to monitor a process (useful, but not programming - because he's not telling you how to do it within a program), that hardware matters for performance (erm, duh...), and then some mumbling about event loops done by taking a quote out of context without really explaining what he was talking about.

(As a side note, I was talking about that post with Milo's prolific systems administrator, and we could not figure out whether the author was an incredibly elaborate troll or just a run-of-the-mill idiot.

Hmm - I'm sure there's a law similar to Muphry's for this scenario...

There is only one thing you should know...

[Anonymous Coward]

When writing code the only thing you should be doing is generating a list of reasons why everything you are doing is not only wrong but total shit.

If you stick with it long enough you may learn and perhaps even somewhat improve the quality of whatever your working on.

Writing software is no fun.. I only do it because I get paid. Dealing with absurdly inflated ego's is also no fun. No chicks in the office also no fun.

Re:There is only one thing you should know...

[euroq]

Wow, your job sucks! I love writing software. I don't have to deal with inflated egos, because the teams I work in are run well and not subject to any one person's whims. And "no chicks in the office" is a totally different subject, but you should have other places in life to see chicks.

MOD PARENT UP

[cyclomedia]

From someone who works where the coding "standard" is indentation set to 2 and spaces it's a nightmare. If it were tabs then your indent size would be up to you, but nooooooooooo.

Re:Some tips from a C++ guy.

[Anonymous Coward]

Just great. Become another programmer who'll call malloc() and memcpy() where a constructor and a copy constructor should be. Become another programmer to introduce buffer overruns for lack of familiarity with C++ strings. Become another programmer who thinks toString() methods are a good idea. (Admittedly the latter applies to programmers who have heard of Java and don't understand C++ streams.)

Comment First

[bill_mcgonigle]

Something that's worked pretty well for me: write your comments first.

That is, describe with comments what each step is going to do logically. When your logic is sound, fill in the code.

Now, realistically, I'm not quite smart enough to get it right all the time before I put the code down (writing the code sometimes shows my logic errors). But then, engaging in a bit of revisionist history, in the same voice as the original comments works well.

I only 'comment the [actual] code' when I'm forced to write some kind of code that is ugly for performance reasons or it appears to be non-intuitive after being written.

Use a thesaurus

[tomhudson]

Yes, comments are good, but NOTHING beats a good self-descriptive variable, struct, class, enum, function or typedef symbolic name.

Not the "Hungarian notation" crap either - that's too strongly tied into the underlying type, and can cause confusion when you change the underlying type but not the symbolic name.

If you can't come up with a good descriptive name, it may indicate that there is something wrong with your underlying code - that it's either trying to be a "god class" or a "spaghetti monster" or some such.

Nothing worse than seeing test1(), test2(), test3(), etc., along with comments telling you what each test is. Every time you see it, you have to refer back to the comments. Stupid, but people still do this. "Oh, I'll clean it up tomorrow." Tomorrow never comes.

Re:Use a thesaurus

[ultranova]

Tomorrow never comes.

Isn't that a bit harsh, no matter how bad the code he made is ?-)

Re:Use a thesaurus

[jd]

Ok, so we all seem to have learned the basics of Computer Science:

• Indent appropriately to mark self-contained blocks
• Braces/begin-end tags should be placed to clarify what is intended by the indentation, so as to provide a sanity-check
• Comments should be short but clear
• Symbols should describe what they do, not how they do it (self-descriptive but abstract)

Then there are the remaining rules:

• Do not pass parameters in a direction they don't need to go in
• Modularize as much as possible - compilers can glue the bits together better than your brain can rip them apart
• One procedure/function/macro, one distinct operation - if it needs to do more, it needs to be divided up
• If you're not validating the input, state your assumptions in the documentation
• Validation is cheaper than an exception, exceptions are cheaper than signal handlers, signal handlers are cheaper than a BSOD on an aircraft computer

Re:Use a thesaurus

[tomhudson]

Modularize as much as possible - compilers can glue the bits together better than your brain can rip them apart

I would disagree with this one. Like normalization, you can carry it to too far an extreme, even to constructing individual classes for every trivial operation, when they form a natural grouping, or could be better handled in a single function, no classes needed.

Classes as necessary, but not necessarily classes. Normalization as much as is natural in the context, and no further. People sometimes get carried away - I've certainly been guilty of it on at least 2 occasions :-)

Re:Use a thesaurus

[jd]

To me, modularizing involves combining those things that are related, encapsulating them properly in a single unit and abstracting out all the stuff that simply doesn't matter. Ok, let's say you've an object-oriented program and you discover that a whole bunch of classes all do some set of operations in common but are otherwise unrelated. Well, to split out those operations into a sub-class is only proper encapsulation if those operations describe an object. If they do not, it is not the correct abstraction. In such a case, you'd probably want to look at macros, patterns or templates, depending on the language and also on what the operations actually do model.

Re:Use a thesaurus

[tomhudson]

I'm not necessarily disagreeing with your analysis - just the "as far as possible". Some people will give a whole new meaning to the phrase. I'm still waiting for some "genius" to create individual classes for each letter of the alphabet - but it wouldn't surprise me if someone already tried that, and it got incorporated as a "feature" in some "enterprisey" thingee :-)

Re:Use a thesaurus

[jd]

Ok, that's fair enough.

Commenting benefits everyone else

[rclandrum]

I am not inclined to waste much time agonizing over things that will help my replacement do his job better than I did mine. I assumed that the purpose for the "tips" was to help *me* be a better programmer and avoid the mistakes of others. To that end, here's my suggestions, derived from 35 years of experience in the real world:

1. Programmers always bitch about deadlines because they insist on coding to some puritanical design guidelines and attempt to use all the latest features of an OS. I never, ever worried about deadlines and always met them because I trained myself to size my coding efforts to the time available.

2. In the end, when the clock runs out, the *only* thing that counts is whether you have something that works that you can show the client. Excuses will not keep you employed.

3. As you learn new coding techniques in the course of a project, resist the temptation to revisit your earlier code to "clean it up" - at least until you have archived a complete working version of the application. Only then - and only if you have time - should you revisit and improve your code.

4. Never, ever get defensive about your code. There is always someone that can code more stylishly than you or can comment better or knows the API by heart. Instead, accept suggestions with grace, be a good team player when you are part of a team, and concentrate on never missing a deadline. Better to have working ugly code, that be pretty and and a day too late.

5. Always jump at the opportunity to be a part of something new, because you will expand your knowledge, your resume, and (usually) your wallet.

6. If you are a team leader, be generous with your time and knowledge, work one-on-one, and avoid organizing meetings that have no concrete agenda and time limit.

7. Have a life and a hobby that does not involve computers. It is important to be able to decompress.

Re:Don't code in C

[EsbenMoseHansen]

You need to learn to use libraries :P

hmmm

[luis_a_espinal]

Eh, isn't that the basic difference between a programmer and an engineer?

A good programmer is a good engineer. A bad programmer is not. The false dichotomy between programming and engineering is nothing more than a clutch and a sorry excuse for having incompetent programmers around.

one hell of a strawman

[luis_a_espinal]

"What's wrong with stored procedures? 1) It means we need to support two languages instead of one, with all that that entails (a proper debugger, expert knowledge, etc)"

Wrong! *you*, not we. Repeat again: data is company's property, not the application's. You can bet your company data *will* be accessed by more than one application at more than one age if it's of any use (if it isn't the app shouldn't be programmed to start with). It will be the applications' side the one that will add supporting needs for more than one language, debugging, expert knowledge while the data will still use the same SQL language, the same tools and the same domain experts.

"2) Stored procedures cannot do ALL business logic"

True. Neither it's expected to do so. Data-bounded logic should be retained at the data engine level if only to be sure every accessor will comply to it. Your specific app logic can and should be developed within the application. Sometimes it takes some cleverness to find the boundaries, usually not.

"3) Because you don't want to have to deal with Oracle's horrid error messages anymore than absolutely necessary"

This is an argument *against* your position, not supporting it. The way you are free from Oracle's whatever is insuring that data managing functionality is as bound as possible to the data management engine (encapsulation, you know the concept). This way it will be the DBA's problem not yours.

Atta boy, you built a strawman that was completely unrelated to the parent post. Also, the strawman was built like a dead horse, then you grabbed your rhetorical e-dick and beat it to death with it. Pat yourself in the back and have a cigar, for it is misshum' accomplished.

shell is more secure than perl

[Gunstick]

yes

you can much more easily inject natsy things into perl or php than into shell.
Have you ever seen a shell executing data? I would need to call "sh" or "eval" to make this work.
On the other side, is perl which calls without you knowing "sh" on simple things as opening a file.

So sorry, perl is more prone to injections as shell.
Georges

Re:shell is more secure than perl

[SanityInAnarchy]

you can much more easily inject natsy things into perl or php than into shell.

Erm... how so?

Have you ever seen a shell executing data?

Uhm... yes? That's what they do. Ever heard of a shell script?

I would need to call "sh" or "eval" to make this work.

So imagine a hypothetical script in which you do something like this stupid little example:

my $cmdline = "convert $x.svg $x.png";
system($cmdline);

And in this case, $x is user input. Well, now you're screwed if anyone enters spaces, so you do this:

my $cmdline = "convert '$x.svg' $x.png'";

Now you've got a classic SQL-injection-style vulnerability. What happens if a user sets $x to "'; rm -rf /; #"? That could be more devastating than a traditional SQL injection vulnerability.

So how, exactly, is Perl more vulnerable than that?

On the other side, is perl which calls without you knowing "sh" on simple things as opening a file.

Sorry, I have to call bullshit on this. Here's the perldoc on 'open'. It tells you exactly under which circumstances the filename will be interpreted as a command. If you don't want to sanitize a pipe character from the beginning and end of your string, there's always sysopen, which is mapped directly to fopen -- if this is vulnerable, then every single program on your system is vulnerable.

If you can't be bothered to RTFM, what business do you have programming?

And again, how is this more vulnerable than the point at which you call the shell from a perl script?

Re:shell is more secure than perl

[Gunstick]

you can much more easily inject natsy things into perl or php than into shell.

Erm... how so?

Give the perl script the following filename to open: "|rm *"

Have you ever seen a shell executing data?

Uhm... yes? That's what they do. Ever heard of a shell script?

No shells execute shellcode, like perl executes perl code. Shells don't execute their parameters. Perl does (see above). Give a shell "|rm *" and it will just try to open the file "./rm *" Give it "$(rm *)" and still it's just complaining with $(rm: file not found and *): file not found. Or if the script is well written it barfs with $(rm *): file not found. Nothing is executed.

I would need to call "sh" or "eval" to make this work.

So imagine a hypothetical script in which you do something like this stupid little example:

my $cmdline = "convert $x.svg $x.png";
system($cmdline);

And in this case, $x is user input. Well, now you're screwed if anyone enters spaces, so you do this:

my $cmdline = "convert '$x.svg' $x.png'";

Now you've got a classic SQL-injection-style vulnerability. What happens if a user sets $x to "'; rm -rf /; #"? That could be more devastating than a traditional SQL injection vulnerability.

So how, exactly, is Perl more vulnerable than that?

The problem with your example is the in shell you do not need to use the clumsy "commandline=:..." construct. It rather looks like this:

#!/bin/bash
x="$1"
convert $x.svg $x.png"

You can't "hack" that.

Or let's see with the ugly command in a variable method:

#!/bin/bash
x="$1"
cmdline="convert $x.svg $x.png"
$cmdline

test run:

# test.sh "; rm -rf /; #"
convert: unable to open image `;': No such file or directory @ blob.c/OpenBlob/2439.
convert: unable to open image `rm': No such file or directory @ blob.c/OpenBlob/2439.
convert: unrecognized option `-rf' @ convert.c/ConvertImageCommand/2177.

On the other side, is perl which calls without you knowing "sh" on simple things as opening a file.

Sorry, I have to call bullshit on this. Here's the perldoc on 'open'. It tells you exactly under which circumstances the filename will be interpreted as a command. If you don't want to sanitize a pipe character from the beginning and end of your string, there's always sysopen, which is mapped directly to fopen -- if this is vulnerable, then every single program on your system is vulnerable.

If you can't be bothered to RTFM, what business do you have programming?

And again, how is this more vulnerable than the point at which you call the shell from a perl script?

With shell you don't need to read any manual or sanitize fancy characters. You just write your program. If perl would have been programmed with security in mind, the default (i.e. empty mode) would have been a forced '<' and if you want to call a program, you should need to make it '<|' or '>|' or anything funky like that. But no, it's too user friendly and the many scripts around who read files with "open MyFile,$filename;" are a common occurance.

A random search skipping the links from perl.org brings up quickly a first instruction example how to read a file. http://www.pageresource.com/cgirec/ptut14.htm See, no '<' here, and so beginners immediately write bad code. Irony is that this is even a webpage for how to write CGI scripts.

And we don't talk about my programming business but the thousands

Re:shell is more secure than perl

[SanityInAnarchy]

Shells don't execute their parameters.

Sure they do:

echo `rm -rf /`

In what way is that not equivalent to your script?

Never mind that, once we sanitize all the variables, we still have to deal with the behavior of every single command we might call.

The problem with your example is the in shell you do not need to use the clumsy "commandline=:..." construct.

However, you do need something like that from any non-shell language, unless you're suggesting that all development should be done in the shell.

#!/bin/bash
x="$1"
cmdline="convert $x.svg $x.png"
$cmdline

Try executing that with arguments which have spaces. Now imagine a few which have dashes, or slashes, or...

I don't know convert's arguments by heart, but I bet I could find something exploitable in there, given free reign to pass any arguments I want, which is more or less what you've provided here.

With shell you don't need to read any manual or sanitize fancy characters.

I've just demonstrated that you do. Both. Without reading the manual, you can't know that 'convert' is safe to use in that way. Without sanitizing characters, you can't know that you can use $x the way you just did.

If perl would have been programmed with security in mind, the default (i.e. empty mode) would have been a forced '<' and if you want to call a program, you should need to make it '<|' or '>|' or anything funky like that.

How would that be any more secure? Wouldn't it make much more sense for these to be separate calls?

A random search skipping the links from perl.org brings up quickly a first instruction example how to read a file.

And covering buffer overruns in a first introduction to C wouldn't be great, either. But, I'm sorry, the manual is there.

And we don't talk about my programming business but the thousands of people who don't care or think about those problems. "Oh it works, then it must be ok".

Those people will generate security hazards no matter what language they use -- as you seem to have already done with the shell.

Re:shell is more secure than perl

[Gunstick]

It starts to become pointless to discuss with you. Come on...

echo `rm *`

is NOT the same as

test.sh '`rm *`'

If you don't believe me, do first your tests.

Also arguing that convert may or may not have unexpected behaviour does not differ if you use perl or bash to call it. We discuss the problematic of perl ADDING insecurity compared to shell.
Because if you have that `rm *` parameter in the perl program, you will actually execute it, but not in shell!

The equivalent perl construct written in shell is:

eval $command

You never do that in shell, as

$command

works as well. And then you are safe.
If you have read my previous post, I *did* execute it with spaces. Rememver the

test.sh "; rm -rf /; #"

It did not execute anything. Just convert did not know anything about the -r parameter. In perl it will have executed. Sanitizing needed for `, and for $ and for | and for > and, and, and. The list of problematic characters is long. I once tried to call convert with allowing to pass some convert parameters by the user too. To do it safely, it was a huge headache. So I resorted to only allowing very specific parameters.

In shell, as well as in perl, you need to know convert's parameter list. Either kill any - in the parameters or simply pattern match away the specific option which makes convert dangerous (I did not find any, looked for writing a file). But (repeating myself) in shell you do not need to know anything about something else (`$|>...) which has nothing to do with convert.

More specific case: awk
That's probably the closest you can get to inadvertently execute code (yeah I help you there in your arguing ...).

calling awk with

awk '/pattern/ {print '$variable'}' is bad

but this is like you should do it and it is safe

awk -v variable="$variable" '{/pattern/ {print variable}'

Georges

Re:shell is more secure than perl

[SanityInAnarchy]

echo `rm *`

is NOT the same as

test.sh '`rm *`'

Actually, now you have an additional bit of fun:

test.sh '';rm -rf /#'

Same issue as SQL injection. If you're going to use things like exec() directly, it then becomes an issue of the security of test.sh, and not of the shell itself.

Also arguing that convert may or may not have unexpected behaviour does not differ if you use perl or bash to call it. We discuss the problematic of perl ADDING insecurity compared to shell.

When was I ever suggesting that Perl should call 'convert' directly?

However, the shell requires multiple independent programs to be used at every step, each of which has its own quirks and calling conventions. CPAN, by contrast, is at least going to have similar behavior and calling conventions, and except where individual functions add "magic" (like that 'open' behavior), you get the added security of being able to treat data as data.

Compare that to any point at which you call a shell script from something which is not a shell script -- you have to be careful, or you're going to treat data as code.

Because if you have that `rm *` parameter in the perl program, you will actually execute it, but not in shell!

Erm, where in Perl would that parameter be executed where it wouldn't in the shell?

The equivalent perl construct written in shell is:

eval $command

You never do that in shell, as

$command

works as well. And then you are safe.

Erm... If $command is constructed insecurely, you have precisely the same problem in the shell as in Perl.

If you have read my previous post, I *did* execute it with spaces.

Maybe I missed something, but I actually took your example and attempted it with spaces.

Just convert did not know anything about the -r parameter.

In other words, you're still allowing the user to pass arbitrary parameters to convert, which is still a bug.

The list of problematic characters is long.

And the solution is trivial -- use sysopen, no sanitizing needed.

Never mind that I can't remember the last time I tried to open a filename provided by user data.

In shell, as well as in perl, you need to know convert's parameter list.

I'm sorry...

You seem to be misunderstanding my position, and what I thought this discussion was about.

I thought it was about the difference between, say, calling "convert" from Perl (or PHP, or Ruby, or Python, etc) and calling the ImageMagick bindings directly.

And I can assure you, the way I use ImageMagick, there's no option for the variable containing user data to ever be interpreted by libmagick as anything other than user data, no need to audit an impressive list of options to verify security.

Unless you are seriously considering using the shell to implement a web service (which is what TFA seemed to be referring to), that is the most likely use case here -- TFA is suggesting we call "convert" from whatever we wrote our web service in, while I'm suggesting we use native bindings.

Re:shell is more secure than perl

[Gunstick]

Sorry, too much hassle to do quoting correctly here :-)

test.sh '';rm -rf /#'

Have you tried this?
This is not even passing the parameter to the test.sh and would barf your system as well if you use an equivalent test.pl
So please try your claims before using them in arguments.

There is no real exec call in shell. Even using the exec instruction explicitely, you won't execute parameters.

> call a shell script from something which is not a shell

Maybe you you meant the reverse?
Anyway, in shell you call stuff which is not a shell from the shell. And this works wery well even when you're ignoring security concerns.
If you don't use CPAN, you will call the shell from perl. At least that's what most users do: system()
I recently directed someone to use CPAN to get his work done, and the resulting perl was just an accumulation of various calls to other utilities. And we paid for that guy!
Injections galore! Transforming the perl into simple shell (what it in effect is) and the injetions are gone.
Other possibility is of course use all the fancy stuff from CPAN to do the job.

But why is everybody using system()? What is wrong with exec()?
And then you have the usage of `` inside perl which I come across quite often.

Perl has way too many places where you can build bad code.
shell has just a few.

> Erm... If $command is constructed insecurely, you have precisely the same problem in the shell as in Perl.
No it's not.
Let's make an insecure command (mind the usage of single quotes, else the example is futile):

command='cat file;$(ls > list)'
$command

cat: file;$(ls: No such file or directory
cat: >: No such file or directory
cat: list): No such file or directory

Oh dang! It did not do what you thought.

> In other words, you're still allowing the user to pass arbitrary parameters to convert, which is still a bug.
does it corrupt the system, well no. yes it's a bug (bad coding style, oh that statement I know as well from the perl world).

> And the solution is trivial -- use sysopen, no sanitizing needed.
which should indeed be the default, screw the open call, it's bringing problems. wizards should use open, beginners use sysopen. But in the real world it's the reverse.

> TFA is suggesting we call "convert" from whatever we wrote our web service in, while I'm suggesting we use native bindings.
In this I'm OK with you, if someone wants to call convert instead of a binding, he should write the webservice in a language meant to be used to call convert, and that's shell. Else he should use a language which has imagemagick bindings or don't do it at all.

Georges

Re:shell is more secure than perl

[SanityInAnarchy]

Sorry, too much hassle to do quoting correctly here :-)

Quoting is precisely what's crucial to the disagreement.

test.sh '';rm -rf /#'

Have you tried this?

No, because I value my data. I've tested it with something similar, and I'll amend it to this:

test.sh '';rm -rf / #'

If spaces are allowed, this is not any harder to implement than the original (wrong) version.

This is not even passing the parameter to the test.sh and would barf your system as well if you use an equivalent test.pl

*facepalm*

You're right, it's not, which is the point. It doesn't care what parameter gets passed to test.sh -- this is someone exploiting a commandline I've constructed, and even if test.sh blows up, they don't care, they've managed to execute a 'nuke everything' command.

And invoking a separate Perl script is no better than invoking a separate shell script. The point is to avoid invoking a separate process at all.

So please try your claims before using them in arguments.

The fact that I've just tried it and the difference was a single space suggests that my argument was valid, just very, very slightly imprecise. Do you really care that much? If so, maybe I should fire up a VM and present screenshots proving that the command I've provided now will, in fact, nuke every file the current user has access to.

There is no real exec call in shell.

...what?

Even using the exec instruction explicitely, you won't execute parameters.

...what? What are you talking about? The above example doesn't rely on parameters being executed. Go ahead, try it, I dare you.

Maybe you you meant the reverse?

Nope, though "shell script" is irrelevant here; the vulnerability is the shell itself whenever I try to system() or exec() anything.

If you don't use CPAN, you will call the shell from perl. At least that's what most users do: system()

Right. And this presents all sorts of opportunities for abuse. Of course...

Anyway, in shell you call stuff which is not a shell from the shell. And this works wery well even when you're ignoring security concerns.

Your shell example using a cmdline variable is vulnerable, and it's not uncommon to want a variable like that.

I recently directed someone to use CPAN to get his work done, and the resulting perl was just an accumulation of various calls to other utilities.

"Other utilities" meaning CPAN modules, or non-Perl stuff?

And we paid for that guy!

How quickly did he get his stuff done? Would you rather pay someone who knows how to glue together the appropriate existing code (CPAN or otherwise), or someone who reinvents the wheel at every turn? Even if they took the same amount of time, I'd much rather maintain something which uses proven libraries than something which reinvents the wheel, badly.

Injections galore! Transforming the perl into simple shell (what it in effect is) and the injetions are gone.

Yes, if what you want is a shell script, I would agree that the shell is usually the best way to do it. It is possible to avoid trivial injections by using the multi-arg version of 'system' (at least in Ruby), but that's a hassle.

But why is everybody using system()? What is wrong with exec()?

Erm, where would you ever use exec when system would work, unless Perl has perverted those calls well beyond their original reason?

does it corrupt the system, well no.

Do you know that for certain? How familiar are you with the arguments convert can take? I count thirteen pages, an

Re:shell is more secure than perl

[Gunstick]

* quoting: doesn't slashdot have a quote feature? Could not find one except typing the <quote> tags. Awkward.

* your example gets it wrong. It's a social engineering attack, not an attack on test.sh

All the following lines are equivalent:
test.sh '';rm -rf / #'
cat '';rm -rf / #'
ls '';rm -rf / #'
perl '';rm -rf / #'

Why? Because you call the command with an empty argument, then you call the rm. So the rm is executing in your interactive shell, not in the program you want to demonstrate to be exploitable.

You are mixing up typing in a command into an interactive shell and calling a shell with a parameter. In shell this is an instersting difference.

So let's try this.

Take a the following test.pl which we want to exploit.

#!/usr/bin/perl
print "parameter 1 is:".$ARGV[0]."!\n";

We would call it how? Probably like this:

$ ./test.pl '$(echo hello)'
parameter 1 is:$(echo hello)!

but not call it by any of these because the parameter gets executed before even the perl script runs.

$ ./test.pl $(echo hello)
parameter 1 is:hello!

$ ./test.pl "'$(echo hello)'"
parameter 1 is:'hello'!

$ ./test.pl "$(echo hello)"
parameter 1 is:hello!

$ ./test.pl ' ';$(echo hello) #'
parameter 1 is: !
hello: command not found

Oh... I don't believe a simple print in perl is exploitable. So the 4 lasts tests are not testing test.pl but the surrounding shell.
only the first, or similar setups, are valid tests.

I hope you see the difference between the interactive shell and the called script, be it perl or bash or anything else.

* the guy's other utilities:
yes, he did not use CPAN but system calls to run other unix utilities. Like calling mailx to send an email (using a user supplied email address, wow!) and he took 3 weeks to get 1 day worth of work done.

* calling git:
well there you need to be careful how you call it. Not knowing ruby I don't know how it's calling external commands. If it's doing an equivalent of "sh -c $command" you need to sanitize arguments.

* webservice in shell
Well not really a webservice, but what about a chatroom?

* dang! Hmm, no, it doesn't seem to, though it would in every other language I've used
This is the cool feature of shell, it does not in fact execute strings which look like they may break your system.
I wrote a file up/download cgi script where any character is allowed in the filename, except / and \0
All written in shell. I was unable to get it to run a program, send me system files or overwrite files.

So that's exactly my point, compared to any other language, the shell surprisingly does not do any of the classic exploit techniques.
This is why I call shell more secure. But of cause not perfect. Just better in security (and not in features or speed ...)

Re:shell is more secure than perl

[SanityInAnarchy]

doesn't slashdot have a quote feature? Could not find one except typing the <quote> tags. Awkward.

Not particularly. Also, there's <ecode> for sample code.

Because you call the command with an empty argument, then you call the rm. So the rm is executing in your interactive shell, not in the program you want to demonstrate to be exploitable.

In other words, I am exploiting the shell.

In other words, anywhere that has a "system"-type call, where a command can be assembled as a string (rather than an array of string arguments), is a potential avenue for abuse. What about my example implied the shell must be interactive?

You are mixing up typing in a command into an interactive shell and calling a shell with a parameter.

Again, no, this is not "typing in". Try the following in Ruby:

arg = "'; rm foo; #"
system "echo '#{arg}'"

The same in Perl:

my $arg = "'; rm foo; #";
system "echo '$arg'";

The same in Python:

arg = "'; rm foo; #"
import os
os.system("echo '"+arg+"'")

I could continue, but I doubt you'd find PHP to be any better about this, and I don't want to touch it. Point is, blame the language or the shell, but either way, executing shell commands this way is a bad idea. Executing them the right way is a hassle, unless you're already in the shell, but unless I'm missing something, implementing a web service in the shell is a hassle.

And that's before we start examining commandline programs for security.

I hope you see the difference between the interactive shell and...

The only real difference between an interactive shell and a noninteractive shell is the way the prompt is displayed. That's why shell scripts are so cool to begin with -- if you know how to type it into an interactive shell, you probably know how to make a basic script, and as your scripting abilities improve, those translate into increased proficiency with the shell.

Yes, I understand the difference between a command executed in the context of a shell and commandline arguments passed to a shell script ($1, $2, etc).

yes, he did not use CPAN but system calls to run other unix utilities. Like calling mailx to send an email (using a user supplied email address, wow!) and he took 3 weeks to get 1 day worth of work done.

Ew.

Not knowing ruby I don't know how it's calling external commands. If it's doing an equivalent of "sh -c $command" you need to sanitize arguments.

That depends. The same method can be used to do either way:

system "echo #{myvar}"
system 'echo', myvar

The first is vulnerable unless I trust myvar. The second is safe, assuming I trust ENV['PATH'], and if I don't, it accepts absolute paths as well.

Well not really a webservice, but what about a chatroom?

Either way.

By the way, I've seen games like tetris written in Bash, so I don't doubt it can be done. I just doubt it's sane -- it strikes me as similar to hanoimania.

This is the cool feature of shell, it does not in fact execute strings which look like they may break your system.

Neither does any other language I know.

As I said, it's the border between the two. I can combine strings as many ways as I like, and they won't jump out and eval themselves -- but if I 'eval' an untrusted string, I'm hosed. If I stay entirely within the shell, I can combine strings any way I like (though it seems slightly trickier), and they won't jump out and eval themselves -- but if I 'sh -c $foo', I'm hosed.

And of course, there are oth

Re:shell is more secure than perl

[Gunstick]

The same in Perl:

my $arg = "'; rm foo; #";
system "echo '$arg'";

And the same in shell

arg="'; rm foo; #"
echo $arg

I even execute it:

$ ls foo
foo
$ arg="'; rm foo; #"
$ echo $arg
'; rm foo; #
$ ls foo
foo
$

Oh, nothing happened. Exactly the expected behaviour. Shell is safe, all other languages are not.

unless you're already in the shell

That's what I'm talking about. Do it all in shell or don't call external programs. So yes, I have gone the way of implementing CGI scripts in shell. Even calling perl from shell, LOL. Shell is built to call external programs. Other languages are not.

The only real difference between an interactive shell and a noninteractive shell is the way the prompt is displayed

The difference being that the user's input (you typing) is directly going to the prompt, whereas the user input in a shellscript is only able to enter via parameters. And this makes a huge difference. As above, first assigning arg a value and then echo it works fine. But merging both into a single instruction is risky if you don't do it right (don't leave out the double quotes you used in the assignement in the 2 line version).

system 'echo', myvar

Oh, cool. Did not know that. It's a little hidden in the ruby manual.

Well, define "written in shell." Does the shell support sockets? Not that I'm aware of -- you'd at least need netcat.

Apache calls the script via cgi environment. The socket is stdin/stdout. The parsing of the cgi variables is done via an awk script. This has just logic against injections of a single quote because the awk result is then fed to an shell eval to give the shell all user variables. Yes this works!
All the rest is quite simple, just write the webpage to be sent back to stdout and you're done.

I'm less interested in what it's like for a bad programmer

There are thousands more bad programmers out there than good ones. And I have the feeling that the ratio is all the time going down. If languages would be more complex to do the bad system call then the good one, this would already improve general computer security.

Re:shell is more secure than perl

[SanityInAnarchy]

arg="'; rm foo; #"
echo $arg

The same exact thing happens in every other language I mentioned. In Ruby:

arg="'; rm foo; #"
puts arg

I'm not sure why you're so passionately tied to the shell, but you're demonstrably wrong to make this claim here:

Shell is safe, all other languages are not.

Can you show me a language which isn't safe with that particular example?

If you insist I actually call /bin/echo, I can do that, too, as I demonstrated:

system 'echo', arg

Works, exactly, naturally, the way you'd expect -- and anyone who's been doing Ruby for very long would expect APIs not to require manual string concatenation or interpolation. This usage is right there in the docs, right exactly next to the docs on the "unsafe" version of system.

However, I don't think your 'echo' command above is safe:

arg='-e \n'
echo $arg

Now, I don't know the correct escape codes offhand, but if I recall, it is possible to gain control of a terminal and execute arbitrary code that way. (If you doubt that, just 'cat' a typical binary file, chances are, you'll accidentally execute a bunch of gibberish commands.)

Shell is built to call external programs. Other languages are not.

You are making a claim about all other languages, yet you admit to not knowing one I've mentioned here. Are you really prepared to say that there is no language which is built to call external programs other than something compatible with /bin/sh?

The difference being that the user's input (you typing) is directly going to the prompt, whereas the user input in a shellscript is only able to enter via parameters.

Now you've changed the topic. I said "interactive" and "noninteractive", I didn't mention shell scripts.

It's a little hidden in the ruby manual.

Are you joking?

ri Kernel#system

On a recent (1.9) version of Ruby, I get this:

system([env,] cmd [, arg, ...] [,options])

along with this description:

The arguments are processed in the same way as for +Kernel::exec+.

Hmm, let's try that:

ri Kernel#exec

And we find:

If optional arguments, sequence of +arg+, are not given, that argument is taken as a line that is subject to shell expansion before being executed. If one or more +arg+ given, they are passed as parameters to _command_ with no shell expansion.

That's very near the top of that page. It may be "buried" in the manual, but it's no deeper than 'man' in the shell.

There are thousands more bad programmers out there than good ones. And I have the feeling that the ratio is all the time going down.

And a bad programmer will find a way to write bad code in any language. The shell doesn't make this particularly easier -- how do I loop over a collection of strings which may contain spaces?

You can write COBOL in any language.

If languages would be more complex to do the bad system call then the good one, this would already improve general computer security.

By a small amount, perhaps. But it wouldn't change the basic fact that a bad programmer is going to introduce tons of security flaws regardless.

The solution is not to coddle the bad programmers. The solution is to make the right approach obvious and easy for the good ones.

Re:shell is more secure than perl

[Gunstick]

sorry, no reply from me because:

Your comment has too few characters per line (currently 36.1).

and then

Filter error: Please use fewer 'junk' characters.

Come on!!!!
I now hate slashdot

Re:shell is more secure than perl

[Gunstick]

because slashdot does not like me, here is my reply:

http://pastebin.com/GPtSNUS8

Re:shell is more secure than perl

[SanityInAnarchy]

a good rule is to put always double quotes around all your parameters.

I don't see how that's improved over other languages, though. A good rule in Ruby is to always make system calls using the multi-arg version of system. A good rule in any language with SQL bindings is to use bind variables instead of string interpolation.

Actually, I find those options to be better, because it's a way of doing things that I have to remember, not that little syntactical thing of remembering quotes. It's why I much prefer bind variables to having to remember to quote and run some escaping routine around user data.

$ arg="count=1 of=foodd"
$ dd if=foo "$arg"
dd: invalid number `1 of=foodd'

Right. But you do still have the case that the user can specify an argument there -- of=/somewhere/bad comes to mind.

For me interactive is a shell with a prompt and non interactive is one without prompt, and then that's a running shellscript.

Well, no. For instance, the shell as invoked by Perl/Ruby/C when using an insecure "system" function is noninteractive, but not necessarily running a shell script -- which I interpret as a self-contained program in a separate file.

I made a quick search on the net and only found a quite small entry for system call. I wonder if everybody programming in ruby is aware what "is subject to shell expansion" has as consequences.

Possible. I'm not sure it's important -- most Rubyists tend to avoid invoking other processes without good reason.

What to do when there are only bad programmers available?

Teach them to be good programmers. If all are unwilling or incapable to learn, find a new line of work -- or better yet, withdraw from society, because someone's embedded system bug is likely to kill you.

I've gone back to school, and I can tell you that there are plenty of good and bad -- definitely plenty of students who genuinely do want to learn, and care about getting it right.

CDMA, phone/plan bundling, and

[tepples]

A lot of people apparently don't know what they're talking about because they can use the exact same plan on a smartphone

In the United States of America, it's hard to buy a smartphone without buying a plan. Most people don't know to look online, and online stores charge a restocking fee. And even if you can find one, the price on Google Product Search for an unlocked Android smartphone is often twice that of an otherwise comparable iPod touch. For example, a Samsung Galaxy S is $600 while Apple's iPod touch 4 is $300.

ImageMagik is nice, but ...

[WindShadow]

I got started doing CLI image processing with netpbm decades ago, and the approach is still valid. It was designed to work as a CLI program, and is a great solution for batch processing. In addition, you have libraries to manipulate data in the native format, so it's possible to add some functionality which you need by just a small program doing the operation, with all the i/o and many of the transforms done for you.

Un-slashdot-worthy

[euroq]

I'm not being a troll, I honestly mean this: those tips/experiences aren't that great. How the heck did someone think this important enough to be put on Slashdot? It's oriented towards someone's very specific experience, which for the most part doesn't translate well to other people.

Re:Un-slashdot-worthy

[neminem]

I'd mod you up, but I decided I'd rather comment that I agree with you - "Programming things I wish I knew earlier" sounded like it would probably be an interesting article. Except, why was it filed under "Linux"? Oh, that would be because it should really have been, "Linux programming things I wish I knew earlier". As a Windows programmer... I admit "don't reinvent the wheel" is excellent advice, and I'd been guilty of that myself a time or two, just out of college, but this article uglifies that advice into "don't reinvent [specific programs x, y and z]". Too bad.

Advice

[Xiver]

The trouble with advice is that you don't know its good until you don't try it.

I see lots of posts here telling programmers they should do this and do that, but the truth can change quite a bit for each developer depending on their situtation. Some software will have millions of lines of code and countless developers who, over several years, have written commented and rewitten the code. The lessons learned by developing, testing, and maintaining that codebase will be completely different for a developer who maintains a codebase who only a handful of developers have ever touched.

Here is my advice.
1. Keep it simple. If its not simple you didn't spend enough time thinking it through. If you can barely understand it now you'll never be able to figure it out later.
2. Code Comments should not explain why or how, only what the intention of the code is. The why is irreleveant and the how will change.
3. Revisit your code from time to time even if you don't have too. Pick what you remember as your best and worst and take a look at them again.
4. Don't stop learning, ever. Keep expanding your knowledge and don't stick to just one language or platform. Having a specialty is fine, but much can be learned from examining other areas.
5. Listen to the advice of senior developers.
6. Make sure you, your superiors, and your clients are all crystal clear on the requirments before any coding begins.
7. There are exceptions to every rule and times when its purdent to break them, especially these.

Do not reinvent the wheel... not so easy for me

[mrflash818]

"Don't use Hadoop MapReduce until you have a solid reason why xargs won't solve your problem. Don't implement your own lockservice when Linux's advisory file locking works just fine. Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job. Modern Linux distributions are capable of a lot, and most hard problems are already solved for you. You just need to know where to look."

Ah, this last sentence is what made it a challenge. You get out of school, where most assignments were to code up projects from scratch, writing it all yourself.

Now you are a professional, and for me there was so much preexisting stuff _that_I_was_unaware_existed_.

I knew how to program using the languages and libraries of the language (C, C++, Java). I learned about Design Patterns. Yet it does not seem that there is a way for people to find reference material, or perhaps a catalog, of preexisting libraries or modules, or even other programs that can be used.

I now know Perl has one (CPAN). I didn't know about it when I was new.
I know PHP has one (PEAR). I didn't know about it when I was new.
etc.

So for the article's point about knowing there are things pre-built to use, like Image Magik: how does the programmer community go about informing each other about creating and using such repositories of preexisting programs, Freshmeat?!

Example: Maybe I can study the source code of Pidgin (I have) and see they are using gtk, and have a library in libpurple of how the program can interact with IM clients.

But, are we then supposed to have and find the time to analyse lots of programs, case-by-case to then ferret out the existance of things we can use that preexist?

I do not think I ever had that much time, unless I never get sleep, and never see my family, and work 20hr days... forever?

I'll be modded down for this, but...

[Money for Nothin']

...don't be such an early-adopter that you invest so much time in something (like Linux) that it's not (yet) profitable.

I remember seeing Linux consultancies go down the toilet 10 years ago because there wasn't enough of a market for them; perhaps now they are. But I shouldn't have spent so much time, at the time, learning Linux and FreeBSD, when there was piles more money to be made in the Windows & .NET world.

That said, the grass is starting to look greener on the Linux side again, in large part because Google is making it relevant both server-side and on mobile devices, and in part because programming still seems like fun there. In the MSFT world, it's all about slavish devotion to sometimes half-baked tools and half-baked requirements. There's more of a scientific & engineering culture in the Linux camp than the MSFT camp... (And I say that with deep personal and professional experience on both sides.)

Maybe I just want to be a coder in my personal time, and not also my professional time...

Yes

[Anonymous Coward]

95% of the work programmers do is like designing a building on the blueprints. The only part that is *at all* like actually building a building, is when you press "Compile" at the end, or when you package your app up and burn it on a DVD and ship it to customers in a box. The programming part is all design.

The source code IS the design.

Comments should just explain the assumptions, references for the algorithms, and other miscellaneous stuff that is not obvious to someone who is simply reading the code.

Write your code as simply and clearly as possible. Choose names for functions and variables carefully. Make sure not to give them misleading names. If the code has evolved and the old name no longer communicates clearly what its for, then refactor the name. Abbreviations should only be used if there is consensus about what it means (e.g. a prefix for a certain library that is used on all of its macros and functions)

Resist the urge to do "clever" hacks, unless they are badly needed for some reason (like performance) and in that case you better explain your "clever" code with a comment explaining WHY it is the way it is.

Tab indentation is problematic

[Envy Life]

From someone who works where the coding "standard" is indentation set to 2 and spaces it's a nightmare. If it were tabs then your indent size would be up to you, but nooooooooooo.

Tabs for indentation are a good theory, and in the context of a single editor, work fine. However step out of that context and it's a roll of the dice.

Go ahead and set up your editor to the preference of your choice... lets say 3 for this example. Save your code. Use "cat" or "type" from the command line and what indentation do you see? Typically 8 space indents. Bring it up a different editor and it's a crap shoot. May be configured for 2 or 3 or 4 or just default to 8. Maybe you want to print it from a box that you don't have "your" editor configured for it. Again, roll the dice. Want to do a diff on the command line? Probably 8 spaces again. Diff in a tool? Again, roll the dice.

You put indents with spaces then there's no question that no matter what computer, what application, you will see 3 space indents. Write new code with whatever you want, modify exiting code with whatever it was written in. Just be consistent. Tabs don't give you cross-tool consistency.

version control

[Crispy Critters]

I picked the easy, lowest common denominator and use CVS. Advantages for me: 1. It is installed on the machines that I use but don't control. 2. It is adequately compatible with emacs (adjust for whatever your preferred coding environment is) 3. Remote access (with no cvs server required) is trivially available over ssh.

CVS (or any other version control system) lets me create a copy of my analysis routines as they existed at a given date, e.g. the month before a certain conference three years ago. Thus I know that I can recreate exactly whatever I did then. I feel more free to rip out and rewrite things knowing that the old version is never lost. Can the same thing be done by archiving all old versions of routines, keeping careful changelogs, and (depending on your setup) frequently running rsync? Of course, but not as conveniently.

In practice, I give code to people I work with and when they complain about bugs, I fix them. I believe that on the local system, cvs checkin and checkout access is governed by unix file permissions, no server required.

People who don't work in this sort of environment don't understand the differences in mindset. Not many people want easy access to code and documents created over a decade ago, but I find this to be common in scientific work. Why do some scientists like TeX and FORTRAN? Because they can still use their work 20 years later. I figured that CVS would also be around forever.

I won't argue with anyone saying that there are better solutions, but using CVS was easy and good enough.

I'll bite :)

[rsilvergun]

I couldn't disagree more. Getting into the nuts and bolts of coding first thing and reinventing wheels isn't the way to start out. You need to learn basic code organization and good practices first, and it's hard to do that when you're spending all you're energy an memory management or what have you.

Put another way, you can take the 'learn how stuff works' approach you're suggesting as far back as you like. If it's great to know how printf & link lists work, wouldn't you want to know how your C code breaks into assembly? Then why not take it a step further to machine language, and then how about the processor and registers? What about the electronics. I saw a /. story about cosmic rays affecting memory, shouldn't you know how to troubleshoot that? See, there's really no end to it

Of course, part of me is also responding to changing times in the US. It's damn near impossible to get the sort of cushy corporate programming job where having general skills pays off. Yeah, if I worked for IBM I'd want to know how to roll my own linked lists in case they handed me a project where I couldn't just pull an OSS library, but IBM isn't hiring in America except at the top end. All the programmer grunt work is in India. Yeah, it's easy to say I should become a superb programmer like you suggest above, but frankly I don't want to work that hard. I'd rather spend my time learning what I need to know to write my own programs and start my own business :). And anyway, as Microsoft has proven time and again good enough is good enough.

Re:I'll bite :)

[Murdoch5]

If you don't learn how to code the basics of the language how are you going to have any clue how to prevent issues that are going to come up later. I can't tell you how many guys even in my own program in university will have a C error / leak occur but NO idea how / why it happened and even less of an idea what to do about it. Most of them are to afraid to touch a pointer.

When all of them learned C they learned from doing the biggest amount of BS ever, structured programming!!!! It is the single biggest issue facing programmers.

You could not imagine how many of my profs go on and on about how important structured programming is, how important it is to never break the extents of structured programming. Why you should never use goto, why you should never break from a loop, why you should never make a infinity loop etc... etc... etc....

Basicly there are two valid forms of thought on this, A) You believe in structured programming but understand that a good programming only need know about it and produce great code by taking short cuts or B) You believe that structured programming is the all time most important thing in the world and that anyone who doesn't make strictly structured progrrammers will die.

Where I'm I going with this, All of the type B profs believe in teaching C by showing you a printf, by showing you a arrays BUT never actually getting you to program anything like that. They also have to make sure you know exactly how far you need your loop to run and keep preaching that one day you'll move into OO programming and then you can stop worrying about memory.

All my profs who are type A) show you printf and then have you program it, they show you why to use infinite loops and why pointers are the great invention to programming ever. They also agree that you never need to use bulky OO programming to solve little problems.

To learn a programming langauge just start by re writting what's already been done, learn how the internals work and learn to program by the seat of your pants and if it works it works. in 100% of cases I've seen you'll be a million times better programmer and debugger and you'll understand why shit happens and what to do about it. You'll become that rare programming coming out of post secondary who knows what memory is and why you need to free it and allocate it.

I can say in 100% of cases this makes a programmer a better programmer.