When Making a Comprehensive Retrofit of your Code...

chizor asks: "My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code) in the interest of consistency and reducing redundancy. We're going to have to make some hard decisions about code style. What suggestions might readers have about tackling a large-scale retrofit?" Once the decision has been made for a sweeping rewrite of a project, what can you do to make sure things go smoothly and you don't run into any development snags...especially as things progress in the development cycle?

Comments

object orientation

[apsmith]

we're doing something similar - and switching to java (JSP's + Tomcat, struts) to replace a lot of old perl cgi's. The java code is much, much cleaner. But object-oriented perl code can help if you don't want to take the plunge too far into a new language. And at least find a way to go mod_perl rather than CGI, for the things where performance matters at all.

Re:object orientation

[TheRain]

why is this flamebait? all this person is saying is that object orientation could make the code easier to manage and, thus, help to reduce redundancy. it's a good statement, modularize your code and it becomes more reusable... and therefore less redundant. who cares if he likes java over perl and CGI.

Re:object orientation

[apsmith]

Alright, maybe I posted a little too soon, but shouldn't "flamebait" be attracting flaming responses? I don't see any...

Anyway, if I'd spent a little more time thinking about the advice side of it, taking a look at appropriate programming methodologies (like Extreme Programming advocated in another thread here) would be one piece I'd advocate. Given the size of the code (1 MB = about 20-30,000 lines?) there's no need for major heavy-weight processes here. More important I'd say is sitting down and figuring out in the appropriate level of detail what exactly your system is doing right now - you can do this using UML diagrams which seems to be becoming a standard, though the main use we've found is to try to get an overall view of things which we then throw out when we get into the details again.

The other thing to do along these lines is look for your use of standard patterns within your code - the Design Patterns book is extremely helpful if you're moving to an object-oriented framework at all; following well-known patterns and indicating clearly what you are doing can make your code much easier for others to follow.

Re:object orientation

[ergo98]

...refactor...

Just as a point of reference: I've noticed that the term "refactor" has gained popularity with the whole XP push. Of course "refactor" is just a cooler way of saying "rewrite". i.e. It's not "We should rewrite that module", but rather it's now cooler to say "We should refactor that module". Blah.

Re:object orientation

[anacron]

Refactoring is not re-writing. Refactoring is a methodical approach to moving, redesigning, and making the code cleaner. It's not throwing away and starting from scratch. The two words are wholly semantically different.

In his book, Refactoring, Martin Fowler talks about how code "smells". He identifies a whole slew of reasons why code can "smell" bad, such as having a method that's too long, having a method on the wrong object, incorrect use of polymorphism, etc. He then outlines "Refactoring Patterns" which can be applied to certain "smells" to make the code more managable.

The last project I managed was a 3 month re-write of over 75,000 lines of code for an auction component that had gotten way out of control. The original component integrated with a very very poorly written (yet still very expensive) auction engine, and our task was to take it out.

Because there was no time allotted to proper requirements gathering, the powers that be decided that we could use the existing system as a requirements base, and just make the new system "act" like the old one.

So I was thrown two things: A staff that was inexperienced in design, and a lack of requirements. We were re-writing from scratch -- throwing all of the code away and starting again. I'm one of the only ones on my team that actually knows what refactoring means, let alone how to apply it.

So we built the new system based on a solid design. We even did the design in UML. But that only goes so far. Given a properly designed system, there is still another lower step -- how to actually implement the code. Invariably, the design doesn't take into account helper methods that are necessary, other objects, etc. Because we were on a tight schedule, I left it up to each developer to design these new tidbits.

If they had known what they were doing, everything would have been fine, but there are several places where the code is absolutely abysmal. It's not that it will perform poorly or is totally unmanagable, but in some places it's close.

Granted, the business rules for the system are some of the most complex I've ever seen, but if the developers had known more about refactoring, they might have made better choices.

There's a whole notion of Quality here that should be discussed as well. Most good has a good enough quality, and though it looks ugly, I'd throw it away. There's an interesting interview by Joel (a former microsoft programmer) which tackles these problems of Quality head on. I also reccommend reading Zen and the Art of Motorcycle Maintenence: An Inqiry into Values for a pretty good discussion of the whole notion of quality.

I'd say just keep the codebase unless you are totally switching languages (or in our case taking out a 3rd party integration). Otherwise if you re-write you'll wish you hadn't. The new codebase will end up being in the same state as the old codebase, it'll just take a little bit more time to get there.

I'd also reccommend just doing a performance test of the site, and find out which modules really need to be re-written. The 80/20 rule will invariably apply -- 80% of the execution time will be spent in 20% of the code -- so at a minimum just re-write the 20% of the code that's executed on the most frequent paths. Leave the rest alone. Re-writing it will just be a waste of time, and you have to fix all the old bugs that were re-introduced during the re-write.

Re:object orientation

[anacron]

anacron, interesting experience and solid advice. I didn't see Extreme Programming mentioned. Did you use it at all?

Actually we didn't use it formally. But we used some of the best ideas. Iterative development, frequent code reviews, programming in pairs etc. We had a pretty small team so XP might have helped a little bit more, but it's a fundamental paradigm shift that most people I work with aren't comfortable making. My team would have been less effective had I said, "You *must* use XP."

I have been on projects in the past that have embraced more of the XP model, but something like Rational's Unified Process (RUP) kind of fits in more with what we do.

For traditional software shops (e.g. those that make products) a methodology can be set in place by the management, because the company is in essense it's own client. For other types of software shops -- such as consulting firms, or web design shops -- where one-off software is commonplace, the type of methodology that's used is usually determined by the client.

For example, if I'm working with a client that isn't very savvy, I probably don't want to do all of my requirements in formal use-cases, followed up by UML design documents. They won't add any value to the client at all, yet we still have to make sure we understand what it is we're building.

It's an interesting challenge and it's usually best overcome by having solid people on the team that can speak to both sides. A solid requirements team which can talk to the client (be it internal or external) and truly understand the business problems, all the whole clearly articulating those to the technical team. Sometimees double documentation is necessary -- one set for the client and one set for the developers or development lead.

There is just no easy way to do software development correctly. It's amazing that in nearly 60 years, no one has come up with a "right" way. There's only so much a methodology can bring to the table -- after that it's about finding and keeping a competant staff who understand the value of what they're doing.

The methodology is just a roadmap. How the car is driven is completely up to the development team. Even a good technical leader can become a backseat driver at times, as there is just not enough hours in the day to provide oversight to every single member of a team.

I think the more traditional software shops should look at the console gaming industry for tips. They have a difficult challenge -- it's not possible for them to release a fix or update for their game once it's completed. So how do companies like Activision, Square, Nintendo and Sega do their software development, and is there anything that the development teams at large can learn? Rigerous testing is one of the most obvious things that comes to mind for them, but how do they know what it is they're testing when a game's vision usually only exists in the mind of the creator?

.anacron

Re:object orientation

[ergo98]

Refactoring is not re-writing. Refactoring is a methodical approach to moving, redesigning, and making the code cleaner. It's not throwing away and starting from scratch. The two words are wholly semantically different.

"Refactoring" is a trendy term (which alone makes it suspect) that came into mainstream acceptance very recently, but that most certainly doesn't mean that those in the software industry having been doing EXACTLY that for many years since? Why is it that once someone gives a name to something that people have done for years they think they've invented it?

I have never known a software project where the participants didn't actively look through, clean up, and where necessary rewrite old code, but now this is "refactoring" and it's a whole new paradigm in software engineering. Blah I say. I hereby name designing quality code and implementing it in a platform independent fashion "requantization", and my new paradigm is requantization development.

Use a pretty printer

[renehollan]

for consistent style...;

refactor as a result of learning from your mistakes and redundancies;

and try to minimize the busy parts (where all developers have a hand) when things change (like lists of unique symbols, numbers, etc.)

Sleeping dogs

[Chairboy]

I'm sure a bunch of code nazis will disagree with me (please note the clever way I attempt to pre-emptively undermine their arguments by labeling them as 'nazis') but sometimes massive engineering re-writes are not necessary.

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

Most companies that decide to massively re-engineer their code (do a big rewrite) usually end up regretting it because it forces them to re-fix the problems that caused the original strange looking code in the first place.

Does your CGI nest work? If so, maybe you should leave it alone. If you are fixing specific problems, then go ahead, but if this is a generalized attempt to fix the 'not invented here' syndrome that plagues engineers (who will almost universally agree that it is easier to write code then it is to read it), perhaps you should reconsider.

Re:Sleeping dogs

[kz45]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

This is a lesson to be learned. Engineer your code from the beginning. Use easy to understand commenting, and strucutured code. Although it takes some discipline, you will almost never have to reconsider "re-writing from scratch".

Re:Sleeping dogs

[smileyy]

++ to that comment.

A general refactoring, without the intent of doing it to add new functionality / fix bugs, isn't worth it. What business value do you gain from it?

However, there are times that it is worth undertaking substantial refactoring to add what may seem like relatively small pieces of functionality. I think it was in his _Refactoring_ book that Martin Fowler likened this to reaching a local peak in a mountain range, where you have to go down into a valley before you can climb the next peak.

ObOfftopic: This is my first post in a while that hasn't been a troll or flamebait. It just isn't as fun being informative.

Re:Sleeping dogs

[Nyarly]

Certainly, but if you find yourself trying to make a change to existing code, it will probably not be the last change you need to make. At least refactoring anything your change touches will make future changes easier to make.

Related note: the original poster doesn't say "refactoring" and he does say "Perl." Informative statements relating to these two facts:

• Buy and read Martin Fowler's Refactoring. The examples are mostly Java, and he references the Group of Four's Design Patterns. "Retro-fitting," in which you probably plan to rewrite portions of code from scratch will break your app, your mind, and your budget. Learn what refactoring implies and entails.
• Learn about design patterns in general, and consider how they might apply to your code. One description of a design pattern is "a target for refactoring."
• (Donning asbestos) You might want to reconsider perl as the language of choice for a large scale application. I realize I'm posting this comment to a Perl system, but Perl hangs together like an immense kludge of a language. That said, you're probably stuck with it, and AFAIK, you may be forging new paths in programming for reusability by applying the above concepts to Perl. Good luck, and be sure you can trust your machete.

Re:Sleeping dogs

[bbqdeath]

I second the concern about PERL. And I offer my advice as someone who has virtually no qualifications to talk about large systems of code. I just like Python better than PERL because it doesn't hurt my eyeballs like PERL.

Yeah, I guess this is a troll. But it's honest. I use Python like most people use toilet paper: several times a day, and for more things than it was originally intended.

Re:Sleeping dogs

[thogard]

You can't engineer your code from the beginning in the part of the real world where I work. I've got a huge mess of perl that do reports and about one out of ten clients wants something different. These request could have have been predicted since many of them have no business reasons and have no function other that look more like their older system which just happed to have done it that way.

Re:Sleeping dogs

[Jeremi]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

... which is why whenever you add one of these "little design decisions" whose purpose isn't blatantly obvious, it's important to put in a comment saying what it does and why it is there. Otherwise someone might come through later on, think it's an error, and remove it.

Re:Sleeping dogs

[sunking]

So you're saying your problem is too hard to be solved with a clean design? Good software design is all about preparing for the unexpected! In fact, if you ever had a perfect understanding of the problem then it wouldn't be worth designing a solution carefully - you wouldn't need the flexibility!

Thinking that writing one-off code is giving you flexibility is grave mistake.

-sam

Re:Sleeping dogs

[PD]

Sometimes, but not always, if your code is a mess, then that means that your business is a mess too. If your programmers have a hard time understanding the business, then your customers will too.

When you investigate those twistly little lines of code, see if a business rule can be simplified. Try really hard. If it can't be, then put it into code. Don't make the opposite mistake of making the rules TOO simple.

Re:Sleeping dogs

[foobar104]

You might want to reconsider perl as the language of choice for a large scale application.

I agree 100%. My company started to bring a commercial application to market a little less than a year ago. I prototyped the code in Perl, and the prototype was sufficiently okay that the decision was made to evolve the prototype into the release code.

This was A Mistake. It was A Dumb Idea. It was also My Decision. I have taken Much Shit for this from my coworkers. But you live and you learn.

I have since (over the past four months or so) rewritten the entire application-- every line, every file-- in C++. The source tree is 3.8 MB, and it compiles to about 100 MB of object code. (The actual executables, of course, are much smaller than that.) It was a pretty big job.

Not only is my code tighter and cleaner than the original Perl stuff (which was actually pretty okay code) but it's between 2 and 10 times faster.

I love Perl, absolutely adore programming in it, but there are some things that are easier to do with C, or C++, or (presumably) Java. When you split a project up among a number of people, for example, using the Bridge design pattern and distributing read-only interface header files makes modular integration so very much easier. That's just one example.

We would not have been able to get our app to market without the Perl prototype. And I don't think it would have been worth a damn if we hadn't rewritten it in C++.

Re:Sleeping dogs

[JamesOfTheDesert]

... because it doesn't hurt my eyeballs like PERL.

Do you write your PERL CODE IN ALL UPPERCASE? That *would* hurt your eyes.

Please; it's Perl, or perl, but not PERL.

Re:Sleeping dogs

[bay43270]

This is a lesson to be learned. Engineer your code from the beginning. Use easy to understand commenting, and strucutured code. Although it takes some discipline, you will almost never have to reconsider "re-writing from scratch". Even if the people maintaining the code had anything to do with it's original design (doubtful), they can't do anything about it now. A more practical solution would be to create tests, and refactor the code piece by piece. I am only beginning to use the ideas presented in Martin Fowler's "Refactoring", but they are really helping me clean up my code. More importantly, the changes are safe. Fowler's book warns me of problems I might run into, and gives hints as to what I should look for. Although I'm using Java and am lucky enough to be using IntelliJ's IDEA (which is an IDE with built in refactoring capabilities), I'm sure there is a solution for Perl (even if it is more manual).

Re:Sleeping dogs

[bmajik]

"Mission Critical" does not imply C. Infact, it points away from C.

Unless you're writing an operating system, why use C ? What does it buy you ?

IIRC, the Avionics system of the F-15 is done in Ada, because its "Safer" than C. I'd say thats "Mission Critical".

I too am weary of perl, but like everything else, its the right tool for _some_ jobs, arguably, some pretty important jobs.

Re:Sleeping dogs

[ameoba]

One other reason I see to move away from Perl as the language for the project is the complete rewrite (IE fundamental changes to syntax, deprecating all existing code) of the language that's currently going on. Perl5 is going the way of COBOL; it will become increasingly difficult to find coders both competant and up to date in their skills.

I'm not really qualified to make any recomendations about which language you should use, but if you're doing a rewrite now, when Perl5 coders are still plentiful, choose a language that you can be reasonably sure will be used 3yr from now. The whole point of the project is to ensure future maintainability for the software..

Re:Sleeping dogs

[dgroskind]

The hurting eyeballs factor only comes in when that quick hack gets progressively promoted to a full mission-critical program with little or no planning.

The trick then is to pick a platform that is suitable for quick hack but can be promoted to mission critical with little of no planning.

Perl is a suitable candidate because

it's reasonably fast in execution

there are a reasonable number of Perl programmers around for future maintenance

it has some cross-platform capablility and scalability

Most people have seen more of the ugly ones.

There is a trade-off between speed of development and maintainability. In the current business environment, speed of development is ascendant. The people who say you should've planned better and used Java or C++ usually come along later and try to score some political points. Where were they when there was work to be done?

Re:Sleeping dogs

[foobar104]

Isn't it possible that your rewrite was better just because it was a 2nd draft? There's really no reason you can't define interfaces for Perl classes and share them among a group of developers, like you did with your C++ header files.

Yes, TMTOWTDI.

But I still think that C++ is my language of choice for big projects because it helps do some of the job of collaboration for me. If you, as a team member, don't implement the interface completely, your module won't compile. And if you change the interface, nobody else's modules will compile. So less time is spent doing module-level integration.

And C++ has a lot going for it in the preemptive bug-squashing arena, too, with strong typing at compile time and such.

But I didn't intend for this to be a "C++ is better than Perl" comment; as I said before, I love Perl. I just think C++ is more appropriate for some things than Perl is, and medium-and-larger programming is one of them.

Re:Sleeping dogs

[Ian Bicking]

Good software design is all about preparing for the unexpected!

I couldn't disagree with you more. Putting generality into the code that you don't forsee a need for is very bad design, IMHO. Generality is messy, and generality in one piece often makes it difficult to make another part general.

For the most part, your code should be designed to do what it is supposed to do, and no more. Of course, you will have an idea of what it might be needed in the future -- and it will sometimes be a good idea to design the code with that in mind. But usually it is not, because you will often be wrong. And if you don't even have an idea of what need your code will fulfill -- i.e., you are truly planning for the unexpected -- then you would be a fool to mess up your code in planning for it.

One-off code is perfectly fine. Of course you want to reuse code as much as possible, but it is not possible to reuse all your code. If someone wants to do something new, then you'll have to write new code. Hopefully you will do it without using cut and paste -- a horrible way to do code reuse -- but you will have to write some new code. Maybe you'll make a new subclass (my favorite), or add a new option, or refactor, or a combination of the above. And, sometimes, it really will be more expedient to copy and paste.

Extremem Programming - Refactor

[Strudleman]

You should read up on Extreme Programming, in particular Code Refactoring. It's a method of cleaning up old code. A very well written book, as well as an excellent code-housekeeping method.

Re:Extremem Programming - Refactor

[Anonymous Coward]

Do 'Extreme Programming' principles dictate that I get an afro perm and adorn my body with piercings and generic "tribal" tattoos?

First time I heard the term, I asked the guy "Is that where you jump out of an airplane with a parachute and a laptop?"

Re:Extremem Programming - Refactor

[(H)elix1]

Yah, I keep hearing this. Read the whole book - it recommends strong test plans, incremental development, pair programming, tons of snacks, and a maximum of a forty-hour work week. In itself, not a bad thing.

Interpreted by management?

• They see the part about everything being stuck out in the open, with the computers set up in the center of the room - cubes are overrated, offices are evil
  
• half as many boxes are required since you are "coding in tandem". See above for hint on how personal space is valued.
  
• Demand complex test plans that will cure everything. These same people never have actual requirement documentation for what the code should do. Classic example? Validation rules for a phone number... gets ugly when someone wants to put in 1 800 FOO-BARR, add an extension, deal with international numbers. Think you can put together a test script? Got to have requirements first, and that is never a hard thing to get from the business (shudder).
  
• I'm sure you have had complete management buy in on realistic time estimates now, so I wonder where the rumors come from that development tends to work more than an eight hour day. When they find out how long it will take to refactor the code base with through testing and daily updates, they will tell marketing and adjust the dates accordingly. Heck, I thought the Microsoft Project Plan defaulted to an eight-hour day - must be one of those rare Pentium errata issues that messes things up for me.
  

Not that I'm bitter, cause I'm not... but XP takes a handful of generally good practices and assumes perfect management buy-in and team communication. With any management support and teamwork, almost any method works.

Modules

[SCHecklerX]

Break the code into re-usable modules (or objects if you go another route besides perl, or even if you do, if you like that sort of pain), each programmer responsible for his own set (ie layout, calculation, database, etc)

CYA

[The Gardener]

Don't stop maintaining the old code code until the new code is on solid ground. No matter how sure you are that you can do it, the new code might never come through.

The Gardener

Don't do it

[mrpotato]

See the recent Joel Spolsky interview here, that was discussed on /. here.

Basically, Joel's take on a similar problem is: don't do it.

Unless you have a _really_ good reason to do huge change to a big codebase, don't bother, and make something more productive instead.

Code style and large rewrites.

[The Panther!]

It always seems to me that large rewrites (though I've never done one as large as the one discussed) tend to make bad code bases worse for a while. If there is any effort made to maintain functionality in a changing codebase, a piecemeal rewrite is more of a headache than a help. So, if you're planning on a big rewrite, please give your developers the full thumbs up to break everything and expect them to put it back together later--and expect to see nothing tangible in the short term. Or call it off.

That being said, my last company sat around and bickered about code style for nearly 4 months and produced no code that wasn't rewritten later. If you are going to concern yourself about style, settle that well in advance and make sure it's logical and consistent.

It's also been my experience that conformant code style is highly overrated. Once the Best Practices document extends beyond language constructs and caveats, into brace styles, spacing, tab size (yes, there was a 3 space tab stop standard at my last job--wretch), and even the naming of locals, parameters, members, constants, enumerations, etc... it got to be a thick ass bible of stuff that only a few people would digest or attempt to adhere to. The point I'm trying to make is, choose your battles. The hope is your developers will make sane choices independently, and use standards to help integrate different peoples' work together. Anything beyond that and it's pissing in the wind.

My $2e-2 or less.

Horror Story

[Brontosaurus Jim]

My firm went through this sort of thing just two years ago. The PHB at the time decided, for some reason, that our 300,000 lines of semi-poorly written C code, and 50,000 associated lines of Java (Dont' ask).

Anyway, it took 7 of us over 2 months to get even halfway done. The pressure the boss was putting on us was awful, and he didn't really even understand what we were doing, even though he was the one demanding it. I think she read it in a trade mag somewhere. God I'd do a lot more work if she didn't read that shit.

Anyway, about halfway through the "Great Leap Forward" (as we [appropriatly] named it) the boss quit, and the next boss, who so far has been fairly clueful. He didn't think the whole deal was needed, but he was pressured by the former bosses husband (the CTO) to get it done. Seriously.

Hope yours goes better than ours. From what we did, heres some tips I can give you.

1. Be consistant through the whole thing.
2. Make sure everything is planned before you start. This was the one part we got right.
3. The team you have should have worked together before, because this sort of task requires previous knowledge of eachother.

Other than that, my condolences. Or maybe it will work better for you.

Good luck!

Rule #1

[ackthpt]

We're still in the middle of a sweeping change and lemme tell ya, make d@mn sure there's someone accountable for managing the whole project from beginning to end, particularly this being their main focus.

Transitioning in new managers or having the current manager only look in on the project once in a while is as sure a path to madness and doom as no management at all.

Our due date was mid-August, we'll be lucky to get it through testing and into production by January 31st. All the while with the logjam we're having to put pieces of it into production and cross our fingers that the new changes don't break anything.

Love to talk more about it, but need another gallon of coffee.

That there is no such thing...

[pyrrho]

You are going to rewrite the system from scratch. Design from scratch. Your new design might be able to use some old code, if the old code is useful.

A large scale retrofit is really an oxymoron.

IMHO, but with 15 years experience.

Re:That there is no such thing...

[awol]

I am working for a large company, which was only tiny when I started working for them (over seven years ago). In the early days we were a bunch of really smart guys, a few of which could be characterised as zealots. One would g home on a friday and come in on a Monday to find a fundamental component of the system rewritten in a new and interesting way, even things like new tools or processes that we would all then haveto use. This kind of retrofit never caused us a problem (seriously never) some of the decisions have even been subsequently validated by industry changes.

BUT

Two things. First, in truth we wre cowboys. That kind of thing is such a bad idea that you cannot do it and develop consistent quality product, particularly in larger teams, greater than six or so (or geographically diverse teams). Second we always were able to run these systems in parallel before we switch over to the new way. This is IMPORTANT and some other posters have said it before. Don't get rid of the old until the new has been proven (as for how to prove it well that is real hard but don't underestimate just how hard).

As an aside, we tride a pretty major redesign of a system a few years back and it did not really add any value, but that was because of a lack of a clear objective and we lost the budget after six months. So Try and make keepable improvements with shorter timscales.

Listen To The Experts

[ChuckPollock]

The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet. -- Michael Jackson

Document

[Tosta Dojen]

I suppose what I have to say almost goes without saying, but I know so many programmers who neglect it, I will say it anyway. There are two things you have to know when going back into old code.

1) Know what you are doing. 2) Know what the code does.

Both are expedited by good documentation. This is so important, it deserves to be written thusly:

DOCUMENTATION!

Write everything down. If the code is not commented, figure out what it does and write it down. When you add a line or a module, write down what it is supposed to do. Declarations? Write those down too. Document everything so that you can figure everything out, both now and down the road when you decide to fix something else.

This is the voice of experience. I have had to reverse-engineer my own code 6 months after I wrote it because I failed to document anything. Learn from my mistake.

Think again

[pkphilip]

From what you say, you are planning on making these changes to clean up the code and make it prettier.

I would strongly urge you to reconsider this as the probability that you will end up breaking parts of the code while "cleaning" it up is quite high..especially since you seem to have a fairly large code base ~1MB

Ugly code containing redundant stuff is still better than beautiful code that doesn't work.

Regression tests

[Weasel Boy]

Make sure you have a suite of tests that produce known output for your old code, so that you can ensure that the new code works in exactly the same way. Don't add anything new until you are proven conformant with what you had before.

See no evil, hear no evil, ...?

[ProtonMotiveForce]

Why do people mod comments about alternatives as Flamebait (and presciently, this one)? Are people afraid to hear that you should't write large scale systems in Perl?

It really is valid (and in my opinion, correct) to say that if you _are_ going to do this you should look at other technologies and languages. Perl is for system administrators and system administrators-cum-developers, not real software development. Look at java. Look at PHP. Look at commercial and non-commerical web application systems, like Zope. Or don't rewrite it at all if it works. But for God's Sake, don't rewrite it in Perl - it's pointless.

Re:See no evil, hear no evil, ...?

[spudnic]

What would the advantage be in changing to another language? Assuming the team develops coding standards and creates clean, documented code that can run under mod_perl, why would PHP or java be better?

The team is obviously already familiar with perl, and some code could be reused after a bit of cleaning. Perl is perfect for applications that require a lot of string manipulation and as a front end for a database. This is what most web programming involves.

The only drawback that I can see in using perl is if your team is messy. Perl will allow you to be messy, but it doesn't have to be. Perl can look just as clean as any other language. I go back to apps I wrote 3 years ago in perl and can follow every bit of it. It's all commented and has plenty of white space to make it easy on the eyes.

I've written several large perl applications for the web and yes, I do know other languages, but perl is better suited for this type of work. It's fairly quick to put together and performs well. Now I wouldn't try to write an OS or first person shoot'em up game in perl, but for this it's just as viable of a choice as anything else.

I'm also not saying that PHP or java would be bad if you have a good grasp on it and are starting a new project. Remember, TMTOWTDI is good!
Just don't use ASP! ;)

Re:See no evil, hear no evil, ...?

[khkramer]

> Perl is for system administrators and system > administrators-cum-developers, not real > software development. Look at java. Look at > PHP.

Mmmm. I write large-scale web applications for a living, and I do it in Perl. By large-scale, I mean sites that are expected to support hundreds of thousands of page-views per day, serve hundreds of thousands of distinct users per month, and collate hundreds of thousands of distinct chunks of content into dynamic pages.

My company is a high-end development shop. We generally bid on projects that will take six to nine months to complete, and we only do jobs for clients who understand how we work and why. Part of our approach is to use very small teams of extremely experienced web developers. We usually deploy four programmers on a project. Other companies that bid against us sometimes use several times that many people on a single development team. Another part of our approach is to build everything on top of our open-source development framework. That sometimes used to be a tough sell ("what, give software away for free, heaven forfend") but these days, most customers are pretty receptive to the twin "more eyeballs means better code" and "you're not locked into our closed, proprietary product" arguments.

We also generally build small clusters of dual-processor Athlon or PIII machines, whereas our Java- (Oracle-, IBM -, BEA-, etc) wielding colleagues often specify absurdly-expensive hardware.

Perl is flexible, complete, performs relatively well and has an extroardinary base of skilled developers and re-usable components (CPAN). We couldn't work as quickly or write code that is as concise and maintainable using any other currently-available language/approach. Most of the lines of code that we write actually go into Perl modules, with HTML::Mason templates handling the dynamic web-page generation. We can push Mason out of the way and use straight mod_perl for small, defined tasks. And we can easily integrate C code into our Perl frameworks in places where performance is really, really critical (though those places are rare, as when push comes to shove, one is almost always waiting on the database).

There are lots of things that aren't "right" about web development. The package that results from gluing HTML and program logic together in a stateless execution environment is sometimes a little lumpy, and unavoidably so. There's no magic bullet toolkit, and (as with other specialized programming arenas, like graphics or embedded systems) a lot of hard-won, domain-specific knowledge goes into the development of a fast, reliable, maintainable web app.

The Perl/Apache/Mason combination that we use is far from perfect. But it's better -- for us -- than any of the alternatives.

I really like Java, and have written big systems in that language, too. If for some reason I had to manage a very large team of programmers, or had to manage a team with a large percentage of less-experienced programmers, I would use a Java-based solution. Java is a more rigid language than Perl, and the structure that the language provides would be a useful management tool in those contexts. But for my small teams of skilled hackers, Perl is more productive. (We have an extensive, evolved, self-imposed "structure," so we don't need the language to impose one on us -- in fact, it gets in the way.)

I would never use PHP for the kind of work we do. PHP just isn't the kind of powerful, flexible, complete environment that Perl is.

Zope and Python are really neat. I'm a fan of the work that folks on that side of the fence are doing. But Python+Zope don't offer us anything new that Perl+Mason+Comma don't. I also like Perl more than Python (which is a subjective preference), and think that the Perl development environment is more mature (which is a subjective judgment).

So don't listen to the folks who tell you to dump Perl. You should certainly consider all of your options and make an informed decision about core tools, but anyone who thinks that Perl is just a "scripting" language, or that it doesn't scale, hasn't been paying attention.

To finish this up with a little more specific advice to the original poster: You mentioned "150+ perl CGIs" in your question. You should consider moving away from the CGI model, if possible. Take a look at HTML::Mason, which is a very good embedded-perl environment. You can build solid, consistent application layers using Mason as a base. Also, I couldn't agree more with the folks recommending writing perl modules and requiring complete regression tests for each module. There are lots of ways to write tests, but in perl-land one of the easiest is to simply make a t/ directory down your module tree, write a bunch of scripts in that directory named <some-test>.t that print out a series of "ok <n>\n" lines, and use make test or Test::Harness::runtests() to invoke them.

Architect it to death

[ADRA]

There is no use in doing a rewrite unless you do it right from the beginning. If you can't spend a decent amount of time planning the architecture of the system, then stop now, and quit.

Also, since you have decided to pour resources into this thing, then my opinion would be to make as much of your code generic so that you don't have to make code changes later. It doesn't matter if there is an initial performance hit with the systems, becuase in the short term, you can convince your boss to get a new leet server, and in the future, hardware needed to run your apps will be trivially cheap anyway.

If you are going to cut a new release, try to avoid going back and taking snippets of code from your old system. It makes people slide back into the odl paradigm and can cause detrimental effects on the bottom line of your new system. This is new, so the least exposure to the old one, the better.

Re:Architect it to death

[Hairy1]

No Don't!

In my early experience in programming I found there was an advantage to up front planning. It seemed like the more planning you did the better the project would do. So there was a tendancy to do more and more up front planning, getting into more minute detail.

However although we spent more and more time planning the payoff in terms of being able to predict schedule and budget diminished. Its classic diminishing return. But why?

Planning a project is like forcasting the weather. You can plan a small project in minute detail for a short time (like being able to predict the weather a day in advance), and you can make vauge predictions for some time out (ie you can know the probability of rain based on the month - statistically on previous years), however you can't plan a large project in minute detail.

The reason is that if you get any of the initial decisions wrong, and there is a minor deviation from the plan early on, this will be reinforced to cause more changes, and this will snowball to change many parts of the final system.

An analogy is trying to get somewhere with a set of instructions. If the instructions are faulty at the very start of the list it doesn't matter that the rest of the instructions are correct - you still get lost. Detailed plans are a little like a set of precise instructions. There is no assurance that all of the instructions are fault free. In fact there is a good chance they will be faulty as they will almost always be written by someone who hasn't been to the destination (ie hasn't written the same program before).

It is better to be an explorer. Your "plan" should be like a map. Only you start out not knowing the terrain - only the general direction of your destination. You map your first days travel and set off. Every day fill in some detail on the map and plan the next days walk - always keeping in mind where you final destination is.

Programming is a journey - not through any physical space, but through the endless possibilies of the programming solution space. This space is filled with broken, useless programs. Its your job to navigate your way to the best solution.

PS : I know this hasn't been very practical, however I have previously posted some more concrete real life advice under this article.

Use framework, seperate code form display.. and..

[cybrthng]

Well, i can't say it enough. Use a web framework, i don't know of any for Perl off the top of my mind, but i use Resin.

Resin is a JSP, Servlet, XML, XSLT application server that support all the latest and greatest EJB components and Managed persistance on the database and makes a great framework to build from.

I have Postgresl for the Database, and use beans to run queries and output via xml and then i have XSLT draft xml data into the wonderfull HTML code.

The beauty is, my code is in beans, servlets and jsp's. My HTML is in .xtp files (in the case of resin) and simple XSLT sheets parse the XML to render to the user.

Just means i can produce output easily for WAP, Palm, CE, Normal Web Browsing, EMail, and what not without modifying the backend. Just create xslt using a session identifier to bring up the corresponding stylesheets for whatever device is acesssing the page.

Enough about java, but something similar, be it in house developed would be your best bet.

I also get away with using a Swing applicatoin to manage the database, users and run reports provide an easy to navigate gui which just interprests the same xml data that would be retrieved by the html client. Not a single change to the backend since i'm using the beauty of soap, jsp's, servlets and java.

Virtualize your interfaces, standardize on your backend and use re-useable components. Perl is similar in many ways that you can load libaries and abstract your code from the display which will save you tons of hours of hassle in any future upgrades compared to the bit of changes you would do now

Don't Listen

[augustz]

A ton of people will tell you a ton of things, never having retrofited anything.

- Do not undervalue the investment you have learning your existing coding language. New challenges await you if you jump on a new language like java. Make the jump if you are excited about learning about the new language.

- If you use your existing coding language you will literally fly through the retrofit. Do it piece by piece. Make all those changes first, then test app, then make next set of changes then test. The simple fact is, most wasted time is spent on bugs not working on performance, and you've already knocked down a lot of bugs, don't let them pop back up by blowing everything up. There are books on this.

- Sometimes blowing everything up is worth it. Do it right this time. Realize it won't be as perfect as you might think it will be.

- Remember there are countless open source and shareware products that tried to create TNG with a total rewrite, got nowhere, and ended up improving their existing product. Remember the lesson, bite off what you can chew.

- Spend a week poking around researching possibilities. I do this all the time, bookmark things I think are important. Then for the next project you've got all the little things you might forget at your fingertips. Optimzations/Tools/Paradigms. Think you know it all? You'd be suprised at what is out there and what you missed. And what you spent a month in house re-inventing. This one's important.

- Use open source software. Nothing beats free. Nothing is more fun. Java's ugly standardization history makes me puke... the BS Sun has pulled with Java is staggering. That the Java Lobby swallows it and loves it even more so. This is irrelevant to your question, and not fair to the Lobby, but I like to give them a hard time.

- Colorary to Java. You need less abstract design then you think. Endless object hierarchies will weigh you and your app down... Their are books on this too.

- You need more documentation then you think. Ever found code someone ELSE wrote too EASY to follow. I don't think so. Especially if you are using perl and someone is enjoying the line noise capabalities perl allows. Perl has 20 ways to do EVERYTHING, you may not know the latest or twistiest. Document as you WRITE the code. Do not leave at the end of the day without catching up the docs. A week of documenting is the worst form of hell, avoided with a minutes worth of clarification each time you write a function/class.

- Hardware is cheap.

Anyways, have fun... and good luck. Be interested to read what others have to say.

Re:Don't Listen

[Beryllium Sphere(tm)]

>- If you use your existing coding language you will literally fly through the retrofit. Do it piece by piece. Make all those changes first, then test app, then make next set of changes then test. The simple fact is, most wasted time is spent on bugs not working on performance, and you've already knocked down a lot of bugs, don't let them pop back up by blowing everything up. There are books on this.

This is good advice. To be more specific:
1. START with your regression test suite
2. Then add self-documentation features like standard naming conventions. Seems dull and bureaucratic and pointless but really truly saves maintenance time.
3. Have a standard comment header for each function. The standard should answer questions like "Can that argument be NULL?" and "What do the error returns mean?"
4. If you're going through every line already, do a security audit.

There's good advice in the refactoring books, for example http://www1.fatbrain.com/asp/bookinfo/bookinfo.asp ?theisbn=0201485672&vm=

Re:Don't Listen

[banky]

hardware is cheap, but bumping up the hardware requirements when you're talking about pushing out your new code to several hundred machines in a hosted data center will quickly kill that Xmas bonus you were hoping for...

Re:Don't Listen

[augustz]

Yes, I agree with this. I actually dislike some of the easy but (what I consider heavy) choices like Java for this reason. You can get by with so much less so easily for many projects, and support literally 5x as many users per server. For a small business this is the difference between profit and no profit. For a large business this can be the difference between .com and .bomb.

All too many times I see sites begging for money for hosting/bandwidth. Take a look at their HTML/CSS and see it is HUGLY bloated (no linked css, prevent all caching including image with default cache buster installs) and not gzipped, and I wonder, if what I can see is so bad, behind the scenes it is probably even worse. (ie dynamic page gens where none are needed). Which I had included this in my list and left of the hardware point, which I agree is the wrong message.

But damn, if you do code right, hardware is so cheap I can't beleive it. I'm convinced some 10 machine projects with bad coding can be supported on a single machine now.

Re:Don't Listen

[Lumpy]

I agreed with everything until the last line. That statement needs to be violently beaten out of every programmer, manager, and engineer on the planet. I almost want to suggest that the death penalty be decreed on that statement od heresy.

The "hardware is cheap" mantra is exactly why we have crapy code and bloatware. "I dont have to optimize, hardware is cheap." "Why make it efficient? hardware's cheap!" "I dont care that it is slow, hardware is so cheap these days"

Please, everyone, cary stones in your pockets and throw them really hard at the next programmer that makes that statement.

Re:Don't Listen

[RollingThunder]

That last line is making me have to BS my boss.

"Sure, the dual tbird appro should handle 10000 users"

omitting, of course, if it was all written in clean tight C, instead of java.

see joel on software

[kubalaa]

He says the exact same thing as you, only better.

Re:see joel on software

[bay43270]

Although I enjoyed his book, and respect his opinion on matters of interface design, Joel Spolsky is no champion of well-designed code. In his book, he even suggests that a programming language shouldn't be used if another can create the same user interface in less time... as if the time it takes to create the first version of a product should take precedence of other factors such as maintainability of code, reuse, and flexibility. He does a good job of regurgitating the works of other interface specialists, but he is not an expert in code maintenance.

KISS

[ZaneMcAuley]

Keep It Simple Stupid. Keep code simple, why have funky lines of code that looks awe enspiring. Its pig ugly. I work on the principle of KISS. When I am working on a code area that is being cleaned up or fixed, I try to simplify that area. The simpler it is, the easier and quicker it is to maintain and bug hunt.

Im a strong believer in managed code (whether its C# or Java, except managed extensions to c++ which are damn pig ugly :) Exception handling for seperating error handling from actual code that performs the actual work).

Make sure there is GOOD in code documentation (and out of code documentation) to explain the intention of that code. (Intentional programming is a research area btw, to program ones intentions).

You know when something is bad when you have to maintain that code later on, 6 months down the line or whenever. Thats when it bites you in the buttie.

Refactoring your project

[adamy]

To expand on the concept of Refactoring:

1. Write a test for a specific block of code.
2. Appliy the refactoring

You are going to want a good testing framework.

To expand on the modules post: Do a dependency analysis. If you are writing DB based code, look at what tables can be logically grouped together.
We did something like this at my company not too long ago. The basical level package we had was the security package, which identifies users and roles. Most other packages depended on this. All contact management stuff went into a package called Directory. All stuff for the people our system was managing went into Participant etc.

For each of the packages, split the code out into a set of interfaces, and a set of implementing code for business logic, and the UI required to drive that business logic. This is the standard breakdown for code. You may want to further pull out code into helper packages. Avoid the urge to call these helper or util, and instead try to name them based on what they contain: we have one called format for stuff like phone numbers and social security numbers.

Don't forget the make scripts. What ever build you use, it should be used to specify which modules you want to compile/deploy

I recommend a little UML modeling session for the end package structure.

Go in little pieces. After each refacotring, makes sure stuff still runs.

Good Luck

Playing with fire...

[ZaneMcAuley]

Do you have the original engineers for that code?

Most knowledge is in theyre head, not on paper unfortunately. Theyre experience in that area can never be written down.

From experience:

[tqbf]

Plan a big sweeping rewrite; idealize the system you design based on your experience with the real one. Make the design clean, simple, and flexible.

Don't build it. Instead, evaluate the code you have now and plot a course towards the idealized system. Approach the actual work of the "retrofit" incrementally. Count on having multiple customer-facing revisions of the software tagged and QA'd before the system you're delivering looks anything like the planned rewrite.

Taking baby steps towards a new design is probably the only way you'll ever migrate your project to that design. With the knowledge you've accrued working on the old system, it probably seems straightforward to start from scratch. Even if this isn't wishful thinking, though, it's a waste of time. Part of the discipline of design is an understanding of where the "hot spots" are that can't tolerate inferior implementation, and how to tell those hots spots from the spongy mass of integration, reporting, tracing, and sanitizing that is neither performance sensitive nor mutable enough to justify engineering effort.

You can take early baby steps that make it easier to make holistic changes down the road. Refactor relentlessly. Migrate code recklessly out of subsystems and into common repositories and libraries. I've found it handy to distinguish between "proper" shared library and "dumps" of utility code that don't need scrupulously conceived interfaces.

Most importantly, design for testability. In this respect, the biggest asset you have is the steaming lump of old Perl code you're facing; use it to figure out the expected behavior of subsystems. Write replacements, in modules with clean interfaces, and unit test them. A unit test probes code (functions, statements, internal states) --- NOT entire programs. You'll work ten times faster when you can move forward as a team knowing what components you can trust and what components you need to worry about. You'll work ten times slower if you haven't clarified your outputs, side effects, and return values enough to know whether your replacement parts are valid!

We've seen articles on Slashdot before about this and I agree with the prevailing opinion: rewrites are often seductive traps and time sinks that don't offer value to customers. A better mentality that will eventually get you where you (think you) want to be is to adopt a strategy of constant measurement (testing, profiling, debugging) and improvement.

Parallel development

[rgmoore]

It might seem like and obvious step, but don't throw away the old system until you're sure that the new one works! Keep somebody minding the existing, working system so that if/when your attempt to completely rework it fails you won't be stuck. Once you have rewritten it, try setting it up on a trial basis in parallel to the working system so you can find the crippling bugs before they take down your system.

While it's not a perfect example, Slashdot is actually a decent example with their switch to their new system. They kept the old, crufty version as the primary and set up a beta site with the new software. They knew that there would be problems and got some of their more loyal users to test the new system and only switched over it after they were pretty confident that they had gotten the worst problems out of the way.

You can afford to take a few more risks as long as you keep a known working system around as a fallback.

Unexploded bombs, not sleeping dogs

[ChaosDiscordSimple]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

Yes, and if they're cryptic and uncommented, they are worthless. Eventually one of these incomprehensible, magical fixes will stop working. Perhaps the bug it works around is fixed. Perhaps how the function is being used changes to previously unexpected behavior. Some poor engineer will look at the little big of magic, scratch his head, and be forced make a blind decision about how to fix it. Perhaps he can change the code while leaving the bit of magic in working, but he can't be certain, since he doens't understand it. If the collection of cryptic tweaks becomes dense enough, any attempt to fix a bug or add a feature becomes highly risky.

On a related note, don't let this happen to you. If you add one of these strange little fixes, for the sake of the programmer that follows you, document them. Just a little "Need to toggle the Foo because Qux 1.4 is correctly fails to do so" will bring tears of joy to the eyes of future programmers.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

You're wrong. The comments you recommend will only /add/ to the mass of code to be grokked. Don't forbid commenting; but don't consider them a solution.

Instead, refactor and unit test. Unlike comments inserted into the middle of the code, unit tests will fail and point you to the reason for the failure. In the above example, when we upgrade to Qux1.5, the unit test which asserts that the Foo is untoggled will fail, and will point right to the function which made the assumption. Bingo -- a quick fix.

-Billy

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Comments used that way are like perfume -- they cover the smell of stinky code. Perfume isn't bad when your code stinks, but better is to make your code not stink.

A better solution would be to extract the thing you're commenting into a function, and name the function so that the comment is unneeded. Don't forget to write a test which checks the purpose of the function. Then (and only then) delete the comment.

MakeNotAccidentallyClickable(widget);

See what I mean? Now it's impossible for your comments to become out of sync with your code -- your code IS your comments.

-Billy

Re:Unexploded bombs, not sleeping dogs

[Anonymous Coward]

My favorite comment to see in the code "This is a bit of a hack"

Re:Unexploded bombs, not sleeping dogs

[p3d0]

I think that the idea that code can somehow take the place of comments is bogus. Code, no matter how clearly written, only tells you what is happening. It can't tell you why.

For instance, why did you make that widget not accidentally clickable? Why not others? Is that widget in particular danger of being accidentally clicked? Are the consequences of accidental clickage very bad?

Code like this seems clearer to me:

/* Make sure user doesn't accidentally delete the document */
disable(widget);

Or:

/* This widget is very large, and liable to be clicked accidentally */
disable(widget);

Re:Unexploded bombs, not sleeping dogs

[medcalf]

If it's that important to you that "your code IS your comments", just write in COBOL. COBOL is documentation that runs.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

I coded for the example I was given, at the level of detail I was given. You've added some more detailed scenarios; and I can code for those too.

The first comment should be a function name, not a comment. There's a LOT more you have to do to prevent document deletion! The comment, as it stands, is misleading, and later modification will possibly have to add more functionality. Just make it a function, so the functionality can be kept together.

The second comment says, "I, the programmer of this system, don't like the UI design." BAD SMELL. Use comments if you must, but a FAR FAR better solution is to fix the UI. Don't you agree?

Now, I'm sure you can invent other details under which those specific solutions are inapplicable -- but for every scenario you invent, I'll invent a way to code it so the comments aren't needed. The only exception is when time or lack of knowledge prevents removing the comments -- and the result is tragic.

-Billy

Re:Unexploded bombs, not sleeping dogs

[p3d0]

The second comment says, "I, the programmer of this system, don't like the UI design." BAD SMELL. Use comments if you must, but a FAR FAR better solution is to fix the UI. Don't you agree?

Hey, you're the one who gave the example, so you tell me why you would call the "MakeNotAccidentallyClickable" function. I can't tell from your code, and that's the exact problem.

It seems to me that my comments have helped you to diagnose problems in the code that were not obvious without those comments.

for every scenario you invent, I'll invent a way to code it so the comments aren't needed

Why would you want to do that? What is the advantage of your scheme over comments?

Your aim of having clearly-written code is admirable. I think code should explain very, very clearly what it is doing. However, explaining the "why" when it is not obvious from the "what" is exactly what comments are good at.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Hey, you're the one who gave the example,

I didn't give that example -- look at the history. That was given by someone who thought that the comment was needed (they provided a comment, too). I made a function which contained the code being commented, gave it a name which was essentially identical to the comment, and called it where the comment was. I also claimed that in any real work, I would have first built unit tests which confirmed that the function did what its name claimed.

so you tell me why you would call the "MakeNotAccidentallyClickable" function. I can't tell from your code, and that's the exact problem.

You use it when you want to make something not accidentally clickable. I haven't defined what "something" is, and I haven't defined in what context it's useful. The first job is for the type system, and the second job is for the object system.

It seems to me that my comments have helped you to diagnose problems in the code that were not obvious without those comments.

No! There were no problems in my code; there couldn't be, since I wrote no code. What I did was respond to someone else's problem sketch with a solution sketch. You responded not by pointing out omissions in my solution, but by inventing new problem details and situations. The fact that my example solution didn't cover them hardly matters; it was a solution to a /different/ problem.

What is the advantage of your scheme over comments?

A superb question.

Obviously, it can do everything comments can do except contain language syntax; so it's clear that for most purposes a good function name equals a comment. So why should we choose a function name _over_ a comment?

The first answer is that when more than one line is covered by a comment, those lines form a functional group and MUST be a function of their own. Once you have that, it's silly to not choose the new name to be as descriptive as the comment.

But there's more. If only a single line is covered by the comment, it's still beneficial to extract a function and delete the comment: by encapsulating the "how", you provide opportunity for later changes to the "what".

But the final BIG statement is this: comments can't be tested. Functions can, and a unit test will make it obvious when a function's name doesn't match its function.

However, explaining the "why" when it is not obvious from the "what" is exactly what comments are good at.

Please tell me SOME information in a comment which could not be conveyed in a function name. Fact is, you can't do it. Function names are exactly as expressive as comments.

There is some info that can't be conveyed by code; it's called "requirements". That info can't be conveyed by comments, though, either. It should be contained in use cases and acceptance tests, one acceptance test per use case. Acceptance tests should be automated, but that's a different story, and up to the customer, not the programmer.

-Billy

Re:Unexploded bombs, not sleeping dogs

[p3d0]

I didn't give that example -- look at the history.

I was referring to this code:

MakeNotAccidentallyClickable(widget);

That's what I meant by "example".

No! There were no problems in my code; there couldn't be, since I wrote no code. What I did was respond to someone else's problem sketch with a solution sketch. You responded not by pointing out omissions in my solution, but by inventing new problem details and situations.

Ok, perhaps I made a straw man. The point I was trying to make is that there must be more details to the situation, whatever they are; and because your code didn't specify them, the problems you found in my commented version of the code may or may not exist in yours. One can't know without more information.

Perhaps I can sum up my thoughts this way. Programmers make mistakes. It's hard to tell mistakes apart from deliberate decisions without some indication of the programmer's frame of mind. Thus, comments are only unnecessary in perfect code. If you choose to encode your comments using function names, rather than the features provided by the language specifically for the purpose, that's your choice; but it's not yet clear to me that it is a beneficial one.

A superb question.

And you have provided some superb answers that deserve some thought. I'm not convinced yet, but your position no longer seems ridiculous. :-)

Have you ever come across (or written) any material that discuss the pros and cons of your approach?

I do have a bit of a problem with the assertion that a block of comments should always become a separate function. There is a "sweet spot" when it comes to decomposing functions: with too few functions, they are too large to understand; with too many, algorithms can become scattered and fragmented, like a book with one word on each page. Splitting every function block into a new function seems to run a great risk of fragmenting the code in this way.

To me, function decomposition already serves three purposes: information hiding, code reuse, and divide-and-conquer problem solving. Adding a fourth criterion--of allowing comments to be encoded as function names--seems to raise the possibility that the criteria may conflict, and one may have to be sacrificed for another. Looking at the list, I know which is first on my chopping block.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

I was referring to this code:
MakeNotAccidentallyClickable(widget);
That's what I meant by "example".

Yes, I wrote that -- but it's a direct translation of the comment someone else told me couldn't be replaced with a function name.

Ok, perhaps I made a straw man. The point I was trying to make is that there must be more details to the situation, whatever they are; and because your code didn't specify them, the problems you found in my commented version of the code may or may not exist in yours. One can't know without more information.

You're absolutely correct. However, this situation MUST be left to the wisdom of the programmer. There's simply nobody else to trust! If the programmer's wrong, he'll write wrong comments as well as wrong function names, and most likely wrong code as well. (See "peer review" and "pair programming" for some solutions to this.)

Perhaps I can sum up my thoughts this way. Programmers make mistakes.

Just a quick interruption: so do maintainers. The programmers shouldn't assume the maintainers are going to make all the mistakes in the book, and neither should the maintainers assume that about the programmers. A lot of common programming practice seems to assume that a horde of idiots are going to take over the code tomorrow, so anything I don't finish today will get destroyed and perverted. This results in late hours, aggressive comments (like "DON'T TOUCH THIS!"), a lack of tests (because you don't think the later programmers will have the wisdom to improve you code), and so on. Some cynicism is appropriate; after all, the later programmers will have at best a foggy memory of what you were thinking when you wrote it; however, at the same time, it's just as likely that the later programmers will understand the overall problem BETTER than you do.

It's hard to tell mistakes apart from deliberate decisions without some indication of the programmer's frame of mind.

This is true, and it's why I want code to be self-documenting. The code has to wisely and correctly make comments unneeded; you can't just delete the information contained in a comment. The advantage is huge: you go from *two* sources if information which may contradict each other (don't tell me that you've never seen out-of-sync comments ;-) to one source which is automatically tested.

Thus, comments are only unnecessary in perfect code.

Not really true. There's no such thing as perfect code; comments are unneeded in code which tells the same story as the comments would. And such code (unlike perfect code) is not overly hard to construct.

If you choose to encode your comments using function names, rather than the features provided by the language specifically for the purpose, that's your choice;

Pardon, but function names are usually more of a language feature than comments; they're checked by the language, they're part of the syntax... Comments are just bolt-ons. And this IS the purpose of function names; but most people don't use them wisely.

but it's not yet clear to me that it is a beneficial one.

Well, I've given some reasons above. To me, the biggest two are as follows:

1. information can't get out of sync.
2. all information appears in two contexts: actual use (in the code), and prototypical use (in a unit test). Function names will look dramatically bad if they're chosen inappropriately. Comments can be ignored.

Comments can get out of sync with the code, in which case all of their "benefits" go to waste; and they can appear in only one place, and aren't checked there.

Have you ever come across (or written) any material that discuss the pros and cons of your approach?

It's a critical component of Extreme Programming, which is being discussed pro and con in a lot of places. I can also link to a few semi-balanced discussions (good paper, but biased toward the rather management-heavier SEI CMM model, which is needed for some projects). BTW, I was going to provide a con link, but the one I found is BAD -- hasty generalizations are just FLYING. Every single "danger" he cites is completely unjustified by experience and study, and many of the quotes he makes are out of context. It looks more like a Usenet post than a web article.

Oh, here's dmoz's directory page.

Anyhow, to implement what I'm describing you'll need at least Refactoring and Unit Testing from XP's arsenal. There's no reason you should have to adopt more, although once you have those in the others usually can be added at will. You also have to practice egoless programming -- don't be offended if someone changes the name of one of your functions; instead, make sure that the name is still appropriate to the use (if it's not, we have a bug being formed, since the person misunderstood the purpose of the function; you should show him a unit test in which the old name makes sense but the new one doesn't).

To me, function decomposition already serves three purposes: information hiding, code reuse, and divide-and-conquer problem solving. Adding a fourth criterion--of allowing comments to be encoded as function names--seems to raise the possibility that the criteria may conflict, and one may have to be sacrificed for another. Looking at the list, I know which is first on my chopping block.

Information hiding: true, but part of the purpose of IH is exposing only the correct information. That's part of the job of the function's name. So really, these two work together.

Code reuse: how can you reuse code if you don't know what it does? By choosing a good name, you are able to tell when things are being reused, and how appropriate they are to the situation. Note that the first time you extract a function, it'll have a very specific name (because you probably took it from a comment). The first time you reuse that function, you'll probably want to think of a little better name, and you'll probably also make some code changes -- the code worked in the one special case before, but now it has to work in two cases. As you keep reusing, your code keeps becoming easier to reuse; the name and the code keep becoming more general. So those two work together as well.

Divide-and-conquer: indeed, this is important -- but it's hard to argue that bad naming will help your strategy here.

-Billy

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Good points, again.

A function's name is not checked against what it does, any more than a comment is.

Well, yes; it is. A function's name appears in many places; if functions are supposed to have clear names, the lack of one will be very clear at a glance. A function which is being used for the wrong purpose should be obvious, and the best way to make it obvious is to make its name describe its purpose. Imagine seeing a function named 'SetFontAngle' in the middle of a vector rotation routine. You'd KNOW it was being used in a hackish way, and later font modifications could hammer it! How would you ever notice that if the function were instead named 'SetGraphicAngle' (i.e. it changed a cell named 'GraphicAngle' which happened to be shared by both vectors and fonts)?

Note that you're recommending naming functions for what they do; but the true story of what a function does is actually told by its implementation code. What a function does should be irrelevant to its use; for example, I may have a function which sets a specific bit in a specific I/O port, but that doesn't mean I name the function SetResetBitAndThenPollFor7854ChipReady. More likely, I name it for its intent: Reset7851Chip. Later on, when a reset bug is discovered, I implement the workaround without changing the name -- your function name would have to be changed to SetResetBitAndThenPollFor7854ChipReadyThenClearAll DMARegisters.

OTOH, suppose I'm writing some export unlocking code for the same chip, and step 5 of the unlocking procedure is to SetResetBitAndThenPollFor7854ChipReady. I would naturally notice your function name and would quite possibly use that function, because the name claims to tell what the function does -- but in your mind, that function is supposed to be a Reset7854Chip. What happens when the workaround has to be put in later? The only disciplined answer is that the person who originally wrote the function has to notice that it's being used elsewhere (hopefully), and rather than changing the old function, create a new function which calls the first and then zeroes the DMA registers. The new function would have the name we mentioned above, of course. Now what happens when a third programmer has to reset the chip? Which function does he choose? There's literally no way to know -- perhaps the comments in the calling functions will tell him, but that's a lot of source digging.

Code reuse: how can you reuse code if you don't know what it does?
I think now you're arguing my side. I have been advocating naming a function for what it does. You're the one arguing that it should be named for why it does it, remember? :-)

I think we're getting confused here between 'what', 'how', and 'why'. I'm using 'what' to mean 'what is intended'; you're using it, I suspect, to mean 'what is being done'. I would not use 'what' for that purpose; 'how' is traditionally used. However, I should not use 'what' for my purpose, since 'why' is less ambiguous.

So I want to name functions based on their 'why'; you want to put the why in a comment, and name functions based on their 'how'.

I hope you don't actually think I'm advocating the use of bad names.

I apologise abjectly. That was a terrible connotation on my part.

However, you were clearly saying that function naming was a constraint which might get in the way of writing good functions. If you're going to abandon my constraint because it hinders divide-and-conquer, then you should equally consider abandoning all other naming constraints.

I'm just advocating naming functions by "what" rather than "why".

I hope you see why I believe 'what' is the wrong thing for a function name to say -- at least if the 'what' is 'what this function does'.

-Billy

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Perhaps we are both mistaken about each other's position? Anyway, any misunderstanding aside, I think we probably still disagree on how to name functions. :-)

I think we're on the same page, actually. If you'd name a function resetThing then you're naming it according to its intent, not according to its contents.

However, you're also missing a point -- I'm not resetting the chip to indicate success; I'm simply setting the bit which usually means 'reset' and polling the device ready (which happens to coincide with the intended way of resetting the chip). The chip doesn't actually reset; it simply finishes unlocking its cryptographic functionality. So I don't call resetChip, because my purpose isn't to reset the chip, even though I think the procedure is the same. I would be wiser to call a function propagateUnlocking (or some such name). Therefore, when the workaround I mentioned is added to resetChip(), it breaks no code -- because everyone knows that a function named resetChip will be used only when a true full reset is needed.

My point is, as you discuss, that the programmer's intent in calling a function should be the main thing conveyed by the function's name. A function should never be called if its name fails to match your intent; if you examine the implementation, and it matches your needs exactly, you should at least change the function's name so that it fits into your intent as well as into the intent of its previous use. That way, when it comes time to modify the old function, the maintainer will know unmistakably that there's someone using the function in a way which was not originally anticipated.

Of course, if you choose to do that, you MUST also create some new unit tests for the renamed function which illustrate its new use. That way the maintainer will not only know _that_ there is a new use and intent for the function; he'll know also exactly what that new use it.

-Billy

Small steps

[ChaosDiscordSimple]

Large overhauls are usually mistakes. Details in the previous code are lost. If the overhaul takes non-trivial time, people become frustrated that two weeks ago they had a working (if problematic) system and today most of the system doesn't work.

Instead, make small incremental changes. Pick something lots of code is replicating and attempt to unify it into a shared code base. Spend some time documenting key parts of the code. Pick a particularlly hairy class or function and untangle some of the worst bits. These sorts of changes can reveal minor bugs, build up to significant improvements, and leave you satisfied at the end of the day that you improved things.

If a signficant overhaul is necessary, try to overhaul portions while maintaining the existing bits.

Compartmentalize & Destroy

[DotComVictim]

1) Identify common functionality.

2) Encapsulate in libraries

3) Be sure to extract enough generality that you don't have special case functions

4) Don't extract so much generality that functional interfaces become unwieldy.

5) Write everything in the same language.

6) Find any complex pieces or algorithms. If they can be simplified or re-written, do it. If not, save it so you don't need to debug it again.

7) Throw everything else away.

Regression test suit

[angel'o'sphere]

The first thing you definitly have to do is Setting up a test suit for regressoin testing.

For those not familair with the term 'regression test':

Program a set of so called "test drivers", programms calling your code: routines/scripts.

Define test data, either in a DB or in flat files, used by those driver programs.

The test programs and test data needs to work with the old code, of course.

As the new code should behave similar you only need to adjust PATH or script names to let the test programs work with the new code.

Plan your project by defining which test cases(test program plus test data) should work at a planned milestone with the changed code.

After making changes rerun all tests.

Well, there is a lot more you could do, but that above is minimum (basic software engineering, sorry no art involved here).

Regards,
angel'o'sphere

Little by little

[Mike Schiraldi]

Don't stop development on the old tree and shift all work to the restructuring project.

Instead, leave most of the manpower working on the old tree as always. Take a small team of your best people and have them gather input from the others, while the others keep working on the old tree. Then have the small team outline the changes that need to be made.

Work on the changes while simultaneously working on the usual stuff. Say, 90% of your manpower should do what they do now, and 10% should work on restructuring things. One mouthful at a time.

When you have a mouthful that you think is ready, branch the old tree. Merge your diffs into the branch, TEST IT, and if things seem to work, land the change onto the old tree.

Re:Little by little

[William Tanksley]

Wise, but may I suggest the obvious extension?

Don't play with code trees. Use only one "tree": the current, live one. Don't let one code base rot unmaintained while the other one is hacked to bits, untested.

Do take one mouthful at a time. Add unit tests, make sure they work. Now add a unit test for something you WANT to work, but which doesn't. Make that unit test work by implementing the feature. Release the result. Repeat.

-Billy

It is called Refactoring.

[burtonator]

Most of the computer industry now calls this Refactoring.

I would HIGHLY recommend the book "Refactoring" by Martin Fowler.

There is a number of things you should do here.

1. Document your plan and come up with an official PROPOSAL document. Allow others to comment on this document and incorporate fix all relevant issues.

I started using this under the Apache Jetspeed project and now a lot of other Apache projects are accepting this practice.

It really allows the community to become involved in your changes and encourages constructive feedback and involvement.

2. Break this into phases. You should NOT attempt to do this all at once. Each phase should be isolated and should consist of one unit of work.

Each phase should be branched off of CVS, worked on, stabilized, brought back into HEAD and tagged. You should then RUN this code in a semi-deployment role for a period of time to correct all issues which WILL arise with the updated code.

After this you can then start your next phase.

3. UNIT TESTS! If management (assuming you have management) has approved the time for this type of refactor then you need to take the time and write Unit Tests for each major component.

It is important that Unit Testing can sometimes be just as hard, if not harder, than the actual development itself.

In some situations you can avoid Unit Testing, some here are going to call me crazy for saying this but it is true. In a lot of high level applications, which are NOT used as libraries by other applications, you can bypass Unit Testing in order to increase development time. This is a dangerous practice but it is often outweighed by the extra functionality you will end up with in your product.

Anyway. Good luck!

Kevin

Re:It is called Refactoring.

[KyleCordes]

I found a couple of things strange in this post:

[unwillingness to sacrifice key features in order to make time to build unit tests for every radio button]

Personally, I do test-intensive coding because it helps me go faster (build more functionality per unit time). If it didn't, I wouldn't. I also don't see the point in unit testing a radio button.

Re:It is called Refactoring.

[KyleCordes]

Oops, posted early. Other strange things:

[asynchronous input from five different sources, where that input consists of five different complex packages]

I've found that it makes a lot more sense to focus on testing the hard stuff, like this, than simple things like the ones you mentioned. By the way, it's also quite useful (totally apart from XP) to decompose such a beast into pieces you can test effectively.

[so don't sacrifice your project and your customer on the altar of XP]

I've never heard of this happening. Can you point to any project which did XP (most or all of it, not just talked about it) and resulted in an unhappy customer for that reason?

I've mostly seen people do little bits of XP (ignoring most of it), fail, and blame XP. A typical story is "we did XP, but we didn't do much testing, we didn't pair program, we didn't have a customer on site, we didn't integrate very often, we didn't track our progress, and our project went badly, therefore XP is bad". Weird.

Re:It is called Refactoring.

[KyleCordes]

[not the fake customer on your XP team, the real customer holding the checkbook]

XP says to have the real customer. If you have a fake customer, it's not "pure" XP. Many people who like XP but can't get a real customer use a "proxy customer", but they (hopefully) admit that they aren't really doing XP per se, just something similar to it.

You are of course 100% right that satisfying a fake customer is irrelevant if you don't satisfy the person with the checkbook! In my mind, customer equals check-writer.

[There's a great deal of XP fundamentalism out there, mostly on the part of development managers]

That actually doesn't match my experience at all. I see many developers who are very excited about XP, but many development managers seem to like some of the ideas but be *far* from "fundamentalists" about it. Rather they want to immediately disregard major portions of it, typically.

[about as atomic as you can get in an asynchronous system]

I've built a number of asyncronous systems, and agree they can be challenging. I do believe it makes testing even more important. If I was building the system you describe, I would probably insist on testing that piece.

Regarding choosings unit tests, the frequent advice is to test everything that could possibly break, which is pretty vague - but it definately includes this async beast.

[Well, I can't point to any project that managed to successfully implement "pure" XP]

[didn't have enough staffing to constantly pair program] etc.

My last project didn't do RUP. And it didn't do Scrum, although we no doubt did bits and pieces of each. I don't use it as an example of either of those methods succeeding or failing.

Your projects didn't do XP, it did bits and pieces of it, omitting large important portions. So why would it indicate anything about XP? Because the people talked about XP a lot while they did something different?

[The Cult of XP claims to have the one right and true solution]

I've read thousands of messages on the XP mailing list, I've been to an XP conference, I've read a few of the books, and I've seen presentations by most of "big names". In the process, I have not seen such claims.

Who is making this claim?

Complete rewrite necessary or a waste of resources

[beamz]

Not too long ago a link was posted to an interview with Joel Spolsky who used to work at Microsoft.

His comments about code reworking and rewriting have a lot of insight in them.

Here are some quotes from his article:

SMS: Joel, what, in your opinion, is the single greatest development sin a software company can commit?

Joel: Deciding to completely rewrite your product from scratch, on the theory that all your code is messy and bug prone and is bloated and needs to be completely rethought and rebuild from ground zero.

SMS: Uh, what's wrong with that?

Joel: Because it's almost never true. It's not like code rusts if it's not used. The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. There's nothing wrong with it.

The point is this... The benefits of spending time rewriting code completely may be a waste of the companies resources, this is for you to determine.

His interview is here:
http://www.softwaremarketsolution.com/index_2.ht m

and his site has more information about the concepts here:
http://joel.editthispage.com/articles/fog0000000 06 9.html

modularity/incremental rollout/unit tests/iterate

[tim_maroney]

I second all that has been said about making sure that you really need to do this and that it is worth the time and risk. One sign that you may need to do so is an excessive reopened bug rate, where fixing one bug often creates another bug due to side effects and component interactions. If you decide that it is, then the three keys to success will be modularity, incremental rollout, and unit tests.

Modularity is probably what you're already thinking about. Go over the old code base, in a code review, and find where the same thing is done over and over either with copy-and-paste code -- the bane of crap engineers -- or with different code that serves the same ends. Look for repeated sequences in particular. Create a new library that encapsulates those pieces of code.

Incremental rollout is vital. Only replace small parts of your system at a time, doing complete retests frequently. Don't write a new encapsulated routine and then roll it out to each of the three dozen places in which it appears in the whole code base. Write the whole function library, with unit tests, and then start applying it to separable modules one by one, retesting as you go. Otherwise I guarantee the whole thing will fall apart and you won't be able to tell why. Ideally, you might set a threshold on the rate of replacement of old modules and work primarily on creating new modules with the abstracted logic.

Unit tests are crucial because, as noted, the messiness of your old code probably conceals a lot of necessary logic. We had this great phenomenon on Apple's Copland where people who had never used the old OS managers were rewriting them in C or C++ from the assembly source. When they saw something in the assembly they didn't understand, they just ignored it. Guess what -- the new managers didn't have any backwards compatibility. The only answer to this is to have a thorough unit test for any module that you replace, against which you can test the new version. This also confers other quality benefits, but during a rewrite it's critical.

Finally, once you have replaced a significant number of your modules, you will find that new levels of abstraction appear. The average size of each function or method will have shrunk considerably, and now it becomes possible to see new repeated code sequences that were not visible due to the old cruft. Move these into your new library modules and start using them in continuing replacement work. In addition, start going back -- slowly and incrementally -- through the already converted modules and replacing the repeated sequences with calls to the new abstractions.

Finally, figure out how you got into this mess in the first place. The worst programmer habit I know of is copy-and-paste coding instead of using subroutines. You can tell people not to do it, but some always will. Those people should be bid farewell -- you can't afford their overhead. Other common problems include lack of planning and review, a code first and think later mentality. Start moving your organization up the levels of the CMM and you may find that you wind up with fewer modules that need replacement.

Hope this helps.

Tim

I'd like some info on this as well...

[josepha48]

in particular how do you write a large program? I work on a program that has over 3000 files that are over 30 to 300 pages each. The software does alot for and has been around for over 30 years and has just had code sloppily added to it. So how does one go about rewriting an application that noone knows all the functionality of?

Go slowly young Jedi

[0xA]

This is a redundant post but I think it's imortant enough to be repeated.

Don't set out to refactor your codebase as one big project. Try and split up the code by functional areas and take them on one at a time. Now, this doesn't really work too well most of the time. You're going to run into way too many places where things are interdependant but try anyways.

If you can, resist changes to your database schema while you are refactoring code. Having both of these thing happen at the same time is pretty scary.

You say CGI so I asume you are using PERL, look into OO PERL, it's worth it. Even if you don't want to go OO all the way the 2 big things that can make your life easier are packages and layering.

Using packages, especially for things like DB access can save you tons of time and headaches. You have one place where you run a query and build a hash, all of your code calls it when it needs the data. HUGE adavantages here.

Layering your design is helpful ass well. I've found that you can do a lot of good if you have designed Data Access, Logic and Presentation layers seperately. All each one of these layers needs to do is take the hash ref passed by the other layer and do X with it. You can rebuild each layer at will as long as the data structures passed between them don't change.

Solicited suggestion

[ciurana]

chizor wrote:

My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code)... What suggestions might readers have about tackling a large-scale retrofit?

My advise to successfully accomplish the changes:

• Where possible, use Python or Java instead PERL
• If there is a line-of-business piece of code (i.e. if your company's bread is on the line when this fails) make sure you can roll the original code back if the new version doesn't perform
• Divide the programming team in groups of 2-4 people
• Use XP techniques for development
• Have an external group evaluate progress to keep people honest; this group is responsible for testing that functionality meets or exceeds that of the original system

I led the development and migration of some very large mission-critical systems in my career. Too many programmers making decisions on-the-fly, totally centralized management, or a "leave the technical folks alone until they're done" attitude are sure recipies for disaster.

Good luck with the changes.

Merry Christmas, and God bless us everyone!

E

Re:Solicited suggestion

[m_ilya]

Where possible, use Python or Java instead PERL.

Indead. I'd like to extend this advice:

• Where possible, use Perl instead of PERL.

DON'T DO IT!

[mcrbids]

As posted elsewhere, re-writing is painful, and generally NOT A GOOD IDEA!

I currently maintain a code base of around 120,000 lines of php and html (written by myself in a long, hard year) and have had to "retrofit" it a few times.

I find that when it's time to do an "over-haul" it's generally best to:

1) Pretend I know nothing - redesign from scratch. Write out a spec with flow charts, DB table definitions, etc. - make it VERY DETAILED. Spend lots of time at it. More time spent here saves even more time later.

2) Ignore your spec. (See step three)

3) When a bug comes up, or new functionality needs to be added to the codebase, refer to the spec built in 1, and build to it, and then put in compatability wrappers to work with the existing codebase.

Make these compatability wrappers log their calling in some way, based on a global variable. This allows you to see when they're no longer needed simply by defining a variable in a config file and waiting a while.

4) You'll be slowly bringing the application up to the new spec - eventually you'll reach a point where it's easier just to bring the remaining pieces up to snuff than to build more abstraction wrappers. When you get to that point, you'll find most of the work is already done, just finish it to the spec and remove the compatability wrappers.

This can still be a painful process, but at least it isn't a "gun to your head"! This allows you to regression test your work as it's done, resulting in a more stable deliverable, and you can still meet clients' needs in the meantime without making them wait 6 months while you re-write all your stuff.

Hope this helps...

Step #1

[Sanity]

...don't use Perl ;-)

[-1 Flamebait]

You have a Big Ball of Mud

[mahlen]

First, read this essay: The Big Ball of Mud. It is an interesting look at why, when we all know that spaghetti, gnarly, twisted code is bad, that it happens anyway (hint: it may mirror your understanding of the problem).

Ignore the "don't touch it" naysayers. Even before it's done, it'll be much nicer code to deal with. You can make decisions with less nagging doubts. You'll code onward with gusto. You'll be able to accurately predict the names of methods without looking them up.

Test the current state of things at all points through the process. I'm hoping that you have lots of automated tests you can run everytime code is checked in; if not, make them FIRST, before the overhaul. You are majorly diverting the intent of the code at hundreds of points; you can run astray in so many places that the above naysayers would be correct. Constantly assure yourseleves that the code is working. Go out of your way to ensure that the code is buildable and runnable, even to the point of writing scaffolding you know will be soon thrown away.

Burning a little incense every day in obesience to the Gods can't hurt, and will make the room smell nice.

mahlen

Shantytowns are usually built from common, inexpensive materials and simple tools. Shantytowns can be built using relatively unskilled labor. Even though the labor force is "unskilled" in the customary sense, the construction and maintenance of this sort of housing can be quite labor intensive. There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth. Shantytowns emerge where there is a need for housing, a surplus of unskilled labor, and a dearth of capital investment. Shantytowns fulfill an immediate, local need for housing by bringing available resources to bear on the problem. Loftier architectural goals are a luxury that has to wait. -- from "The Big Ball of Mud"

Quick suggestions...not so quick solution

[runswithd6s]

As everyone else should be telling you, your existing codebase is a very valuable resource. It's been tested and debugged a number of times. It's mature, it's stable, it's there. Don't throw out the baby with the bathwater.

Now, on coding standards and how to incorporate them into a legacy project. Your concern is NOT format. The format of your code, such as indentation, spacing, etc, should be the least of yoru concerns. Everyone has their own style, but there are wonderful tools that you can use to force everyone to a single style, ident and astyle just to name a couple. Use a wrapper script to the CVS (or similar system) on checkins. Force the code through an automated cleanup. check the code back out and make sure it compiles/runs as expected.

What you should worry about is how much your design team has embraced the "black box" design principal. Parameters go in, results come out with no "side-effects" that impact the remainder of the code. Make your code re-entry safe, i.e. stay away from globally scoped variables as much as possible.

Someone's going to give you the whole OO-Design sales pitch. Yes it's nice on paper, but don't sell out because something looks nice on paper. I learned this the hard way. I have a tendency to overdesign things. When OO, this gets to be really scary. I waste my time writing object classes for "everything" instead of simply designing the software to its functionality spec. Make things more "object oriented", "functional", or "blackboxed" when you find yourself repeating code elsewhere in the application.

Don't spend a lot of time with naming standards such as Hungarian, Modified Hungarian, etc. Find a style that you and your team is comfortable with for the Interface API level. Below the Interface API, be more lenient. It's likely that portion of the code will undergo many changes anyway.

And most importantly, document! This is the singlemost important issue of any coding project. Either force your developers to write docs as they go, use embedded documentation solutions, or hire a techwriter to follow you and your team around for a few months. Documented API is the quickest way to start someone off in the project, and a great way to keep track of the flow of the program.

Does it work?

[litewoheat]

If it works don't break it. Pointless rewrites do nothing but feed the programmers ego and knock the company out of business. Its happened over and over and over and over again. Progress, not regress.

My suggestions

[Pinball Wizard]

My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code

First of all, I think its important to realize that you have a medium-sized website and not a big software project. Therefore, some of the above comments recommending refactoring, UML, and eXtreme programming may be a bit overkill.

Web programming != software development! Its usually done at a much faster pace. Even if an object-oriented approach is taken, you are still probably talking about simple function libraries rather than complex C++ or Java classes. Again, overkill.

150 files is still a small enough project to be managed by one or two decent coders. Actually, I just looked at the amount of stuff I've written over the years for my online bookstore and its more like 500 files and over 4 megs of code. I don't feel like its too much of a job to manage this codebase by myself.

So, here are my recommendations.

You probably have gotten better at programming since the time you started your project. Take a few of the most recent CGIs you have written and compare them to the first ones you wrote. You just might notice a glaring difference in the quality. Also, the first pages you wrote are likely to be among the most important in your project, yet they are also likely the worst quality-wise.

Regardless of what language you program in, I think its important that you can tell whats going on in the program by reading the comments. If a manager can understand what a program does by reading the English bit, there's a good chance other programmers will be able to jump in and help as well. One specific rule I also follow: if you do regexes, say IN ENGLISH what those regexes do. I say this because regexes are one of the hardest things to read.

Look for any code that can be "factored out" of your scripts and put those into function libraries. Then include those in your program. The only problem with this occurs when you have huge function libraries that slow down your scripts when you include them. In that case you would logically separate your functions into different files. I have included very common functions in different include files, so I can make the actual code compiled or interpreted as small as possible.

Consider using a flowcharting tool as an aid to programming and/or documenting your code.

Standardize how you name variables and functions, write comments, identation, and spacing.

Be sure and include the date you write your scripts in the comments, in case the filesystem wipes this out.

I'm sure theres other things I've left out, but following the above guidelines have helped me do exactly what you are trying to do: manage a growing codebase. But don't forget, this is web programming, not rocket science, and some of the above suggestions may be more trouble than they are worth. Keep it simple.

IBM's (re-) write of OS/400

[Degrees]

IBM was faced with a simlar problem when they came out with the RISC version of the AS/400 hardware. They needed to re-write, but they had to maintain absolute compatibility. I think the articles describing this were in Dr. Dobb's Journal (although I might be wrong on this). Unfortunately, I do not have URL's for this.

One of the success factors they found was documenting the interfaces for each and every call between modules. The documentation turned out to be excruciatingly precise - but this led to zero ambiguity (and thus 100% interoperability). It also required meetings (sometimes arguments) between programmers to hash out what was actually going to happen. Another factor was that they decided to allow zero 'overloading' of functions by different modules. A programmer was not allowed to duplicate someone else's work, nor create a second, incompatible version of a function provided in a different module. If the function was provided by someone else's module, the programmer had to call it (properly). The result was that they reaped the benefit of object oriented programming - reuse and refinement of modular libraries.

It would be better if you could get the real scoop from the real programmers - but this might give you something to think about.

Microsoft, paragon of quality code

[dmoen]

This advice makes sense if you are embedded in a culture like Microsoft, where you have a huge quantity of legacy code, and a really shitty software process. Microsoft has no record of what design decisions were responsible for making their code the way it is; they just have the code itself. They can't change it because they don't know what they might break by doing this.

If you enforce a good software process for every line of code that is written, then you have more flexibility. For example, suppose that you use Extreme Programming. Then you will have a unit tests for every bug that was fixed. Each time you re-run the unit tests, you find out if any of your previous bug fixes have become undone. This gives you the freedom to refactor your code whenever it needs to be refactored.

Doug Moen.

Re:Microsoft, paragon of quality code

[William Tanksley]

Microsoft doesn't have quality code, but it DOES have reams of documentation. They have, and follow sporadically, a really strong process -- in fact, a huge number of really strong processes.

In many of their products, the design decisions are documented, and documented well.

As with all documentation at that level of detail, good luck finding the right page...

Other than that, you're right.

-Billy

Re:Microsoft, paragon of quality code

[dmoen]

Interesting. I think that a software process can be "strong" (tedious, bureaucratic) without necessary being "good" (ie, effective). Does Microsoft's current process actually deal with the problem of not being able to refactor your code without losing all of your bug fixes? Keeping all of your design history in labrynthine documentation that is difficult to find anything in is expensive, but doesn't solve this particular problem.

Doug Moen.

Re:Microsoft, paragon of quality code

[William Tanksley]

Well said, on all points. Microsoft's problem is that they have too many processes, and IMO most of them are too heavy (and therefore they aren't really being well-applied). So they don't have A current process; they have many of them.

I personally have a bias against strong processes. I do admit that they can work, and work well; but they also cost a LOT, in every way; it's very hard to tell whether you're applying enough of the process to make it work. I'm definitely an XP fan; there, your process is light (although it IS hard), and feedback is immediate and automatic.

-Billy

Perl is a *tool*

[tmoertel]

Perl is for system administrators and system administrators-cum-developers, not real software development.

Baloney.

Contrary to what lots of people around here seem to think, especially those who like to make wide-sweeping declarations about what things are and what they are not, Perl is a tool. Nothing less, nothing more. Like all tools, it can be used to create well-engineered systems, and it can be used to create crap.

The community that grew up around Perl is all-welcoming and generally free of elitism. That's why a lot of newbie programmers and "system administrators-cum-developers" use Perl -- because they can without getting crapped on by others who think they know better. As a result, there is a lot of ametaurish-looking Perl code out there, but that's not a result of the language, that's a result of the all-inclusive set of people who use Perl.

Let's be clear: If you write code in any language and the code sucks, it's your fault, not the language's -- the language is just a tool.

Don't blame the problems of programmers on their tools.

Learn from those who have succeeded at this...

[William Tanksley]

Some here are warning you that major changes always require a total rewrite; yet in real life, total rewrites result in inability to compete (look how long the Netscape rewrite paralysed Netscape, unable to meet Microsoft's challenge!). There's some good discussion of the danger of rewriting at a former MS software engineer's site, and some limited advice about how to get away without doing it.

But you've decided to rework rather than rewrite, you say, so I have no doubt you'll ignore the naysayers here. So what CAN you do? After all, as you recognise, reworking is dangerous!

The following rules have worked for me; I've refined my own experience with advice from Fowler's Refactoring, a book as useful as Design Patterns, and with study of Extreme Programming, a design methodology forged in the traditions of Smalltalk, and in the knowledge that maintainance, the most important and expensive part of software engineering, is also the least studied.

First, do the simplest thing that could possibly work. Don't EVER take your program out of commission for more than a day; make sure it runs at the end of each day. If you're doing something and at the end of the day your code base is broken, STRONGLY consider throwing away your changes and going back to the design stages.

Second, rely on unit tests extensively. Start every change by writing as extensive of a unit test as possible. Unit test every function you touch, BEFORE you touch it, and after. Unit test every change you make, and run the unit test BEFORE you make the change to ensure that it fails (i.e. it detects the change). Write your unit tests BEFORE you write code, whenever possible; you'll objectively know your code is done when your unit tests pass.

Third, don't design too far ahead; you don't know what tricks the old code is going to throw at you. Implement one feature at a time, bringing the code into compliance. Once everything has a unit test (thanks to your following the above principles), THEN you can safely embark on larger design changes -- and in the meantime, you have working code with new features, a win even if your customer/boss/manager decides not to continue.

Fourth, don't be afraid to redesign your own code. The stuff you wrote has more tests, so it's safer to change, but it's more likely than the old code to lack some critical understanding only age can give.

Fifth, use the principles of refactoring. Whenever possible split each code change into two parts: first, a part which changes the structure of the code without changing its function (and which therefore allows you to run the same unit tests); and second, a part which uses the new structure to perform a new function (thereby requiring new unit tests).

Good luck. If you want more advice, read up on Extreme Programming.

-Billy

Re:Learn from those who have succeeded at this...

[startled]

Much agreed. We've found, though, that sometimes it's tough to write as many tests as you really should. Before you even start changing code, write a pile of tests-- it's best if you can just allocate a week or three for it.

Beyond that, it's tough to stay motivated about writing tests; people want to write new code, not test the old. So mix it in-- we've decided to require a test to be checked in with every bug fix, figuring that if it broke once, it'll break again.

While I'm at it, I totally disagree that a total rewrite is always the wrong choice-- you just have to keep significant resources going ahead on the old stuff. If your old stuff really was that bad, the new will catch up with it without too much trouble. If it doesn't, well, you didn't really need the rewrite.

Re:Learn from those who have succeeded at this...

[William Tanksley]

Well said. It really, really helps, in my experience, to develop the habit of coding to the tests.

That is, you first write the tests; do the design while you're writing the tests (and have the tests express the design). Then write the code, trying constantly to run it against the tests. AS SOON as all your tests pass, commit the code; you're done. Don't refine anything until you have written a test which fails under the current code.

You see what happens? Pretty soon, instead of tests being your enemies which break your code, they're your friends which tell you when your job is done. You'll quickly find that they're reliable friends -- and long-lasting ones, too.

-Billy

Perlmonks.org

[consumer]

You might want to reconsider perl as the language of choice for a large scale application. I realize I'm posting this comment to a Perl system, but Perl hangs together like an immense kludge of a language.

What a monstrous Christmas troll you are. What qualifies you to make this judgement? Perl, like any other mature language, has people who write kludges with it and people who write clean, elegant code with it. Your lousy Perl code is not indicative of a language problem.

That said, you're probably stuck with it, and AFAIK, you may be forging new paths in programming for reusability by applying the above concepts to Perl.

And this shows how much you know, since the Perl community is full of activity around design patterns, refactoring tools, unit-testing, and other practices which are in favor among experienced people trying to write solid, maintainable code.

My suggestion for those who are looking for actual useful advice rather than this kind of "throw away all your work and learn Java" crap, would be to head straight for http://perlmonks.org/ and read up. There's tons of advice there for serious Perl coders. You would also do well to start reading the mod_perl mailing list, which often has informative discussions about these issues.

Re:Perlmonks.org

[benedict]

I know some serious perl heads who say they
wouldn't suggest using it for a large project.

Minimise Untested Documentation!

[William Tanksley]

Depend on tests, not documents. Documents lie. Tests don't.

Use as little documentation as possible, BUT NO LESS. (In other words, for heaven's sake don't ever try to get away without any documentation.) Documentation should state fundamental premises -- things like "The customer wants X." and "This code checks that I'm fulfilling the customer's requirement of X." Documentation should not state intrinsic properties -- the statement "this code does X" should be made as a test, not a document.

-Billy

Re:Minimise Untested Documentation!

[Arandir]

I'll have to disagree. Document as much as possible, BUT NO MORE! But this documentation must be meaningful and relevant. Otherwise it is worse than useless.

Document every function, listing the purpose of every parameter and the meaning of the return value. Document why you are doing something if there is more than one way to do it. If a section of code fixes a bug, document what it does and do not just document a bug number. Use self-documenting code whenever possible (ei. name your variables and functions meaningfully). Use a document generation tool if possible (javadoc, doxygen, etc). Write the user docs at the same time you're writing the code. Incorporate the user docs into the code if at all possible.

Here is a bad comment: // store x2 in fu
Here is a good comment: // save the index because we will use it later
Here is a bad comment: // this is not meant for you to understand
Here is a good comment: // please see Smith's "The Black Magic of Filesystems" for details on this algorithm

The most important part of commenting is realizing who is going to read it. It may be you. But in all likelihood it will be someone you never met long after you have left the project or even the company. It may be code or design reviewers who don't know programming but do know how to block projects they can't understand. If it's Open Source, it may be some brilliant programmer wanting to fix a bug but without the time to puzzle over your constructs.

In every code review I have ever been in, someone has made some silly assumption about the code, with the final recommendation that that section of code be commented better to avoid future silly assumptions.

A Code Nazi Disagrees

[dmoen]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

If you need to make big changes to the code, then you are already hosed. If you have no record of what bugs you fixed, and no way of testing if those bug fixes are still working after you make code changes, then keeping all of your spagetti in place is no guarantee that you won't re-introduce bugs later on.

The Code Nazi approach is to write a unit test for each bug you fix. And you have an easy way to re-run all of your unit tests. After you make a big change, you can test if all of your bug fixes are still working. And now you have the flexibility to refactor your code whenever you want. Which means you can keep your code base clean and elegant, and you'll never reach the crisis point that this group has reached. This is the approach advocated by Extreme Programming, as well as by other software disciplines.

Doug Moen

Hardware isn't cheap

[drig]

I agree with your point and I'd like to expand on it.

Yeah, adding a gig of ram or increasing the CPU from 800Mhz to 1.8Ghz isn't expensive. But, if you go beyond what a singl-processor machine can handle, you run into another host of problems.

Adding a second CPU means *MUCH* higher chances of race conditions and other threading bugs. If you know you're coding for a single processor, you can often use a single-threaded model which makes life so much easier.

Adding clustering brings a whole hoist of data synchronization problems. It's *ALWAYS* easier to code for a single-machine than to code for a cluster. There are tools you can use to make shared memory easier, but those often flood the network.

Re:Hardware isn't cheap

[Arandir]

If the machine is obsolete, I'll upgrade it. If it can't meet the demand that the clients put on it, I'll upgrade.

But I don't upgrade the machine just because I'm a lazy programmer.

Now on to the rant: I've been with the PC since day one. I learned how to program on a machine with only 8K memory. I remember when if you couldn't fit a program into 64K you were a wus. But over the years we have been taken over by lazy programmers. Their programs get larger and larger. Bloat is king. And the needs of a GUI is no excuse. I remember running a full Motif based GUI desktop on 640K (Geoworks).

And don't talk to me about speed. You're obsolete 800Mhz machine was state-of-the-art the last time you changed your underwear. You don't throw away last year's machine just because there's a new review at Tom's Hardware.

While we're busy upgrading our machines like there's no tomorrow, our landfills are filling up with perfectly functional machines, and leeching enough toxic heavy metals into the ground water to make Love Canal jealous. People like you are the vanguard of the throw-away society.

If you can't dish out webpages on a 800Mhz machine, it's time you looked for a career more suited to your abilities.

Don't rewrite, and don't refactor

[splattertrousers]

Don't rewrite your code. That's a waste of time.

Don't spend a big chunk of time refactoring it either. Waste of time too.

Instead, make slight refactorings as you go. But make sure you are doing what you are really being paid for: implementing business value.

And you'll find that you'll have much more courage to refactor if you have a full set of automated tests, so maybe you should work on tests first.

Having done complete redesign of dynamic sites

[f00zbll]

I've rebuilt dynamic sites from scratch twice. First and foremost, if you're rewriting because of serious scalability or design weakneses, then it is unavoidable. If it's just to get rid of annoying things, then I would say don't even try it. I consulted at a fairly big E-Commerce site that was crawling and couldn't handle the traffic. The original site was built by a programmer who scaled examples provided by MS. After it was done the whole site was a dog and would crash constantly. They finally brought in a programmer who was able to rewrite parts of it and make it work. After 7 months of intensive work, they 2 people stabilized the site. They decided to completely rewrite the site and I was contracted to help.

In this particular case, it was necessary because the site was right at the max. If the traffic increased, it would kill the site. Since it was an E-Commerce site, rewriting it was fairly straight forward. The old code kept running, until we were able to finish the new system and make sure it was stable and ready.

As a consultant, one of the most important aspects is detailed documentation that explains both the high and low level details. Often I will include very specific details about why a design was chosen and what limitations it has. When applicable, I will also describe how to extend, or modify the code to support additional features. This means you spend a lot of time doing documentation, but it forces you to think about a design more thoroughly and will expose weaknesses. Always keep an open mind and never fall in love with your design. There is no right way to build something, only right for the situation you are given.

Don't break the #1 Cardinal Rule

[Pedrito]

Obviously you didn't read this slashdot article about Joel on Software.

What is the #1 thing he says causes software companies to fail? Rewriting from scratch!!!

I won't deny that there are times it has to be done. Joel points out some of those time (and yours isn't one of them even from the little you've written). Ours happened to fit everything he said, and we did rewrite, and not only was it the right thing to do, it was the only option for us, but the company just barely survived the process.

Don't take that lightly. I speak with experience and Joel is right. You don't rewrite from scratch unless there is no conceivable alternative. Joel describes it well, so I won't "rewrite" it. You can just click the link and read it yourself.

Paradigm Caution

[Tablizer]

There is NO decent evidence that OOP factors better, or does *anything* else better than procedural/realational programming (or other non-OO paradigms).

IOW, don't switch paradigms just because something is in fashion right now.

Look carefully before leaping into something that may just turn out to be a fad or only shines in specific domains.

php project mngt app

[gol64738]

a good project management application is important for any development team. usually, these are hard to come by unless you plunk down $10,000 or more, although these come with a gazillion features that you probably won't end up using.

i discovered a new tool on sourceforge which is an open project written in php.

i'm impressed with it. the code is also well documented.
the homepage can be found here.
i recommend checking out the screenshots as well.

99% Planning, 1% Coding

[freality]

When you first design and implement some module, a
lot of time is involved in cycling between "ok, I
know what to do" and "huh, maybe not". I've found
this crucial, esp. in team work, in order to gain
a good conception of the scope of the task. Also,
many external issues, e.g. how the module interacts
with the system, efficiency, etc. that aren't pure
functional issues, are first grappled with here.

Refactoring is different from this, in that you're
probably very comfortable with the "state of mind"
of the code. Instead of creating, you'll be
clarifying. So, most of the refactoring is in
your head (99%). All the external issues have been
addressed before (or else this probably isn't really
refactoring), so just work at a white board with
your team until writing the code will basically
be transcription (1%).

I've found this to yield the best code.

Joel Spolsky's Take...

[ijx]

Joel Spolsky has an article up on his blog site that speaks to this point.

He uses Netscape's decision to rewrite Netscape 6 from scratch as an example, and expands upon many of the points mentioned above.

Rewriting from scratch

[horza]

I have just carried out a similar project, and one thing I learned to beware of is don't suggest any timelines until you know *all* the functionality. You may think you've explored the site but there may be a lot of hidden functionality you don't know about, eg xml exports to affiliates.

I disagree with the "code doesn't get rusty" comment. Often there is a lot of pressure to get a project out the door and the initial launch version contains many compromises the designers didn't want to make. I don't think there is any project I could look back on and think "hey, the next time I would do it *this* way". People learn from other people's (or their own) mistakes, it's how we move forwards.

In response to the person that said don't rewrite as you will have to rediscover all the little quirks that the old site fixed: by doing a rewrite and documenting the little quirks as you find them means you will have a record of them rather than staring blankly at a screen of code wondering why on *earth* the previous coder did *that*. By rewriting from scratch you won't miss any quirks, which then come back and bite you inevitably at the least convenient time.

As for the Perl vs Java vs PHP debate, I'd say use the one you all know best. No language is any better or worse than any other. Some have libraries to make life more convenient, others have cleaner OO support, but in the end if you all stick to the accepted coding standards then half decent programmers can write maintainable code in any of the languages. Just bear in mind that your replacement will need to to know the same language so don't pick something obscure or dying where there will be no pool of available talent.

Phillip.

Unnecessarily long variable names

[i_am_nitrogen]

I saw a quote once that said something to the effect of you can always tell a computer scientist's code because he won't have any comments: his variables are his comments (something like that, not sure of the exact words). While descriptive variable naming is important, excessive variable naming is annoying. For one thing, it's a lot easier to type no_acc_click() than it is MakeNotAccidentallyClickable(). Comments are necessary. When code does anything of even a slightly complex nature, there's no such thing as intuitive code. There will always be a need for comments in anything beyond the simplest programs. The best code is properly indented, case is consistent, variables have descriptive but not lenghty names (i is sufficient for an iteration variable, malloc is sufficient for a memory allocation function, etc.), a description at the beginning of the function of all expected uses and known bugs, and last but not least, comments of each change in the code, ChangeLog, and CVS logs.

Re:Unnecessarily long variable names

[Dwonis]

The best code is properly indented

I think this point needs clarifying: If a project is using one indentation style (this includes tabs vs spaces), for $DIETY's sake, use it!

Also, if you are selecting the coding style style yourself, use K&R style. I don't care if you think it's not 100% logical -- it's readable, and every programmer worth his/her stuff is familiar with it. The only deviation that is really tolerable is one particular hack on function declarations (i.e. some projects use the hack so the regexp /^funcname/ will work, and this is usually OK with most people).

Yes, I'm an asshole, but if you follow these rules, you'll find that your successors will curse your name much less.

Re:Unnecessarily long variable names

[Anonymous Brave Guy]

For one thing, it's a lot easier to type no_acc_click() than it is MakeNotAccidentallyClickable().

For every time you type a variable name, it will be read hundreds of times. MakeNotAccidentallyClickable() isn't a great name for a function, but its meaning is clear and unambiguous. It's infinitely better than the meaningless no_acc_click().

Comments are necessary.

Yes, but far less often if your other names are well thought out.

Re:Unnecessarily long variable names

[p3d0]

Here are my laws of naming:

1. Names should be as long as necessary to describe what a thing is.
2. The shorter a name is, the better.

The ability to satisfy these two rules simultaneously is a symptom of simple, clear abstractions.

Ahhh Refactoring the Sport of Kings...

[j-jahnke]

I want to start by saying, before you do ANYTHING make it a priority for your team to start writing test cases and implementing them. If you are going to change your system you want your tests to find the problems not your users. Once you have a test suite that effectively tests your system (you will be surpised how quickly these tests will lead to enhancements in your software BTW) then you need to decide if you really want to maintain two seperate teams.

One team will be doing maintainence on your old system that is in use. You can't not keep this team in place, you have to keep customers happy while you do the new thang. You need the second team developing the new sytem (their job will be difficult with good tests, but impossible with out them.) Without tests they will have something that looks kinda like the old thing, but behave differently in a million small ways, some without a doubt will be better, many will not.

The other thing, and more sensible IMHO, is to tell pairs of programmars, to spend some hours each week working on the system together. Pairs becuase this is an excellent opportunity for people with deep knowledge of their systems to share it with other memebers of the team as they ferret out ways to refactor the code so more of it can be shared.

One day a week you can have a team working on some part, documenting spaghetti code, or pulling out examples of where code was cut and pasted and create a function which is called from both places. The stated goal should be fore each team to remove some number of lines of overall code from the project each week.

Remember the metric is the more lines of code you have the more errors you have, make it a mandate for your team to be removing code as well as adding, actually on our team we don't brag about how many lines we added in a week, we brag about how many we removed.
Anyway, the idea is by getting teams to do pair programming looking a specfic areas with good tests you can whip your code into shape pretty fast, and spread the knowledge of the system around as well. AND instead of a jarring change when you unveil the new system you can have gradual changes as fixes are tested and added.

Obviously if your overall architecture is broken and can't move forward this won't work. But it seems to me you are not fully convinced of that fact yet. At the very least by having teams look at code they may not be overaly famililar with you might find that team members can cross pollinate and share good ideas.

Jer,

It's called depreciation

[JohnsonWax]

There's a concept in the world of stuff you can touch called 'depreciation'. Why the software world hasn't caught onto this is beyond me, but from my experience it seems to apply reasonably well.

The value of your code goes down in value over time. Now, don't confuse that with the value of your design - your design could be grand, but the code is part of your equipment, like your hardware. Depreciate it over time to reflect the increasing cost of maintenance and integration costs in a migrating business.

Reworking your code allows you to make adjustments to your design to reflect a new environment, or to move away from languages/APIs/toolkits that might be hard to maintain.

I depreciate my code over about 3 years. It's all modular, and I replace code about as frequently as I add. A number of years ago, I had tons of bandwidth but not much CPU power, so I tended to push data rather than compute. The reverse is now true, so I made some design changes as part of a standard rewrite - no need to wait until it broke. Overall, most of my code hasn't radically changed in it's design, but it has been rewritten several times. I've had code cut back to 10% of it's original size by adopting a new toolkit, etc. I've made it more robust, faster, cleaner, better documented. I can't think of a case where it's gotten worse, and I can't think of a case where the rewrite took much longer to write than the original code - and more often than not took much less time. It's worked well enough that I've added a considerable amount of functionality, but spend no more time reworking code because it becomes increasingly efficient and is never too far removed from future additions.

Many people suggest that code rewrites are a waste of time, but it's a maintenance function. People that only budget time to write new code often find those extra work hours devoted to maintenance. Budget it in - and the best way is through rewrites.

Some advice:

[Anonymous Coward]

I have been through this before - with a different language, but the same principles apply. Here is some adivce, and some ideas:

1. Don't do a 'big bang' rewrite. A Big Bang rewrite is one where you totally rewrite all the code, but have no functional - or very little functional difference.

2. Do refactor or rework small portions of the code. Work on the code as you implement new functionality that the system needs. In other words don't waste time on rewriting working code unless it needs additional features or is broken.

3. Do Introduce Unit Tests and Acceptance Tests. This means you will know quickly when changes you make break code. Don't do a massive "unit test" building exercise however. Write a Unit Test for every peice of new code you do, and every peice of code that needs fixing.

4. Identify "Problem Code". Problem code is the bits which seem to take up most of the maintenance time. Identify that code, and try to work out ways to fix it.

Note : Some people think they are 'clever' by writing complex code which is hard to understand, but may perform better etc. This is the kind of code which usually ends up being a problem. As a professional programmer I try to be clear to understand - not only for the benefit of others, but for my benefit when trying to maintain it.

5. Read about how to manage projects. Extreme Programming is popular now, and I can say that it has some very good ideas. However don't become religious about it, and take a look at some of the 'classics'. I can recommend:

- The Mythical Man Month (tad dated, but a classic).
- Code Complete (also a little dated, but still good advice)
- Extreme Programming Explained

To take a look at my Open Source Resource, which has many articles around the subject of project management, take a look at:

Devcentre

I have to agree with the concensus...

[Codifex Maximus]

if it works... don't fix it.

If you feel you have to fix it, then prioritise the most problematic parts and fix them according to a set plan/policy. Use a naming and calling convention. Break functions that do more than one thing up into component functions that can be tested, verified and reused by other parts of your program. Fix it incrementally not all at once. Try using an interface contract when you make objects; that way, your new functions can call new methods and the old code can depend on old methods to be there. Deprecate the old methods when there is no code that depends on it. Don't forget to comment - comment the code then come back the next day and read your own comments. Make changes to the comments so they make sense today.

Blah... blah... blah... Etc... etc... etc...

None of us know your situation

[LoveMe2Times]

There's a few things that I'll say. One, not a single person posting here really knows what your situation is (well, unless your coworkers read /. and are posting advice for you...). I don't want to be mean, but most of what's been posted here is likely not applicable to your situation. For starters, ignore all the threads arguing over languages: they'll never agree, and in the meantime you have work to get done. Next, I would suggest ignore all of the posts from people who aren't talking about web sites. There's a lot of methodology recommendations there, and it takes a long time to really understand *any* methodology. So if you don't already know it, it won't help for *this* project. If you're interested in that kind of stuff, read up on it; maybe it'll be useful on a future project.
Next, no matter what you decide, from your current position, there's substantial risk involved. If you don't have a good way of estimating the costs and benefits of your alternatives, then you will wind up shooting in the dark no matter what you choose. This isn't really unusual, but it can be stressful! Somebody in management has to decide if they want to play high stakes poker with this project. This will establish how many nice risk mitigating activities you can budget (like unit tests, code reviews, documentation, etc). One thing a lot of technology people have difficulty grasping (this isn't directed at anybody in particular), is that if your management decides on a high risk course of action, it's not WRONG because it's technically inferior. Now, if management doesn't understand that it's a high risk proposition, then there's a communication failure somewhere that will screw you no matter what gets chosen.
Ok, now that you've decided where you're going, I've got 3 pieces of useless advice (all advice is useless advice, btw...)

1) Know your priorities. This is the most important thing to getting ANY solution.
2) Understand your requirements(I don't care how you do this). This is the most important thing to getting (close to) a CORRECT solution.
3) Remember it's only a job. This can be very important for retaining sanity :)

Re:Perl vs. PHP ?

[tmoertel]

Could I possibly ask for a fairly realistic little example of something that Perl does well that PHP can't?

Middleware.

For example, most enterprise applications do something important -- trade shares, manage accounts, track patient records, etc. How it's done is governed by business logic -- the company rules, policies, regulations, procedures, and so on. Now, you can spread this logic across your web site (as one might do using PHP, which is tied to the web site). Or you can bundle it up into an independent application, keeping all of the business logic in sensible, cohesive compartments that run on an application server (e.g., using one of the existing Perl app servers or one you've rolled yourself via, say, POE). This not only makes the business logic easier to understand and manage but also makes the logic independent of and accesssible from any number of front ends that you might need. Simultaneous web, client-server, and even command-line interfaces become possible, and for enterprise projects, multiple simultaneous interfaces is often a requirement for backward compatibility with older interfaces.

In summary, for shallow web-only stuff, PHP is a reasonable tool. For stuff beyond web work, PHP is out of its design envelope. However, Perl works here just as well as it does for web work.

my suggestions

[medcalf]

1. review all of your existing code and list the things that each script does

2. decide how to consolidate and divide out these tasks, and make sure that any functions or data that needs to be shared, or may in the future need to be shared, is separated out into modules

3. document all of the programs to be written, their interfaces and their effects

4. then, and only then, begin coding

-jeff

comment unneeded?

[volpe]

...and name the function so that the comment is unneeded


The only function whose name makes the comment unneeded is "exit()".

Re:comment unneeded?

[William Tanksley]

The only function whose name makes the comment unneeded is "exit()".

That's a stupid universal; I've already shown one example, which therefore disproves it. (exit() sometimes still needs a clear name, anyhow -- you may need to make it clear why you're exiting.)

But is there a form of your statement which has value? I say NO. Comments are only needed when clear code is impossible. Clear code is ALWAYS better. Sometimes it's unattainable, in which case comments are the only thing that can be done.

-Billy

My analogous situation

[Chris Johnson]

I'm currently doing something similar- the program I maintain is Mastering Tools, which has long been an over-featured, densely packed program operating on a text input basis with a certain amount of visual feedback on the text input.

However, I've long known that there are two things my real users (ideal users? the serious mastering engineers) want: familiar interface and realtime processing. I can't deliver realtime processing without literally doing the whole thing over again in a language I don't know (it's done in REALbasic, which is little better than a scripting language for speed, though it's got really, really nice prototyping abilities and GUI support.

However, the time's come to completely overhaul the interface, partly because I have some ideas for mid/side processing and don't have any room in the current layout to fit them! The ideas have to do with rectifying the side channel and using it to either enhance or remove signals that aren't in both channels equally- where regular mono is a 'node' that totally eliminates out-of phase content, it's also possible to completely eliminate R and L-only content and keep only R+L and also the out-of-phase content.

That part's the easy part- it's simply signal processing (and will be duly released under the GPL as soon as it's done, as always). However, the interface is asking for a total, complete overhaul in several ways, and that's what's taking all my effort currently. Here's the situation and how I'm handling it...

Layering. It's no longer possible to fit the whole app in a 640x480 area, even with small print. There are several possible answers to this: one would be having separate windows. This would be more adaptable to larger screens, but it's untidy and there are issues with closing windows and still referring to controls they contain- so that's out. What's looking more reasonable is tabbed panels- RB implements a nice little drag and droppable tabbed panel control that appears quite easy (though I had some trouble attempting to do nested tabbed panels- after one experience of having all the nested panels (at identical coordinates) switch to the top panel of their parent panel, I quickly gave up on that concept. Instead I'm using more panel real estate and trying to divide the controls into logical categories. That is, of course, a real headache- doing interface properly is hard! (says 'interface is hard Barbie') It's only somewhat easier with the additional space. Complicating matters further, is the expectation of the intended audience here. It has to both be organized and look and feel like a mixing board, amplifier or rackmount box of some sort.

Solution: implementing controls like knobs and meters. It's actually quite fun to code a knob appearance out of graphics primitives- and surprisingly hard to get mouse gestures to work on the damn thing- using a two-arg arctangent routine in RB that I don't fully understand. I also have a single meter control already implemented for azimuth display- think that it might be best to tear that apart and re-implement it in a more general sense.

That's because the Knob class turned out to be the right thing to do- it's implemented almost totally separate from the main body of the program, as if it were a RB control or something. Knowing I was going to be using it in different sizes, possibly different colors etc, I wrote the knob code as completely scalable- from maybe 20 pixels to over 100. It reads its size from the control width and height, runs itself as far as handling mouse input and storing a control value, and does not inherit comparable interfaces to the controls it will be replacing- so when it's time to plug 'em in I can run the program and see what routines crash and burn (and need to be rewritten for the new control interface). I'm thinking meters need to also be handled in the same way- somehow- not clear on the form yet.

So: dunno what else to tell you, but it seems like the things I'm doing that are helpful are: compartmentalize new interfaces, make them adaptable and get them working independently of the existing code, while leaving the existing code in working condition. Then when the new stuff is brought online do it in such a way that you could do it one control at a time or in small batches.

Dunno if that's relevant to where you're coming from- but it's what I've found necessary when facing a major re-implementation.

My solid advice.

[mindstrm]

First, decide WHY you want to re-write all of it.
Second, document, in detail, what it is you want to accomplish (what should each script do, etc).
Thirdly, write up a project plan, IN DETAIL, so you have something to work from
Lastly, actually translate that into code. Do not start writing code until you have the modules fully documented.

Programmer buy-in

[arabbit]

There are lots of good reasons to refactor, but "consistency and reducing redundancy" doesn't seem very compelling. Anyway.

A lot of folks have talked about process and tools, and those are important and I assume you know what you're doing. The real issue is morale. Let's face it, at this point you're requirements are probably stable and your team is familiar with the system and everyone's probably been thinking about how they might do differently in the back of their head. And code is easy to write.

If you have people on your team see what they've done and know they can do better and want to give it a try, things will go smoothly. If you have people on your team who regard this as a needless rewite, on the other hand, morale will suffer and production will be surprisingly low.

If you have some of both, take the people who want to rewrite it and do it from scratch in another branch. If you don't have buy-in from all of the developers and try to fold changes into the original product and it doesn't go smoothly, company politics will turn against you and you'll probably be looking for a new job.

finding and keeping a competant staff

[KyleCordes]

[There's only so much a methodology can bring to the table -- after that it's about finding and keeping a competant staff who understand the value of what they're doing.]

You are so utterly right. The key idea is "people over process". I've rather have 5 great people with any old process, than 25 mediocre people with the perfect process.

[look at the console gaming industry for tips]

I have always been impressed with the gaming industry for this reason. Sometime people erroneously think of game programming as being somehow less "heavy duty" than business apps, etc. But games are usually under incredible schedule pressure (for vital marketing reasons) yet also need to ship very few bugs, because buyers are prone to return a game that doesn't work, not become a source of upgrade revenues.

Re:finding and keeping a competant staff

[Anonymous Brave Guy]

The key idea is "people over process". I've rather have 5 great people with any old process, than 25 mediocre people with the perfect process.

Absolutely. 5 great people will look at the process they've started with, and if it's ****, they'll throw it out, come up with a new one that works, and then after the five minutes that took, they'll get on with the job.

Re:Minimize Untested Documentation!

[wik]

If you are going to use descriptive names (which I think is a fine idea), particularly like the one you mentioned, make sure the steps of the algorithm are clearly denoted. When one programmer sees an algorithm (or a maintainer looks it up in a book), they may be looking at the same algorithm, broken into different steps. There is nothing more frustrating than seeing the right thing (if you look in book A), labeled in a completely different way (because you learned it from book B).

I think one of the biggest problems is that people believe that because they named something clearly, it must automatically be clear and logical to others. More generally than just naming steps of an algorithm, this is a problem with naming commands, functions, variables, etc. I think this paper makes a strong case http://citeseer.nj.nec.com/furnas87vocabulary.html for the naming problem (yes, this is even a problem with experts in a particular field). Names are great, they don't always stand on their own. I'd highly suggest that people read the short section on "armchair" naming. I haven't seen a programmer who wasn't tempted to use this at one point or another.

Re:some friends

[benedict]

When I said "serious perl heads", I didn't mean
"drunks I met on the street". I meant "people
involved with the perl design process".

Don't make the same mistake twice

[dgroskind]

The main reason for redevelopment is when the underlying data structures change and the legacy code won't work with the new data base product or schema.

Typical reasons for changing the data base are:

the platform is obsolete, e.g, the vendor abandoned the product or went out of business; it doesn't run on a new operating system.

some critical feature is only available on a new platform

the original data schema is inadquate or poorly designed

In general, if the underlying data design remains the same, then you don't need to do a massive redevelopment.

Complete redevelopment to make maintenance easier, that is, for consistency and reducing redundancy, is rarely cost-effective. It's better to begin adding new features in a more robust or easier-to-maintain platform and convert legacy systems as time permits according to some set of priorities.

If you do redevelop, make note of why and make sure you don't make the same mistake again. Often a poorly designed system is the result of some business requirement or limited resources. If those pressures remain, the new system will have the same problems as the old one.

K&R style

[Anonymous Brave Guy]

Also, if you are selecting the coding style style yourself, use K&R style. I don't care if you think it's not 100% logical -- it's readable, and every programmer worth his/her stuff is familiar with it.

K&R style was developed for presenting C code in a compact format for publication. It has popularity going for it, but there are better styles available for everyday code, some at least as common. K&R style becomes hard to read far more often in C++, where you have things like try-catch blocks around, in-line function definitions in class declarations, and so on. As always, of course, it's more important to choose a consistent style and stick to it anyway.

Re:Screw Comments DO DESIGN DOCS

[William Tanksley]

Design documentation is highly overrated (after all, it's a duplication of the code itself, which is the most detailed possible design document). Requirement documentation is essential. I suspect that this is what you're asking for, anyhow, since you're talking about showing it to the person who asked for it.

I think that requirement documentation is the only universal documentation requirement. Everything else can be ignored in some (special) situations.

-Billy

console games, PC games

[KyleCordes]

That's a very good point about console games.

I hadn't thought of that, I was thinking of games that run on a PC. PC games get the "best of both worlds", so to speak.

Fighting redundancy by repeating yourself?

[Zero__Kelvin]

"My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code) in the interest of consistency and reducing redundancy."

Do you mean that you have consistently repeated the act of re-writing your code in an effort to reduce redundancy? Or perhaps you are seeking to reduce redundancy by consistently repeating yourself? 8^}

Documentation not comments.

[MikeFM]

I think an important thing to clarify is that documentation is good and comments usually mean your code is bad. Comments are not documentation.

I attempt to keep a documentation/history somewhere for each function, class, or any other major component. In this documentation I typically explain what the code is for, it's inputs and outputs, and a history of why certain things were done in the way they were. This is often just one or two lines but despite good naming practices and good coding there are times where it just isn't obvious that on platform Foo you have to do it this way or you see Bar bug. I don't however go through code line by line commenting what it's doing and why. That should be perfectly obvious if your code is any good.

Re:Screw Comments DO DESIGN DOCS

[Anonymous Brave Guy]

Design documentation is highly overrated (after all, it's a duplication of the code itself, which is the most detailed possible design document).

Not if it's done properly, it's not. Attempting to duplicate your code is actually damaging, because your docs will become out of date and misleading. However, maintaining a 10 page summary of the basic foundations of your system is far more helpful when bringing a new team member up to speed than sending them away to look at your "most detailed possible" 500,000 lines of existing code.

Hardware is cheap?

[Anonymous Brave Guy]

The "hardware is cheap" mantra is exactly why we have crapy code and bloatware. "I dont have to optimize, hardware is cheap." "Why make it efficient? hardware's cheap!" "I dont care that it is slow, hardware is so cheap these days"

I agree wholeheartedly. I think most of the problem is McManagers failing to understand what "hardware is cheap" means.

Hardware is cheap, but this is a relative statement. Shipping your kit on a platform with a 1.5GHz processor instead of 1GHz is often a better choice than having your whole development team spend six weeks profiling and optimising the code to get a 50% speed improvement. This is very expensive in terms of development time (= time to market hit and extra wages). Worse, it tends to lead to obscure optimisations in otherwise clean code that rarely get documented properly and thus hinder maintainability. So, in this respect, hardware is cheap.

However, this is not an excuse not to write good code in the first place. Don't prematurely optimize, sure, but for goodness' sake get the basics right.

• Use suitably efficient data structures and algorithms.
• Design your code in a modular way, so that if you really do need to apply optimizations, you can do so in an isolated, controlled and easily testable way.

The cheapness of hardware does not justify getting sloppy and ignoring the above. These are the major causes of bloatware and the reason why I could start typing faster with Word 5 for DOS on an 8086 more than a decade ago than I can with Word 2002 on a 2GHz Athlon running WinXP.

Re:Complete rewrite necessary or a waste of resour

[Martin S.]

>Not too long ago a link was posted to an interview with Joel Spolsky who used to work at Microsoft.

WOW, you are easily impressed. Just because somebody used to work for Microsoft does not make them an expert software engineer. In fact judging from poor quality of most Microsoft software we can safely assume the absolute opposite.

Re:Inner loop?

[William Tanksley]

When you scream "must," you seem to claim that "there are no exceptions." What if this new function needs to use a lot of a function's local variables without passing them as parameters (expensive for stack memory bandwidth, especially in the inner loop)?

Optimization /is/ a special case, I admit. For optimised code, expect readability to go out the window, in general. You have to give up something. Thus, you use comments rather than proper coding. However, keep in mind that micro-optimizations like that are dwarfed by better algorithms -- and sometimes (rarely) they're even trumped by a decent compiler. So keep those comments ready, but TRY not to use them.

However, in the general case, there's an alternative you aren't considering. Rather than fiddling with the stack or with messy hand-optimization, put the variables and the two methods into a class of their own. It's a standard, documented refactoring.

There is some info that can't be conveyed by code; it's called "requirements". That info can't be conveyed by comments, though, either.

However, you can refer to and briefly quote the use cases in the comments, to give future maintainers a bit of context.

Most requirements can't be expressed that way; any that can be, should be expressed as function names, as with all the examples which have been given.

-Billy

Re:Screw Comments DO DESIGN DOCS

[William Tanksley]

And having him look at the requirements, and then work on the unit tests, and finally pair with a more experienced programmer, is better still -- and this way there's no extra paperwork to maintain.

-Billy

Re:Screw Comments DO DESIGN DOCS

[Anonymous Brave Guy]

Do you have any evidence to support that, or is it just a little gratuitous XP evangelism? ;-)

But seriously, on a large project, you can't always afford a sufficiently experienced programmer to work as a pair with all the newbies. For example, on the project I'm working on at the moment, there are only two people with anything like complete knowledge of the project. This is a serious liability, and we're trying to spread the knowledge out to the rest of the team. However, there are several more than two junior guys. It's far more efficient for those who know to document the basic knowledge as a starting point, than it is to try to convey it to each junior team member directly during day-to-day development.

Re:Screw Comments DO DESIGN DOCS

[William Tanksley]

Do you have any evidence to support that, or is it just a little gratuitous XP evangelism? ;-)

The second. ;-)

I seriously do recognise some need for "get up to speed" documentation; it's just that it's almost impossible to write enough of it, and not write far too much. Of course, if you're doing pair programming regularly you won't need any (and you will have ALL team members knowledgable about the code in general), but if you can't do that (and politically it's almost impossible) you'll HAVE to write that getting-started documentation.

A pity that almost nobody will read it; the only exception will be the people so concientious that they hardly need it. Of course, you could be the exceptional project in which the people who know the code are also talented writers. But I doubt it.

-Billy

Re:Screw Comments DO DESIGN DOCS

[Anonymous Brave Guy]

I seriously do recognise some need for "get up to speed" documentation; it's just that it's almost impossible to write enough of it, and not write far too much.

Sad, but true. :-)

A pity that almost nobody will read it; the only exception will be the people so concientious that they hardly need it. Of course, you could be the exceptional project in which the people who know the code are also talented writers. But I doubt it.

Probably not exceptional at all, actually. I'm just noticing that around 18 months ago, as I came onto this project, I started a steep uphill learning curve to get up to speed. So far, about four people have made it to anywhere near the top in seven years of the project's history, not counting the guys who designed it all in the first place. These are the only people who can do serious maintenance or development on a project critical to the company. Everyone else on the project does their bit, but there is a certain point when they can't do any more to help.

Now, a year and a half later, I'm responsible for training up the next guy to join the team. He's enthusiastic, and has a great attitude, and you can work with someone like that. But he has the same problem I did back then, and I have the same one my own team leader had back then: there's no basic docs to teach him the ropes, so I have to spend large amounts of my valuable time -- like my team leader before me -- training him up instead. He may or may not ever grok the really key stuff at the heart of the system, but I hope that he will. However, if someone would authorise the time for just one person who knew the stuff to document the basics, it would literally save weeks of effort in future on the part of senior team members, particularly in training up those who are only ever going to use the basics.

And then I woke up. :-)