When Making a Comprehensive Retrofit of your Code...

chizor asks: "My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code) in the interest of consistency and reducing redundancy. We're going to have to make some hard decisions about code style. What suggestions might readers have about tackling a large-scale retrofit?" Once the decision has been made for a sweeping rewrite of a project, what can you do to make sure things go smoothly and you don't run into any development snags...especially as things progress in the development cycle?

Comments

Re:fp

[Anonymous Coward]

but oh what a contribution it is!

object orientation

[apsmith]

we're doing something similar - and switching to java (JSP's + Tomcat, struts) to replace a lot of old perl cgi's. The java code is much, much cleaner. But object-oriented perl code can help if you don't want to take the plunge too far into a new language. And at least find a way to go mod_perl rather than CGI, for the things where performance matters at all.

Re:object orientation

[TheRain]

why is this flamebait? all this person is saying is that object orientation could make the code easier to manage and, thus, help to reduce redundancy. it's a good statement, modularize your code and it becomes more reusable... and therefore less redundant. who cares if he likes java over perl and CGI.

Re:object orientation

[Plisken]

How the hell is this flamebait. Moderators, layoff the christmas crack.

Re:object orientation

[apsmith]

Alright, maybe I posted a little too soon, but shouldn't "flamebait" be attracting flaming responses? I don't see any...

Anyway, if I'd spent a little more time thinking about the advice side of it, taking a look at appropriate programming methodologies (like Extreme Programming advocated in another thread here) would be one piece I'd advocate. Given the size of the code (1 MB = about 20-30,000 lines?) there's no need for major heavy-weight processes here. More important I'd say is sitting down and figuring out in the appropriate level of detail what exactly your system is doing right now - you can do this using UML diagrams which seems to be becoming a standard, though the main use we've found is to try to get an overall view of things which we then throw out when we get into the details again.

The other thing to do along these lines is look for your use of standard patterns within your code - the Design Patterns book is extremely helpful if you're moving to an object-oriented framework at all; following well-known patterns and indicating clearly what you are doing can make your code much easier for others to follow.

Re:object orientation

[Anonymous Coward]

Do have an architecture to your code? Look at how to refactor/rearchitect. If you can move to an OO language great. Move away from Perl, even better.

Don't use Extreme Programming. You have a big job ahead. Extreme Programming is probably how you got that mish mash of poorly knit together code in the first place. I've seen four companies either fail or get to the brink of failure trying to use Extreme Programming to build large, high-throughput systems.

Re:object orientation

[ergo98]

...refactor...

Just as a point of reference: I've noticed that the term "refactor" has gained popularity with the whole XP push. Of course "refactor" is just a cooler way of saying "rewrite". i.e. It's not "We should rewrite that module", but rather it's now cooler to say "We should refactor that module". Blah.

Re:object orientation

[anacron]

Refactoring is not re-writing. Refactoring is a methodical approach to moving, redesigning, and making the code cleaner. It's not throwing away and starting from scratch. The two words are wholly semantically different.

In his book, Refactoring, Martin Fowler talks about how code "smells". He identifies a whole slew of reasons why code can "smell" bad, such as having a method that's too long, having a method on the wrong object, incorrect use of polymorphism, etc. He then outlines "Refactoring Patterns" which can be applied to certain "smells" to make the code more managable.

The last project I managed was a 3 month re-write of over 75,000 lines of code for an auction component that had gotten way out of control. The original component integrated with a very very poorly written (yet still very expensive) auction engine, and our task was to take it out.

Because there was no time allotted to proper requirements gathering, the powers that be decided that we could use the existing system as a requirements base, and just make the new system "act" like the old one.

So I was thrown two things: A staff that was inexperienced in design, and a lack of requirements. We were re-writing from scratch -- throwing all of the code away and starting again. I'm one of the only ones on my team that actually knows what refactoring means, let alone how to apply it.

So we built the new system based on a solid design. We even did the design in UML. But that only goes so far. Given a properly designed system, there is still another lower step -- how to actually implement the code. Invariably, the design doesn't take into account helper methods that are necessary, other objects, etc. Because we were on a tight schedule, I left it up to each developer to design these new tidbits.

If they had known what they were doing, everything would have been fine, but there are several places where the code is absolutely abysmal. It's not that it will perform poorly or is totally unmanagable, but in some places it's close.

Granted, the business rules for the system are some of the most complex I've ever seen, but if the developers had known more about refactoring, they might have made better choices.

There's a whole notion of Quality here that should be discussed as well. Most good has a good enough quality, and though it looks ugly, I'd throw it away. There's an interesting interview by Joel (a former microsoft programmer) which tackles these problems of Quality head on. I also reccommend reading Zen and the Art of Motorcycle Maintenence: An Inqiry into Values for a pretty good discussion of the whole notion of quality.

I'd say just keep the codebase unless you are totally switching languages (or in our case taking out a 3rd party integration). Otherwise if you re-write you'll wish you hadn't. The new codebase will end up being in the same state as the old codebase, it'll just take a little bit more time to get there.

I'd also reccommend just doing a performance test of the site, and find out which modules really need to be re-written. The 80/20 rule will invariably apply -- 80% of the execution time will be spent in 20% of the code -- so at a minimum just re-write the 20% of the code that's executed on the most frequent paths. Leave the rest alone. Re-writing it will just be a waste of time, and you have to fix all the old bugs that were re-introduced during the re-write.

Re:object orientation

[Anonymous Coward]

Good point about object orientation.
It's often a good thing to take a couragous step like this and rewrite & refactor code (unfortuneatly management tends to see it as unproductive time, but it can pay off in the long run)

Now my main point: I can't imagine having to deal with megabytes of Perl code. It's hard enough to figure out the intent of a simple perl script I wrote a few months ago, I can't imagine having to slog through MEGABYTEs of the stuff. Perhaps while your at it you should take another couragous step and rewrite in another language.
May I suggest something like Ruby: it feels pretty familiar to Perl folks and its quick to learn. Ruby's object oriented-ness certainly would beat trying to create an OO system in Perl (it makes my head hurt just to think of OO perl!)
Check out: http://www.ruby-lang.org
or http://www.rubycentral.com
for more info!

Re:object orientation

[Anonymous Coward]

anacron, interesting experience and solid advice. I didn't see Extreme Programming mentioned. Did you use it at all?

Re:object orientation

[anacron]

anacron, interesting experience and solid advice. I didn't see Extreme Programming mentioned. Did you use it at all?

Actually we didn't use it formally. But we used some of the best ideas. Iterative development, frequent code reviews, programming in pairs etc. We had a pretty small team so XP might have helped a little bit more, but it's a fundamental paradigm shift that most people I work with aren't comfortable making. My team would have been less effective had I said, "You *must* use XP."

I have been on projects in the past that have embraced more of the XP model, but something like Rational's Unified Process (RUP) kind of fits in more with what we do.

For traditional software shops (e.g. those that make products) a methodology can be set in place by the management, because the company is in essense it's own client. For other types of software shops -- such as consulting firms, or web design shops -- where one-off software is commonplace, the type of methodology that's used is usually determined by the client.

For example, if I'm working with a client that isn't very savvy, I probably don't want to do all of my requirements in formal use-cases, followed up by UML design documents. They won't add any value to the client at all, yet we still have to make sure we understand what it is we're building.

It's an interesting challenge and it's usually best overcome by having solid people on the team that can speak to both sides. A solid requirements team which can talk to the client (be it internal or external) and truly understand the business problems, all the whole clearly articulating those to the technical team. Sometimees double documentation is necessary -- one set for the client and one set for the developers or development lead.

There is just no easy way to do software development correctly. It's amazing that in nearly 60 years, no one has come up with a "right" way. There's only so much a methodology can bring to the table -- after that it's about finding and keeping a competant staff who understand the value of what they're doing.

The methodology is just a roadmap. How the car is driven is completely up to the development team. Even a good technical leader can become a backseat driver at times, as there is just not enough hours in the day to provide oversight to every single member of a team.

I think the more traditional software shops should look at the console gaming industry for tips. They have a difficult challenge -- it's not possible for them to release a fix or update for their game once it's completed. So how do companies like Activision, Square, Nintendo and Sega do their software development, and is there anything that the development teams at large can learn? Rigerous testing is one of the most obvious things that comes to mind for them, but how do they know what it is they're testing when a game's vision usually only exists in the mind of the creator?

.anacron

Re:object orientation

[Error27]

>>Alright, maybe I posted a little too soon, but shouldn't "flamebait" be attracting flaming responses?

It's because the good people of slashdot recognized your feeble attempt at attracting flames and did not deign to give you the pleasure.

Slashdotters are known world wide as the most discerning and reasonable of all people. Please don't consider a personal afront that you were not flamed. Lesser mortals most certainly would have.

Re:object orientation

[ergo98]

Refactoring is not re-writing. Refactoring is a methodical approach to moving, redesigning, and making the code cleaner. It's not throwing away and starting from scratch. The two words are wholly semantically different.

"Refactoring" is a trendy term (which alone makes it suspect) that came into mainstream acceptance very recently, but that most certainly doesn't mean that those in the software industry having been doing EXACTLY that for many years since? Why is it that once someone gives a name to something that people have done for years they think they've invented it?

I have never known a software project where the participants didn't actively look through, clean up, and where necessary rewrite old code, but now this is "refactoring" and it's a whole new paradigm in software engineering. Blah I say. I hereby name designing quality code and implementing it in a platform independent fashion "requantization", and my new paradigm is requantization development.

Re:object orientation

[anacron]

"Refactoring" is a trendy term (which alone makes it suspect) that came into mainstream acceptance very recently, but that most certainly doesn't mean that those in the software industry having been doing EXACTLY that for many years since?

I agree -- the term 'refactoring' is pretty trendy right now. But for the first time, there's a methodology behind it. Your argument is like saying, "Medicine has been around for 2000 years. All the advances that have been are just trendy and don't really benefit us."

Just because something is done, doesn't mean that it's done correctly, to plan, and for a good reason. Refactoring formalizes the best of what most people have already been doing, puts rules behind it, and quantifies exactly the reasons for doing it.

Re:object orientation

[ergo98]

Your argument is like saying, "Medicine has been around for 2000 years. All the advances that have been are just trendy and don't really benefit us."

I would say that a more apt analogy would be saying "For 2000 years doctors have checked your temperature and done an initial physical evaluation, but now we've renamed it `physioevaluation" and it's a whole new paradigm in health care.

Use a pretty printer

[renehollan]

for consistent style...;

refactor as a result of learning from your mistakes and redundancies;

and try to minimize the busy parts (where all developers have a hand) when things change (like lists of unique symbols, numbers, etc.)

Re:Use a pretty printer

[Anonymous Coward]

retrofit = "to change from slow perl code to faster C/C++ code" (if you actually had a team of people to work on such a huge project)

Sleeping dogs

[Chairboy]

I'm sure a bunch of code nazis will disagree with me (please note the clever way I attempt to pre-emptively undermine their arguments by labeling them as 'nazis') but sometimes massive engineering re-writes are not necessary.

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

Most companies that decide to massively re-engineer their code (do a big rewrite) usually end up regretting it because it forces them to re-fix the problems that caused the original strange looking code in the first place.

Does your CGI nest work? If so, maybe you should leave it alone. If you are fixing specific problems, then go ahead, but if this is a generalized attempt to fix the 'not invented here' syndrome that plagues engineers (who will almost universally agree that it is easier to write code then it is to read it), perhaps you should reconsider.

Re:Sleeping dogs

[kz45]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

This is a lesson to be learned. Engineer your code from the beginning. Use easy to understand commenting, and strucutured code. Although it takes some discipline, you will almost never have to reconsider "re-writing from scratch".

Re:Sleeping dogs

[smileyy]

++ to that comment.

A general refactoring, without the intent of doing it to add new functionality / fix bugs, isn't worth it. What business value do you gain from it?

However, there are times that it is worth undertaking substantial refactoring to add what may seem like relatively small pieces of functionality. I think it was in his _Refactoring_ book that Martin Fowler likened this to reaching a local peak in a mountain range, where you have to go down into a valley before you can climb the next peak.

ObOfftopic: This is my first post in a while that hasn't been a troll or flamebait. It just isn't as fun being informative.

Re:Sleeping dogs

[Plisken]

What we plan on doing is rewriting a module at a time and then doing extensive testing on that module. I know that sounds obvious, but alot of outfits will _figure_ out some grandiose design and then parse out the parts to different developers, working simultaneously. Many times you end with something just as bad because either you didn't learn from your mistakes or that the design was flawed from the beginning.

Then again, if your code is one _huge_ pile of spaghetti, you might have no choice except to rewrite all of it at the same time.

My 2 cents

Re:Sleeping dogs

[pyrrho]

But the design is probably flawed at the modular level as well... you will find making cleaner versions of modules in place is like making a fossil... in place replacement can turn flesh to stone, but it will keep the overall shape unchanged. In the case of hard to maintain or troublesome code, this is probably not what you want. If it is... you could just debug the modules and not rewrite them.

Re:Sleeping dogs

[Nyarly]

Certainly, but if you find yourself trying to make a change to existing code, it will probably not be the last change you need to make. At least refactoring anything your change touches will make future changes easier to make.

Related note: the original poster doesn't say "refactoring" and he does say "Perl." Informative statements relating to these two facts:

• Buy and read Martin Fowler's Refactoring. The examples are mostly Java, and he references the Group of Four's Design Patterns. "Retro-fitting," in which you probably plan to rewrite portions of code from scratch will break your app, your mind, and your budget. Learn what refactoring implies and entails.
• Learn about design patterns in general, and consider how they might apply to your code. One description of a design pattern is "a target for refactoring."
• (Donning asbestos) You might want to reconsider perl as the language of choice for a large scale application. I realize I'm posting this comment to a Perl system, but Perl hangs together like an immense kludge of a language. That said, you're probably stuck with it, and AFAIK, you may be forging new paths in programming for reusability by applying the above concepts to Perl. Good luck, and be sure you can trust your machete.

Re:Sleeping dogs

[bbqdeath]

I second the concern about PERL. And I offer my advice as someone who has virtually no qualifications to talk about large systems of code. I just like Python better than PERL because it doesn't hurt my eyeballs like PERL.

Yeah, I guess this is a troll. But it's honest. I use Python like most people use toilet paper: several times a day, and for more things than it was originally intended.

Re:Sleeping dogs

[curunir]

The rule that I've learned is, If there is something wrong with the way it is architected,
then go ahead and re-write it if you have the time. If you aren't going to change the design, then there is no reason to throw away the resource you have.

Proper program design (modularity, well defined interfaces, bla bla bla...you've heard it all before) makes code much easier to manage and improve. If you don't have it, then re-writing might be worth it.

However, far too often, people feel the need to re-write code because it feels messy and it is hard to maintain. The fact is that it is easier to write code than to read it and an engineer will almost always find more enjoyment in writing something from scratch than in fixing/upgrading someone else's code. Be careful listening to engineers bitch about messy code. It usually isn't half as bad as they say it is.

However, in this case, it sounds like they do need a serious design revision. I'm not sure I can think of any justification for using Perl CGI for an app that size (mod_perl, java, c++ and even asp would be better).

Re:Sleeping dogs

[Anonymous Coward]

Since there was a /. article on this very subject recently, it is no surprise that there are now numerous posts parroting the "no rewrites" position.

When badly kludged code designed for ancient programming paradigms or with tools better suited to smaller jobs becomes untenably brittle, it is time for a rewrite. If you can't fix the most minor of bugs without having huge ramifications extending out in to mainline code paths, you have a real problem that begs for a fresh start. Call it a chaotic butterfly effect for software. :)

For a rewrite not to be a disaster, however, you need competent, well-documented design from the start. If your staff isn't capable of that, or if your development cycle won't allow it, you may as well stick with what you've got, because either way you will continue to be condemned to debugging hell.

Re:Sleeping dogs

[inbox]

Perhaps the reason for the code-rewrite is because everyone is being fired.

There's your business value. The current group is not cutting it, but their code is crap. They need to go, but the code is useless without them. An idea: get them to clean it up and then fire them and bring in a new team that now has nice, new, clean code.

Conspiracy theory, perhaps. Possibility, definitely.

Re:Sleeping dogs

[Anonymous Coward]

possibly but consider this : how long do you want to stay on one project ? i typically get tired of a codebase i wrote after one year. if the company doesnt want to fire me i usually quit anyway. if the company shifts me to a different project i say great and get on with it.
dont stick to a job just cause you think the economy is down or whatever. its not worth the effort.

Re:Sleeping dogs

[thogard]

You can't engineer your code from the beginning in the part of the real world where I work. I've got a huge mess of perl that do reports and about one out of ten clients wants something different. These request could have have been predicted since many of them have no business reasons and have no function other that look more like their older system which just happed to have done it that way.

Re:Sleeping dogs

[thogard]

If perl hurts your eyeballs, you don't understand it.

At work we have one system that was built with python and its just not as stable as everything we have written in perl. At this stage there will be no more large project written in python but I'll be watching how the small project go.

Re:Sleeping dogs

[Jeremi]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

... which is why whenever you add one of these "little design decisions" whose purpose isn't blatantly obvious, it's important to put in a comment saying what it does and why it is there. Otherwise someone might come through later on, think it's an error, and remove it.

Re:Sleeping dogs

[ZeroConcept]

Another strategy is to have a group of persons updating/fixing the current code while another group writes a framework and migrate the pages one by one to the new codebase.

Try to make the iterations as small as possible, that way if you missed something in your design you can fix your design earlier.

I've seen projects where ppl create a HUGE design, implement it and at the end, requirements changed/were missed and the whole thing needs to be hacked to make it work...test your design the earliest possible by migrating pages early.

Re:Sleeping dogs

[sunking]

So you're saying your problem is too hard to be solved with a clean design? Good software design is all about preparing for the unexpected! In fact, if you ever had a perfect understanding of the problem then it wouldn't be worth designing a solution carefully - you wouldn't need the flexibility!

Thinking that writing one-off code is giving you flexibility is grave mistake.

-sam

Re:Sleeping dogs

[Trepalium]

If perl hurts your eyeballs, you don't understand it.

Perl makes it easy to write a quick hack that looks ugly but fixes a real problem. The hurting eyeballs factor only comes in when that quick hack gets progressively promoted to a full mission-critical program with little or no planning. There are well written, structured perl apps, and there's the unstructured ugly ones. Most people have seen more of the ugly ones.

Re:Sleeping dogs

[PD]

Sometimes, but not always, if your code is a mess, then that means that your business is a mess too. If your programmers have a hard time understanding the business, then your customers will too.

When you investigate those twistly little lines of code, see if a business rule can be simplified. Try really hard. If it can't be, then put it into code. Don't make the opposite mistake of making the rules TOO simple.

Re:Sleeping dogs

[foobar104]

You might want to reconsider perl as the language of choice for a large scale application.

I agree 100%. My company started to bring a commercial application to market a little less than a year ago. I prototyped the code in Perl, and the prototype was sufficiently okay that the decision was made to evolve the prototype into the release code.

This was A Mistake. It was A Dumb Idea. It was also My Decision. I have taken Much Shit for this from my coworkers. But you live and you learn.

I have since (over the past four months or so) rewritten the entire application-- every line, every file-- in C++. The source tree is 3.8 MB, and it compiles to about 100 MB of object code. (The actual executables, of course, are much smaller than that.) It was a pretty big job.

Not only is my code tighter and cleaner than the original Perl stuff (which was actually pretty okay code) but it's between 2 and 10 times faster.

I love Perl, absolutely adore programming in it, but there are some things that are easier to do with C, or C++, or (presumably) Java. When you split a project up among a number of people, for example, using the Bridge design pattern and distributing read-only interface header files makes modular integration so very much easier. That's just one example.

We would not have been able to get our app to market without the Perl prototype. And I don't think it would have been worth a damn if we hadn't rewritten it in C++.

Re:Sleeping dogs

[JamesOfTheDesert]

... because it doesn't hurt my eyeballs like PERL.

Do you write your PERL CODE IN ALL UPPERCASE? That *would* hurt your eyes.

Please; it's Perl, or perl, but not PERL.

Re:Sleeping dogs

[V.P.]

What we plan on doing is rewriting a module at a time and then doing extensive testing on that module.

An even better way to go (a.k.a. extreme programming) would be to write the "extensive tests" first. This way you can uncover problems in your design before you start messing with the original code. Besides, having the tests available from the beginning makes it much easier to ensure that your changes don't mess up the intended functionality.

Write tests first!

Re:Sleeping dogs

[Anonymous Coward]

An even better way to go (a.k.a. extreme programming) would be to write the "extensive tests" first. This way you can uncover problems in your design before you start messing with the original code. Besides, having the tests available from the beginning makes it much easier to ensure that your changes don't mess up the intended functionality.

I agree completely, but unfortunately our test methods have always kindof been ad hoc(read the developer does the testing), and we don't have a decent test suite.

Another unfortunate thing is writing a test suite is a major project and we just don't have the manpower to devote to it. We tell management, but they never listen.

A third problem we have is that we just don't have the mainframes on-site to do testing with it, so it ends up asking one of our clients to run a couple tests for us. This is probably the biggest problem. Much of the time we just can never be sure about the behavior of code unless we have clients test it, or write _extensive_ simulations, which once again we just don't have the manpower for.

Re:Sleeping dogs

[Dwonis]

Exactly. I think that if it's mission-critical, it should probably be written in C. If you aren't confident that you can write mission-critical code in C, then you shouldn't be writing mission-critical code at all (even in Perl).

Re:Sleeping dogs

[bay43270]

This is a lesson to be learned. Engineer your code from the beginning. Use easy to understand commenting, and strucutured code. Although it takes some discipline, you will almost never have to reconsider "re-writing from scratch". Even if the people maintaining the code had anything to do with it's original design (doubtful), they can't do anything about it now. A more practical solution would be to create tests, and refactor the code piece by piece. I am only beginning to use the ideas presented in Martin Fowler's "Refactoring", but they are really helping me clean up my code. More importantly, the changes are safe. Fowler's book warns me of problems I might run into, and gives hints as to what I should look for. Although I'm using Java and am lucky enough to be using IntelliJ's IDEA (which is an IDE with built in refactoring capabilities), I'm sure there is a solution for Perl (even if it is more manual).

Re:Sleeping dogs

[bmajik]

"Mission Critical" does not imply C. Infact, it points away from C.

Unless you're writing an operating system, why use C ? What does it buy you ?

IIRC, the Avionics system of the F-15 is done in Ada, because its "Safer" than C. I'd say thats "Mission Critical".

I too am weary of perl, but like everything else, its the right tool for _some_ jobs, arguably, some pretty important jobs.

Re:Sleeping dogs

[Anonymous Coward]

This is a very common mistake that I've been seeing lately.

Frameworks can be used to great effect where each client wants different things out of the same module. Understand the problem at the highest level and create protocols for solutions for the class of problems. Implement common aspects of all solutions as part of the framework. Then each specific solution for a customer can be implemented as a very small add-on module.

Check out the ideas of framework in OO.

I've used this pattern successfully in several large-scale systems over the last 8 years. It works whereever there are uncertainties about customer requirements. And, especially, where it is certain different customers will have different requirements.

Re:Sleeping dogs

[consumer]

Isn't it possible that your rewrite was better just because it was a 2nd draft? There's really no reason you can't define interfaces for Perl classes and share them among a group of developers, like you did with your C++ header files.

Re:Sleeping dogs

[Anonymous Coward]

So you're saying your problem is too hard to be solved with a clean design? Good software design is all about preparing for the unexpected!

You simply can't prepare for all the "unexpected", because different feature requirement changes can occur, which involving conflicting changes in the design. You might wish complete input rollback, distributed systems, os or platform independance, integration with some given CGI scripts OR with some other plateform, optimization for heavy load and speed, optimization code safety (only write slow checked algorithms), addition of scripting, control via HTTP interface, via SNMP, etc... not to mention usual specification changes.

If you design for all these modification possibilities, probably you'll end up with bloated design, and your changes will fit only, say, 3/4 or 9/10 of our design, demanding 1/4 or 1/10 of hacks and kludges. That's why Martin Fowler in "Refactoring" advocates to remove all unused code (and all the tests) : of course you start with good overall design structure, but the idea is when you need to change, you *will* have to refactor, to get a proper design with no hack at each change iteration.

Re:Sleeping dogs

[ameoba]

One other reason I see to move away from Perl as the language for the project is the complete rewrite (IE fundamental changes to syntax, deprecating all existing code) of the language that's currently going on. Perl5 is going the way of COBOL; it will become increasingly difficult to find coders both competant and up to date in their skills.

I'm not really qualified to make any recomendations about which language you should use, but if you're doing a rewrite now, when Perl5 coders are still plentiful, choose a language that you can be reasonably sure will be used 3yr from now. The whole point of the project is to ensure future maintainability for the software..

Re:Sleeping dogs

[dgroskind]

The hurting eyeballs factor only comes in when that quick hack gets progressively promoted to a full mission-critical program with little or no planning.

The trick then is to pick a platform that is suitable for quick hack but can be promoted to mission critical with little of no planning.

Perl is a suitable candidate because

it's reasonably fast in execution

there are a reasonable number of Perl programmers around for future maintenance

it has some cross-platform capablility and scalability

Most people have seen more of the ugly ones.

There is a trade-off between speed of development and maintainability. In the current business environment, speed of development is ascendant. The people who say you should've planned better and used Java or C++ usually come along later and try to score some political points. Where were they when there was work to be done?

Re:Sleeping dogs

[m_ilya]

choose a language that you can be reasonably sure will be used 3yr from now.

Perl6?

Re:Sleeping dogs

[Anonymous Coward]

One other reason I see to move away from Perl as the language for the project is the complete rewrite (IE fundamental changes to syntax, deprecating all existing code) of the language that's currently going on.

Total bullshit. Perl 6 is being designed to run around 90% of Perl 5 code unchanged, and with a Perl5->Perl6 conversion module to automatically "port" the other 10% in memory before running it. You'll also be able to run the conversion manually, using it to do the "heavy lifting" of updating your code.

Re:Sleeping dogs

[foobar104]

Isn't it possible that your rewrite was better just because it was a 2nd draft? There's really no reason you can't define interfaces for Perl classes and share them among a group of developers, like you did with your C++ header files.

Yes, TMTOWTDI.

But I still think that C++ is my language of choice for big projects because it helps do some of the job of collaboration for me. If you, as a team member, don't implement the interface completely, your module won't compile. And if you change the interface, nobody else's modules will compile. So less time is spent doing module-level integration.

And C++ has a lot going for it in the preemptive bug-squashing arena, too, with strong typing at compile time and such.

But I didn't intend for this to be a "C++ is better than Perl" comment; as I said before, I love Perl. I just think C++ is more appropriate for some things than Perl is, and medium-and-larger programming is one of them.

Re:Sleeping dogs

[tradervik]

Does your CGI nest work? If so, maybe you should leave it alone.

Hmm, sounds like COBOL.

Re:Sleeping dogs

[Ian Bicking]

Good software design is all about preparing for the unexpected!

I couldn't disagree with you more. Putting generality into the code that you don't forsee a need for is very bad design, IMHO. Generality is messy, and generality in one piece often makes it difficult to make another part general.

For the most part, your code should be designed to do what it is supposed to do, and no more. Of course, you will have an idea of what it might be needed in the future -- and it will sometimes be a good idea to design the code with that in mind. But usually it is not, because you will often be wrong. And if you don't even have an idea of what need your code will fulfill -- i.e., you are truly planning for the unexpected -- then you would be a fool to mess up your code in planning for it.

One-off code is perfectly fine. Of course you want to reuse code as much as possible, but it is not possible to reuse all your code. If someone wants to do something new, then you'll have to write new code. Hopefully you will do it without using cut and paste -- a horrible way to do code reuse -- but you will have to write some new code. Maybe you'll make a new subclass (my favorite), or add a new option, or refactor, or a combination of the above. And, sometimes, it really will be more expedient to copy and paste.

Re:Sleeping dogs

[shannara256]

> it's Perl, or perl, but not PERL.

No, not PERL, but the first two aren't interchangeable... Perl is the language, perl is the compiler. Intro to the Camel book, v5, IIRC.

-Jason-

Re:Sleeping dogs

[Anonymous Coward]

unfortunately, Mr Larry Wall has already demonstrated a willingness to make massive changes to the fundamental structure of the language...

It's kinda like when we let Hitler take over a few countries, and then expected him to play nice with France.

Extremem Programming - Refactor

[Strudleman]

You should read up on Extreme Programming, in particular Code Refactoring. It's a method of cleaning up old code. A very well written book, as well as an excellent code-housekeeping method.

Re:Extremem Programming - Refactor

[Anonymous Coward]

Do 'Extreme Programming' principles dictate that I get an afro perm and adorn my body with piercings and generic "tribal" tattoos? I can't even spell snowboard. Oh my...

Re:Extremem Programming - Refactor

[Anonymous Coward]

Do 'Extreme Programming' principles dictate that I get an afro perm and adorn my body with piercings and generic "tribal" tattoos?

First time I heard the term, I asked the guy "Is that where you jump out of an airplane with a parachute and a laptop?"

Re:Extremem Programming - Refactor

[(H)elix1]

Yah, I keep hearing this. Read the whole book - it recommends strong test plans, incremental development, pair programming, tons of snacks, and a maximum of a forty-hour work week. In itself, not a bad thing.

Interpreted by management?

• They see the part about everything being stuck out in the open, with the computers set up in the center of the room - cubes are overrated, offices are evil
  
• half as many boxes are required since you are "coding in tandem". See above for hint on how personal space is valued.
  
• Demand complex test plans that will cure everything. These same people never have actual requirement documentation for what the code should do. Classic example? Validation rules for a phone number... gets ugly when someone wants to put in 1 800 FOO-BARR, add an extension, deal with international numbers. Think you can put together a test script? Got to have requirements first, and that is never a hard thing to get from the business (shudder).
  
• I'm sure you have had complete management buy in on realistic time estimates now, so I wonder where the rumors come from that development tends to work more than an eight hour day. When they find out how long it will take to refactor the code base with through testing and daily updates, they will tell marketing and adjust the dates accordingly. Heck, I thought the Microsoft Project Plan defaulted to an eight-hour day - must be one of those rare Pentium errata issues that messes things up for me.
  

Not that I'm bitter, cause I'm not... but XP takes a handful of generally good practices and assumes perfect management buy-in and team communication. With any management support and teamwork, almost any method works.

Modules

[SCHecklerX]

Break the code into re-usable modules (or objects if you go another route besides perl, or even if you do, if you like that sort of pain), each programmer responsible for his own set (ie layout, calculation, database, etc)

Re:Modules

[Anonymous Coward]

Define a healthy architecture for your software (but do it in small steps so you can change it). Move all your software into modules, then replace each module as you need it.

CYA

[The Gardener]

Don't stop maintaining the old code code until the new code is on solid ground. No matter how sure you are that you can do it, the new code might never come through.

The Gardener

Don't do it

[mrpotato]

See the recent Joel Spolsky interview here, that was discussed on /. here.

Basically, Joel's take on a similar problem is: don't do it.

Unless you have a _really_ good reason to do huge change to a big codebase, don't bother, and make something more productive instead.

Re:Don't do it

[Anonymous Coward]

To define Joel's opinion better - he doesn't believe in scraping code and starting from scratch. He believes in migrating code slowly to where you want and never just scraping it.

Re:Don't do it

[Anonymous Coward]

Joel's an idiot. He doens't know what he's talking about half the time. Our company did a massive re-write of it's core component and yeah it took about 4 month to re-design, re-code and etc... But this gave us a major advantage over competitors. The product is twice as fast, as stable and ~5 times smaller. The beauty about it is that this opens new doors since the architecture's design is much better we can respond to customer demans much faster. Of course, there's the *right* way to do it and the *wrong* way. Netscape chose a bad way to do it by dumping it's old engine.

The only reason why Joel is famous is because of where he comes from and because software eng. is in it's infancy.

Code style and large rewrites.

[The Panther!]

It always seems to me that large rewrites (though I've never done one as large as the one discussed) tend to make bad code bases worse for a while. If there is any effort made to maintain functionality in a changing codebase, a piecemeal rewrite is more of a headache than a help. So, if you're planning on a big rewrite, please give your developers the full thumbs up to break everything and expect them to put it back together later--and expect to see nothing tangible in the short term. Or call it off.

That being said, my last company sat around and bickered about code style for nearly 4 months and produced no code that wasn't rewritten later. If you are going to concern yourself about style, settle that well in advance and make sure it's logical and consistent.

It's also been my experience that conformant code style is highly overrated. Once the Best Practices document extends beyond language constructs and caveats, into brace styles, spacing, tab size (yes, there was a 3 space tab stop standard at my last job--wretch), and even the naming of locals, parameters, members, constants, enumerations, etc... it got to be a thick ass bible of stuff that only a few people would digest or attempt to adhere to. The point I'm trying to make is, choose your battles. The hope is your developers will make sane choices independently, and use standards to help integrate different peoples' work together. Anything beyond that and it's pissing in the wind.

My $2e-2 or less.

Re:Code style and large rewrites.

[Anonymous Coward]

I have worked on many many large projects, and for the most part, I find the bickering over tabs and paren/bracket/brace placement to be just silly. I work on a large codebase with my company and everyone has a different preferred style. The answer? Use Tidy, or TogetherJ and reformat the code to your preferred style to work, then back to a stanard when you check it in. Everyone's happy.

While at my previous job, I did rewrites of large subsystems of an application server -- some for performance, some because the system was just ugly. This was both possible and mostly painless because the initial design drew clear lines between subsystems. Interface compatibity and clear lines make this possible. I also disagree with another poster that said rewrites make bad habbits work. Alot of times, you get ugly/slow/buggy code because the dev is working 16hour days trying to make a release date. Going back an cleaning up behind that is important for the long term viability of an app.

The only thing that complicates the situation here is the fact that your are not only cleaning up, but moving to new language. That could be a serious problem, esp since perl and java dont necessarily make for a clean "port".

Rewriting Perl??! Print it out and burn it!

[SimHacker]

There's no point to rewriting Perl code.

The Perl language was designed to maximise maintenance costs, and to keep your coders employed and busy with nickle and dime shit work for the rest of their lives.

Print it all out and burn it. That's the only way to deal with Perl code.

I hope you fired the irresponsible slob who decided to write that must code in Perl a long time ago.

-Don

Re:Rewriting Perl??! Print it out and burn it!

[Anonymous Coward]

You must got really hairy balls.

Horror Story

[Brontosaurus Jim]

My firm went through this sort of thing just two years ago. The PHB at the time decided, for some reason, that our 300,000 lines of semi-poorly written C code, and 50,000 associated lines of Java (Dont' ask).

Anyway, it took 7 of us over 2 months to get even halfway done. The pressure the boss was putting on us was awful, and he didn't really even understand what we were doing, even though he was the one demanding it. I think she read it in a trade mag somewhere. God I'd do a lot more work if she didn't read that shit.

Anyway, about halfway through the "Great Leap Forward" (as we [appropriatly] named it) the boss quit, and the next boss, who so far has been fairly clueful. He didn't think the whole deal was needed, but he was pressured by the former bosses husband (the CTO) to get it done. Seriously.

Hope yours goes better than ours. From what we did, heres some tips I can give you.

1. Be consistant through the whole thing.
2. Make sure everything is planned before you start. This was the one part we got right.
3. The team you have should have worked together before, because this sort of task requires previous knowledge of eachother.

Other than that, my condolences. Or maybe it will work better for you.

Good luck!

Re:Horror Story

[bolthole]

Anyway, it took 7 of us over 2 months to get even halfway done.

That was the problem. Too many people involved.

Re:Horror Story

[ScottMaxwell]

The pressure the boss was putting on us was awful, and he didn't really even understand what we were doing, even though he was the one demanding it. I think she read it in a trade mag somewhere. God I'd do a lot more work if she didn't read that shit.

Maybe he/she was just distracted by his/her sex change operation.

the former bosses husband

I'm not touching that. (So to speak.)

Don't ask this here!

[Will The Real Bruce]

Slashdot is a site built around perl scripts, so one might expect there to be a considerable amount of knowledge and interest about a topic such as this.

Could you, dear writer, perhaps be the maintainer of a GIANT collection of perl scripts badly in need of a rewrite?

Are you the maintainer of Slash??

Rule #1

[ackthpt]

We're still in the middle of a sweeping change and lemme tell ya, make d@mn sure there's someone accountable for managing the whole project from beginning to end, particularly this being their main focus.

Transitioning in new managers or having the current manager only look in on the project once in a while is as sure a path to madness and doom as no management at all.

Our due date was mid-August, we'll be lucky to get it through testing and into production by January 31st. All the while with the logjam we're having to put pieces of it into production and cross our fingers that the new changes don't break anything.

Love to talk more about it, but need another gallon of coffee.

If it ain't broke...

[bc90021]

...don't fix it!

"In the interest of consistency and reducing redundancy." There's only 1 MB of code, and while that is a lot considering it's all text, I suspect that you will find yourselves recreating (and then refixing) most of the original problems that are more than likely fixed by your existing codebase.

Old adages got to be old adages because they represent sound wisdom. So if it ain't broke, don't fix it. ;)

bc90021

"Python is executable pseudocode; Perl is executable line noise". - Someone wiser than I.

That there is no such thing...

[pyrrho]

You are going to rewrite the system from scratch. Design from scratch. Your new design might be able to use some old code, if the old code is useful.

A large scale retrofit is really an oxymoron.

IMHO, but with 15 years experience.

Re:That there is no such thing...

[BryceH]

Mod parent up.

very well said.

Re:That there is no such thing...

[awol]

I am working for a large company, which was only tiny when I started working for them (over seven years ago). In the early days we were a bunch of really smart guys, a few of which could be characterised as zealots. One would g home on a friday and come in on a Monday to find a fundamental component of the system rewritten in a new and interesting way, even things like new tools or processes that we would all then haveto use. This kind of retrofit never caused us a problem (seriously never) some of the decisions have even been subsequently validated by industry changes.

BUT

Two things. First, in truth we wre cowboys. That kind of thing is such a bad idea that you cannot do it and develop consistent quality product, particularly in larger teams, greater than six or so (or geographically diverse teams). Second we always were able to run these systems in parallel before we switch over to the new way. This is IMPORTANT and some other posters have said it before. Don't get rid of the old until the new has been proven (as for how to prove it well that is real hard but don't underestimate just how hard).

As an aside, we tride a pretty major redesign of a system a few years back and it did not really add any value, but that was because of a lack of a clear objective and we lost the budget after six months. So Try and make keepable improvements with shorter timscales.

Re:That there is no such thing...

[Kirruth]

For a major refit, re-design should be the bigger part of the effort. For one thing, eliminating features or modules that nobody really uses can cut a huge amount of effort. That's not to say code should be thrown out wholesale: quite often real gems that solve real problems are found buried amongst the junk.

Not all at once, either

[dimer0]

This is an excellent suggestion. What I always try to do is break the codebase up into loose-coupled modules, and then take the nastiest one of them (well, I guess you could pick which one you want to do first based on anything, really) - and convert that to your new technology.

Doing this will help big time when something goes wrong, which it will. The area that caused the problem will be much more focused.

This also lets you start taking advantages of the benefits of your new technology much sooner. Roll the new in with the old - in production. Less scary as well. Small steps.

Listen To The Experts

[ChuckPollock]

The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet. -- Michael Jackson

Document

[Tosta Dojen]

I suppose what I have to say almost goes without saying, but I know so many programmers who neglect it, I will say it anyway. There are two things you have to know when going back into old code.

1) Know what you are doing. 2) Know what the code does.

Both are expedited by good documentation. This is so important, it deserves to be written thusly:

DOCUMENTATION!

Write everything down. If the code is not commented, figure out what it does and write it down. When you add a line or a module, write down what it is supposed to do. Declarations? Write those down too. Document everything so that you can figure everything out, both now and down the road when you decide to fix something else.

This is the voice of experience. I have had to reverse-engineer my own code 6 months after I wrote it because I failed to document anything. Learn from my mistake.

Re:Document

[JeffFurry]

Tosta Dojen wrote:
This is the voice of experience. I have had to reverse-engineer my own code 6 months after I wrote it because I failed to document anything. Learn from my mistake

Which leads to one of my favorite development sayings:
Write maintainable code, so that a stranger working onit six months from now will be able to figure it out. And remember, that stranger will most likely be you!

"I may make you feel, but I can't make you think" - J. Tull

Re:Document

[chris_mahan]

Ever so true...

I have a saying: There should be 5 lines of comment for each line of code. (i'm including the lines of ************ in there too though)
Each program should have a complete changelog, with dates and emails and www pages of authors, a complete breakdown of all functions, at the top, in english, not a mix of english and pseudocode.

Also, a second document should exist, in html (or whatever) that shows the purpose of the program, the strengths and shortcomings of the program, the runtime requirements (dependencies, OS versions, hardware, memory, HDD access permissions, etc), and any outstanding issues, requests for change, etc.

Also, there should be a third document, created before, outlining in english all the expectations of the program to be written. Specs, in fact. This can be html or text, as the programming team prefers.

I program in ASP, so it's very easy to spaggettize. I have learned to enforce iron discipline. Because there is nothing like trying to read through 900 lines of html-producing code for that one thing management wants to change (I call those manabugs--things management considers bugs)).

Re:Document

[Arandir]

Also keep around a UML diagrams for the program. On Linux, the best UML tool I have seen is Dia. Save these as png and put them in your html design document. I would have one large "overview" diagram, with detailed diagrams for reference. When someone isn't familiar with the code, a roadmap goes a long ways. A UML diagram is a roadmap for your code.

Re:Document

[mbrod]

We have a new manager of the department that is pushing the documentation aspect real hard. We have a nice tidy drive share with volumes of documentation. I have been laying the groundwork for a massive new web system we will be creating and I find myself spending more time changing and organizing the framework document and DFD's than writing code.

I think it is very dificult to balance getting things done and writing the documentation. It seems to me a number of the programmers are a bit burnt out on a project they were excited about by the time they do the Milestone graph, the DFD's, etc.

My point is there has to be a balance and I see programmers who are hard enough to motivate to begin with just spin their wheels in the documentation and keeping it all up to date if it is to extensive can kill them.

Anyone else have a success story of balancing making developers document well with writing kick ass code please share how you did it.

avoid the "sweeping rewrite"

[CamShaft]

In my experience, if you have something, and it is functional, then you are better off to migrate and refactor what you have to end up with improvements rather than doing the slash and burn and rewrite.

The advantage of the migration and refactor approach is you can do it in a way that leaves you with a functional product the entire time, giving you the ability to respond to new requests/requirements as they arise.

If you try to do a full rewrite, and some new urgent deliverable comes up in the middle of your rewrite, the only way you are going to be able to deliver in time is to hack it into your old functional code.

Think again

[pkphilip]

From what you say, you are planning on making these changes to clean up the code and make it prettier.

I would strongly urge you to reconsider this as the probability that you will end up breaking parts of the code while "cleaning" it up is quite high..especially since you seem to have a fairly large code base ~1MB

Ugly code containing redundant stuff is still better than beautiful code that doesn't work.

Re:Think again

[flavor]

I agree with most of what's being said here. Indulge me while I try to rephrase it a little differently.

Rewriting a large codebase may gain you flexibility in building out future functionality, and reduced operational costs.

But be warned that there is very little you can say to justify a rewrite to management/marketer types. To them, you're taking something that works and making it ... still work.

I've found that many users are willing to risk encountering bugs if they're going to be rewarded with better functionality. But you would risk introducing bugs while offering users no reward. This could get rough if they're the ones paying the bills.

Regression tests

[Weasel Boy]

Make sure you have a suite of tests that produce known output for your old code, so that you can ensure that the new code works in exactly the same way. Don't add anything new until you are proven conformant with what you had before.

Re:Regression tests

[Doctor Memory]

Why is this modded Offtopic? Absolutely the first thing you do when you are rewriting code is to write tests against the existing code, so you can make sure you haven't broken anything.

I would suggest that you start small, have lots of code reviews so that you can build consensus on best practices (clue: "best practices" that only half the team agree with are not "best"), and try to do this *without* incorporating any bug fixes. This way your new code will be "bug-compatible" with the old, and you'll get consistent results from your test suites.

If it aint broke...

[ZaneMcAuley]

Dont break it.

Seriously, yes I understand there are maintenance costs involved in maintaining old code but really, I believe the statistic is that for every 10 times uou modify a part of code, a new bug is introduced.

How often is that code hit during a maintenance cycle? How problematic is that code?

Re:If it aint broke...

[de_boer_man]

More realistically:

For every time you modify a part of code, ten new bugs are introduced.

Re:If it aint broke...

[ZaneMcAuley]

Has anybody got links to real world stats on coding for things like bug introduction vs modification rates etc...

See no evil, hear no evil, ...?

[ProtonMotiveForce]

Why do people mod comments about alternatives as Flamebait (and presciently, this one)? Are people afraid to hear that you should't write large scale systems in Perl?

It really is valid (and in my opinion, correct) to say that if you _are_ going to do this you should look at other technologies and languages. Perl is for system administrators and system administrators-cum-developers, not real software development. Look at java. Look at PHP. Look at commercial and non-commerical web application systems, like Zope. Or don't rewrite it at all if it works. But for God's Sake, don't rewrite it in Perl - it's pointless.

Re:See no evil, hear no evil, ...?

[Anonymous Coward]

Yay! Not only are we doing a wide-ranging rewrite, we're switching to a new set of technology! That the dev team have no experience in!

Re:See no evil, hear no evil, ...?

[hondo77]

Why is this marked as interesting instead of flamebait?

Perl isn't used by real software engineers but PHP is? Gimmie a break (and don't get me started on Java--talk about pointless).

Not understanding Perl is no excuse for name-calling.

If they want to rewrite their Perl (which is probably a bad idea in the first place), Perl IS an excellent choice.

Re:See no evil, hear no evil, ...?

[m_ilya]

I can understand when people say that Java is better than Perl for large scale systems (I don't agree but I understand why people think so). But PHP??? Why do you think it is better suited for large scale systems than Perl?

Re:See no evil, hear no evil, ...?

[chromatic]

Perhaps people consider these comments flamebait because your comment boils down to "Perl is unsuited for large applications because I don't like it."

If you'd care to back up your assertions with specific examples, you might have a better chance of being taken seriously. I'd tend to avoid assertions like "because Matt Wright wrote some awful stuff several years ago, it's impossible for anyone to write clear, thoughtful, or well-designed programs in Perl."

Since you didn't spell it PERL, I'm assuming you do have a serious point.

Re:See no evil, hear no evil, ...?

[spudnic]

What would the advantage be in changing to another language? Assuming the team develops coding standards and creates clean, documented code that can run under mod_perl, why would PHP or java be better?

The team is obviously already familiar with perl, and some code could be reused after a bit of cleaning. Perl is perfect for applications that require a lot of string manipulation and as a front end for a database. This is what most web programming involves.

The only drawback that I can see in using perl is if your team is messy. Perl will allow you to be messy, but it doesn't have to be. Perl can look just as clean as any other language. I go back to apps I wrote 3 years ago in perl and can follow every bit of it. It's all commented and has plenty of white space to make it easy on the eyes.

I've written several large perl applications for the web and yes, I do know other languages, but perl is better suited for this type of work. It's fairly quick to put together and performs well. Now I wouldn't try to write an OS or first person shoot'em up game in perl, but for this it's just as viable of a choice as anything else.

I'm also not saying that PHP or java would be bad if you have a good grasp on it and are starting a new project. Remember, TMTOWTDI is good!
Just don't use ASP! ;)

Re:See no evil, hear no evil, ...?

[khkramer]

> Perl is for system administrators and system > administrators-cum-developers, not real > software development. Look at java. Look at > PHP.

Mmmm. I write large-scale web applications for a living, and I do it in Perl. By large-scale, I mean sites that are expected to support hundreds of thousands of page-views per day, serve hundreds of thousands of distinct users per month, and collate hundreds of thousands of distinct chunks of content into dynamic pages.

My company is a high-end development shop. We generally bid on projects that will take six to nine months to complete, and we only do jobs for clients who understand how we work and why. Part of our approach is to use very small teams of extremely experienced web developers. We usually deploy four programmers on a project. Other companies that bid against us sometimes use several times that many people on a single development team. Another part of our approach is to build everything on top of our open-source development framework. That sometimes used to be a tough sell ("what, give software away for free, heaven forfend") but these days, most customers are pretty receptive to the twin "more eyeballs means better code" and "you're not locked into our closed, proprietary product" arguments.

We also generally build small clusters of dual-processor Athlon or PIII machines, whereas our Java- (Oracle-, IBM -, BEA-, etc) wielding colleagues often specify absurdly-expensive hardware.

Perl is flexible, complete, performs relatively well and has an extroardinary base of skilled developers and re-usable components (CPAN). We couldn't work as quickly or write code that is as concise and maintainable using any other currently-available language/approach. Most of the lines of code that we write actually go into Perl modules, with HTML::Mason templates handling the dynamic web-page generation. We can push Mason out of the way and use straight mod_perl for small, defined tasks. And we can easily integrate C code into our Perl frameworks in places where performance is really, really critical (though those places are rare, as when push comes to shove, one is almost always waiting on the database).

There are lots of things that aren't "right" about web development. The package that results from gluing HTML and program logic together in a stateless execution environment is sometimes a little lumpy, and unavoidably so. There's no magic bullet toolkit, and (as with other specialized programming arenas, like graphics or embedded systems) a lot of hard-won, domain-specific knowledge goes into the development of a fast, reliable, maintainable web app.

The Perl/Apache/Mason combination that we use is far from perfect. But it's better -- for us -- than any of the alternatives.

I really like Java, and have written big systems in that language, too. If for some reason I had to manage a very large team of programmers, or had to manage a team with a large percentage of less-experienced programmers, I would use a Java-based solution. Java is a more rigid language than Perl, and the structure that the language provides would be a useful management tool in those contexts. But for my small teams of skilled hackers, Perl is more productive. (We have an extensive, evolved, self-imposed "structure," so we don't need the language to impose one on us -- in fact, it gets in the way.)

I would never use PHP for the kind of work we do. PHP just isn't the kind of powerful, flexible, complete environment that Perl is.

Zope and Python are really neat. I'm a fan of the work that folks on that side of the fence are doing. But Python+Zope don't offer us anything new that Perl+Mason+Comma don't. I also like Perl more than Python (which is a subjective preference), and think that the Perl development environment is more mature (which is a subjective judgment).

So don't listen to the folks who tell you to dump Perl. You should certainly consider all of your options and make an informed decision about core tools, but anyone who thinks that Perl is just a "scripting" language, or that it doesn't scale, hasn't been paying attention.

To finish this up with a little more specific advice to the original poster: You mentioned "150+ perl CGIs" in your question. You should consider moving away from the CGI model, if possible. Take a look at HTML::Mason, which is a very good embedded-perl environment. You can build solid, consistent application layers using Mason as a base. Also, I couldn't agree more with the folks recommending writing perl modules and requiring complete regression tests for each module. There are lots of ways to write tests, but in perl-land one of the easiest is to simply make a t/ directory down your module tree, write a bunch of scripts in that directory named <some-test>.t that print out a series of "ok <n>\n" lines, and use make test or Test::Harness::runtests() to invoke them.

Re:See no evil, hear no evil, ...?

[Tablizer]

(* Why do you think it is better suited for large scale systems than Perl? *)

IMO, it depends on how people organize the system. One style may favor Perl (or any lang) but not another.

The reason for so many "language wars" is that people tend to pick languages that best fit the way that they think, and people's heads all came from different molds. Thus, there is not One Right Answer.

I tend to use tables/DB's to manage the larger aspects of a system, thus the difference between projects of difference sizes does not seem to make that much difference in my coding approach. The DB is like a big (mother board) bus, with individual tasks (chips) not too directly tied to other tasks (chips). The communication/messaging is mostly between a given task and the DB tables, rather than task-to-task.

I personally don't like Perl because it has a culture of criptic code and "there are (too) many ways to do the same thing". Lack of consistency makes maintence harder, for me at least. But, another Perl fan may enjoy the decription, and thus are more likely to stay awake. It is a challenging puzzle to them.

Re:See no evil, hear no evil, ...?

[m_ilya]

Having seen different style projects I strongly dislike "Database centric" design (i.e. very weak or missing middle layer). I'm not sure if you talking about such design but it looks like you mean it.

Bad programmers can write cryptic Perl but for good developers it is not a problem to develop clean middle layer and frontend layer in Perl. Perl is full featured language which can be used to build big OO projects. On other side I always was under impression that PHP is quite limited language because initially it was designed as quite niche language (embeding in HTML web pages). This is why I think that Perl have more potential than PHP for large scale projects.

Architect it to death

[ADRA]

There is no use in doing a rewrite unless you do it right from the beginning. If you can't spend a decent amount of time planning the architecture of the system, then stop now, and quit.

Also, since you have decided to pour resources into this thing, then my opinion would be to make as much of your code generic so that you don't have to make code changes later. It doesn't matter if there is an initial performance hit with the systems, becuase in the short term, you can convince your boss to get a new leet server, and in the future, hardware needed to run your apps will be trivially cheap anyway.

If you are going to cut a new release, try to avoid going back and taking snippets of code from your old system. It makes people slide back into the odl paradigm and can cause detrimental effects on the bottom line of your new system. This is new, so the least exposure to the old one, the better.

Re:Architect it to death

[Hairy1]

No Don't!

In my early experience in programming I found there was an advantage to up front planning. It seemed like the more planning you did the better the project would do. So there was a tendancy to do more and more up front planning, getting into more minute detail.

However although we spent more and more time planning the payoff in terms of being able to predict schedule and budget diminished. Its classic diminishing return. But why?

Planning a project is like forcasting the weather. You can plan a small project in minute detail for a short time (like being able to predict the weather a day in advance), and you can make vauge predictions for some time out (ie you can know the probability of rain based on the month - statistically on previous years), however you can't plan a large project in minute detail.

The reason is that if you get any of the initial decisions wrong, and there is a minor deviation from the plan early on, this will be reinforced to cause more changes, and this will snowball to change many parts of the final system.

An analogy is trying to get somewhere with a set of instructions. If the instructions are faulty at the very start of the list it doesn't matter that the rest of the instructions are correct - you still get lost. Detailed plans are a little like a set of precise instructions. There is no assurance that all of the instructions are fault free. In fact there is a good chance they will be faulty as they will almost always be written by someone who hasn't been to the destination (ie hasn't written the same program before).

It is better to be an explorer. Your "plan" should be like a map. Only you start out not knowing the terrain - only the general direction of your destination. You map your first days travel and set off. Every day fill in some detail on the map and plan the next days walk - always keeping in mind where you final destination is.

Programming is a journey - not through any physical space, but through the endless possibilies of the programming solution space. This space is filled with broken, useless programs. Its your job to navigate your way to the best solution.

PS : I know this hasn't been very practical, however I have previously posted some more concrete real life advice under this article.

Agreed

[ackthpt]

A large scale retrofit is really an oxymoron.

Often I've had to modify systems which, once I understand how they work, which parts are actually still used, what's redundant/poorly written, I've usually pulled off changes in 1/2 to 1/5 the amount of code. Nice, neat, clean well documented stuff, too.

Use framework, seperate code form display.. and..

[cybrthng]

Well, i can't say it enough. Use a web framework, i don't know of any for Perl off the top of my mind, but i use Resin.

Resin is a JSP, Servlet, XML, XSLT application server that support all the latest and greatest EJB components and Managed persistance on the database and makes a great framework to build from.

I have Postgresl for the Database, and use beans to run queries and output via xml and then i have XSLT draft xml data into the wonderfull HTML code.

The beauty is, my code is in beans, servlets and jsp's. My HTML is in .xtp files (in the case of resin) and simple XSLT sheets parse the XML to render to the user.

Just means i can produce output easily for WAP, Palm, CE, Normal Web Browsing, EMail, and what not without modifying the backend. Just create xslt using a session identifier to bring up the corresponding stylesheets for whatever device is acesssing the page.

Enough about java, but something similar, be it in house developed would be your best bet.

I also get away with using a Swing applicatoin to manage the database, users and run reports provide an easy to navigate gui which just interprests the same xml data that would be retrieved by the html client. Not a single change to the backend since i'm using the beauty of soap, jsp's, servlets and java.

Virtualize your interfaces, standardize on your backend and use re-useable components. Perl is similar in many ways that you can load libaries and abstract your code from the display which will save you tons of hours of hassle in any future upgrades compared to the bit of changes you would do now

Don't Listen

[augustz]

A ton of people will tell you a ton of things, never having retrofited anything.

- Do not undervalue the investment you have learning your existing coding language. New challenges await you if you jump on a new language like java. Make the jump if you are excited about learning about the new language.

- If you use your existing coding language you will literally fly through the retrofit. Do it piece by piece. Make all those changes first, then test app, then make next set of changes then test. The simple fact is, most wasted time is spent on bugs not working on performance, and you've already knocked down a lot of bugs, don't let them pop back up by blowing everything up. There are books on this.

- Sometimes blowing everything up is worth it. Do it right this time. Realize it won't be as perfect as you might think it will be.

- Remember there are countless open source and shareware products that tried to create TNG with a total rewrite, got nowhere, and ended up improving their existing product. Remember the lesson, bite off what you can chew.

- Spend a week poking around researching possibilities. I do this all the time, bookmark things I think are important. Then for the next project you've got all the little things you might forget at your fingertips. Optimzations/Tools/Paradigms. Think you know it all? You'd be suprised at what is out there and what you missed. And what you spent a month in house re-inventing. This one's important.

- Use open source software. Nothing beats free. Nothing is more fun. Java's ugly standardization history makes me puke... the BS Sun has pulled with Java is staggering. That the Java Lobby swallows it and loves it even more so. This is irrelevant to your question, and not fair to the Lobby, but I like to give them a hard time.

- Colorary to Java. You need less abstract design then you think. Endless object hierarchies will weigh you and your app down... Their are books on this too.

- You need more documentation then you think. Ever found code someone ELSE wrote too EASY to follow. I don't think so. Especially if you are using perl and someone is enjoying the line noise capabalities perl allows. Perl has 20 ways to do EVERYTHING, you may not know the latest or twistiest. Document as you WRITE the code. Do not leave at the end of the day without catching up the docs. A week of documenting is the worst form of hell, avoided with a minutes worth of clarification each time you write a function/class.

- Hardware is cheap.

Anyways, have fun... and good luck. Be interested to read what others have to say.

Re:Don't Listen

[Beryllium Sphere(tm)]

>- If you use your existing coding language you will literally fly through the retrofit. Do it piece by piece. Make all those changes first, then test app, then make next set of changes then test. The simple fact is, most wasted time is spent on bugs not working on performance, and you've already knocked down a lot of bugs, don't let them pop back up by blowing everything up. There are books on this.

This is good advice. To be more specific:
1. START with your regression test suite
2. Then add self-documentation features like standard naming conventions. Seems dull and bureaucratic and pointless but really truly saves maintenance time.
3. Have a standard comment header for each function. The standard should answer questions like "Can that argument be NULL?" and "What do the error returns mean?"
4. If you're going through every line already, do a security audit.

There's good advice in the refactoring books, for example http://www1.fatbrain.com/asp/bookinfo/bookinfo.asp ?theisbn=0201485672&vm=

Re:Don't Listen

[banky]

hardware is cheap, but bumping up the hardware requirements when you're talking about pushing out your new code to several hundred machines in a hosted data center will quickly kill that Xmas bonus you were hoping for...

Re:Don't Listen

[augustz]

Yes, I agree with this. I actually dislike some of the easy but (what I consider heavy) choices like Java for this reason. You can get by with so much less so easily for many projects, and support literally 5x as many users per server. For a small business this is the difference between profit and no profit. For a large business this can be the difference between .com and .bomb.

All too many times I see sites begging for money for hosting/bandwidth. Take a look at their HTML/CSS and see it is HUGLY bloated (no linked css, prevent all caching including image with default cache buster installs) and not gzipped, and I wonder, if what I can see is so bad, behind the scenes it is probably even worse. (ie dynamic page gens where none are needed). Which I had included this in my list and left of the hardware point, which I agree is the wrong message.

But damn, if you do code right, hardware is so cheap I can't beleive it. I'm convinced some 10 machine projects with bad coding can be supported on a single machine now.

Re:Don't Listen

[Lumpy]

I agreed with everything until the last line. That statement needs to be violently beaten out of every programmer, manager, and engineer on the planet. I almost want to suggest that the death penalty be decreed on that statement od heresy.

The "hardware is cheap" mantra is exactly why we have crapy code and bloatware. "I dont have to optimize, hardware is cheap." "Why make it efficient? hardware's cheap!" "I dont care that it is slow, hardware is so cheap these days"

Please, everyone, cary stones in your pockets and throw them really hard at the next programmer that makes that statement.

Re:Don't Listen

[Arandir]

Remember there are countless open source and shareware products that tried to create TNG with a total rewrite, got nowhere, and ended up improving their existing product.

And there are dozens of examples where the reverse is true. Enlightenment is one example where the entire code base was rewritten from scratch resulting in a far better product. E17 is going through the same process all over again, but for a different reason. From what I can see it's going to be a huge improvement that will spill over into other projects.

Re:Don't Listen

[RollingThunder]

That last line is making me have to BS my boss.

"Sure, the dual tbird appro should handle 10000 users"

omitting, of course, if it was all written in clean tight C, instead of java.

Re:Don't Listen

[Anonymous Coward]

You need more documentation then you think. Ever found code someone ELSE wrote too EASY to follow. I don't think so. Especially if you are using perl and someone is enjoying the line noise capabalities perl allows. Perl has 20 ways to do EVERYTHING, you may not know the latest or twistiest. Document as you WRITE the code. Do not leave at the end of the day without catching up the docs. A week of documenting is the worst form of hell, avoided with a minutes worth of clarification each time you write a function/class.

Worse than having to read someone elses poorly commented code is having your own code come back to haunt you. You may laugh, but after the first 100K or so lines of code I wrote, I suddenly realized that I couldn't trust myself to remember all the details, especially if the code needed an update say 1 year or more after I wrote it. In my case the most important things I need to know are what the data objects represent and what the methods (functions/subroutines or whatever you want to call them) do. Trust me, if you complain to your boss that you can't read someone else's code you might get a sympathetic ear, but that trick is not going to fly with your own code. In the current job market, it will be hard to move to another job and leave the code behind, and new employees are not likely to be hired to do your maintenance, so you need to make sure that you can come back to that code and run with it in a very short time frame.

Project is Dead

[Anonymous Coward]

A rewrite is a sure way to waste time and resources (namely money). Unless you are an hourly consultant just doing the rewrite for the money. I can see refactoring your code but a rewrite is death.

see joel on software

[kubalaa]

He says the exact same thing as you, only better.

Re:see joel on software

[Anonymous Coward]

I was thinking the exact same thing. Word for word it's ripped off.

Re:see joel on software

[bay43270]

Although I enjoyed his book, and respect his opinion on matters of interface design, Joel Spolsky is no champion of well-designed code. In his book, he even suggests that a programming language shouldn't be used if another can create the same user interface in less time... as if the time it takes to create the first version of a product should take precedence of other factors such as maintainability of code, reuse, and flexibility. He does a good job of regurgitating the works of other interface specialists, but he is not an expert in code maintenance.

Re:see joel on software

[kubalaa]

I haven't read his book; his site deals with coding issues, not interface ones. And I don't agree with a lot of what he says, but he's very practically-minded, and a good antidote to academic thinking.

KISS

[ZaneMcAuley]

Keep It Simple Stupid. Keep code simple, why have funky lines of code that looks awe enspiring. Its pig ugly. I work on the principle of KISS. When I am working on a code area that is being cleaned up or fixed, I try to simplify that area. The simpler it is, the easier and quicker it is to maintain and bug hunt.

Im a strong believer in managed code (whether its C# or Java, except managed extensions to c++ which are damn pig ugly :) Exception handling for seperating error handling from actual code that performs the actual work).

Make sure there is GOOD in code documentation (and out of code documentation) to explain the intention of that code. (Intentional programming is a research area btw, to program ones intentions).

You know when something is bad when you have to maintain that code later on, 6 months down the line or whenever. Thats when it bites you in the buttie.

Don't

[Pussy Is Money]

Don't do it. Don't go over your code making sweeping changes with the intent of making sweeping changes to your code.

Instead, very globally, identify the features and structures that you require in 3, 6 or 12 months time, from a marketing or similarly broad perspective. Identify the parts of your codebase that would need to be changed in order to achieve those goals (this is as good a definition of "out of shape" as any).

Then as you fix up your code: look at your fixes. And from these, formulate the rules that should guide your next development cycle.

that is LARGE?

[Anonymous Coward]

You must do some non-serious programming, I know of several people that their code far exceeds 100 megs. They are making changes to reflect new data structures and file types. I personally dont have alot of code to change, if there needs to be a 'change' I just start from scratch if the project is small enough or I take the existing and start hacking away.

Re:that is LARGE?

[Doppler00]

Code is unlikely to be that large. That would mean that you would have been typing 100,000,000 characters. I don't think that existing code libraries count, but original code does. Try to write a 1,000,000 character document and see how long it takes.

Re:that is LARGE?

[Anonymous Coward]

commercial software really is that large. actually 1MLOC of code is much larger than 1 megabyte and a lot of commecial systems have 3-4MLOCs of code.
windows NT has some 40MLOCs of code but thats an extreme example of bloated commercial software.

Re:that is LARGE?

[Anonymous Coward]

He did say he used perl. Anything more than a few kbytes becomes overwelming ;-)

Refactoring your project

[adamy]

To expand on the concept of Refactoring:

1. Write a test for a specific block of code.
2. Appliy the refactoring

You are going to want a good testing framework.

To expand on the modules post: Do a dependency analysis. If you are writing DB based code, look at what tables can be logically grouped together.
We did something like this at my company not too long ago. The basical level package we had was the security package, which identifies users and roles. Most other packages depended on this. All contact management stuff went into a package called Directory. All stuff for the people our system was managing went into Participant etc.

For each of the packages, split the code out into a set of interfaces, and a set of implementing code for business logic, and the UI required to drive that business logic. This is the standard breakdown for code. You may want to further pull out code into helper packages. Avoid the urge to call these helper or util, and instead try to name them based on what they contain: we have one called format for stuff like phone numbers and social security numbers.

Don't forget the make scripts. What ever build you use, it should be used to specify which modules you want to compile/deploy

I recommend a little UML modeling session for the end package structure.

Go in little pieces. After each refacotring, makes sure stuff still runs.

Good Luck

Re:Refactoring your project

[Anonymous Coward]

Refactoring is actually at http://www.refactoring.com/

Write the regression test suite _first_

[CatherineCornelius]

If you really have decided that you _must_ rewrite your code (and I presume that you have examined all the alternatives and found none of them viable) then my advice is to write test suites for your code _first_. To perform a functional rewrite, you need a way of automatically testing that the function is performed effectively and faultlessly by the rewritten code.

Get a copy of Expect, or even better, DejaGnu, and start scripting your tests.

Playing with fire...

[ZaneMcAuley]

Do you have the original engineers for that code?

Most knowledge is in theyre head, not on paper unfortunately. Theyre experience in that area can never be written down.

Re:Playing with fire...

[ZaneMcAuley]

erm, how is this a troll?

Re:Playing with fire...

[ZaneMcAuley]

Maybe the moderators (which seem to be on crack) should have to write an explaination for each moderation done to explain why they moderated that way. :)

Write the test suite first

[z19752002]

I suggest writing a comprehensive test suite first and making sure that the existing code either works or that the bugs are documented. Then run the test suite against the re-write as often as is reasonable. Use a good source control system. Have seperate development and test systems (it's amazing the number of people who try to test on development systems).

From experience:

[tqbf]

Plan a big sweeping rewrite; idealize the system you design based on your experience with the real one. Make the design clean, simple, and flexible.

Don't build it. Instead, evaluate the code you have now and plot a course towards the idealized system. Approach the actual work of the "retrofit" incrementally. Count on having multiple customer-facing revisions of the software tagged and QA'd before the system you're delivering looks anything like the planned rewrite.

Taking baby steps towards a new design is probably the only way you'll ever migrate your project to that design. With the knowledge you've accrued working on the old system, it probably seems straightforward to start from scratch. Even if this isn't wishful thinking, though, it's a waste of time. Part of the discipline of design is an understanding of where the "hot spots" are that can't tolerate inferior implementation, and how to tell those hots spots from the spongy mass of integration, reporting, tracing, and sanitizing that is neither performance sensitive nor mutable enough to justify engineering effort.

You can take early baby steps that make it easier to make holistic changes down the road. Refactor relentlessly. Migrate code recklessly out of subsystems and into common repositories and libraries. I've found it handy to distinguish between "proper" shared library and "dumps" of utility code that don't need scrupulously conceived interfaces.

Most importantly, design for testability. In this respect, the biggest asset you have is the steaming lump of old Perl code you're facing; use it to figure out the expected behavior of subsystems. Write replacements, in modules with clean interfaces, and unit test them. A unit test probes code (functions, statements, internal states) --- NOT entire programs. You'll work ten times faster when you can move forward as a team knowing what components you can trust and what components you need to worry about. You'll work ten times slower if you haven't clarified your outputs, side effects, and return values enough to know whether your replacement parts are valid!

We've seen articles on Slashdot before about this and I agree with the prevailing opinion: rewrites are often seductive traps and time sinks that don't offer value to customers. A better mentality that will eventually get you where you (think you) want to be is to adopt a strategy of constant measurement (testing, profiling, debugging) and improvement.

Re:From experience:

[Anonymous Coward]

I've done several major rewrites. I agree with tqbf and other posters that it's risky, time-consuming, and the results might not be worth the risk and time.

But there are times when you have to do it. I migrated one 250,000 line code base off a bad, bug-ridden 3rd party vendor library onto one that was better. Putting more patches into the code to work around the vendor's problems was getting the team nowhere. It was time to cut the cord and ditch them.

Now lately I'm working on another huge program that has a bad case of bit rot, having been worked on by multiple people, some of whom were clever but too rushed to do the right thing, and some of whom hadn't a clue and were making the mess worse. It's code that used to have an architecture before it was hacked to death.

One piece of advice: even if your goal is a radical rewrite, never tear the old stuff so far apart that it stops working for a long period of time. Creating a lot of cleaned-up but non-functional code is not progress. Plus if you keep it in a working state, you can always halt the cleanup in a less than 100% state if you actually have to make a release.

Try INTERCAL

[Doppler00]

Rewrite the code in intercal to really impress people!
You're code base will go from over a meg to at least 10 times that, and as a bonus, no one will ever be able to understand it.

Re:Try INTERCAL

[Anonymous Coward]

as a bonus, no one will ever be able to understand it.

They're writing in PERL now - so that part won't change :o)

profile your code

[avandesande]

I have coworkers that throw around issues like performance, which can be a red herring if you are doing a web site with only a few users. (one guy insisted on doing all is string stuff as string buffer instead of as strings because it would increase performance)
You may want to look at your code in a similar fashion. There may be just a few modules that are causing problems with redundancy(or whatever is making you want to rewrite them). Why rewrite the ones that have worked faithfully without any problems?

Re:profile your code

[Anonymous Coward]

While I take your point that it can be fruitless chasing an extra 0.001% performance increase, I also know that there is a correct way to do everything. If you are doing string manipulation, the correct way is with a stringbuffer, since that is what it's for. If you want the simplest, most generic way of doing things, use a scripting language. If you're in java or c++, use the classes correctly.

Re:profile your code

[avandesande]

Performance is nice, but so is fast time to market and maintainability. Unless a site is going to have high traffic and hardware constraints, the latter two come first.

Parallel development

[rgmoore]

It might seem like and obvious step, but don't throw away the old system until you're sure that the new one works! Keep somebody minding the existing, working system so that if/when your attempt to completely rework it fails you won't be stuck. Once you have rewritten it, try setting it up on a trial basis in parallel to the working system so you can find the crippling bugs before they take down your system.

While it's not a perfect example, Slashdot is actually a decent example with their switch to their new system. They kept the old, crufty version as the primary and set up a beta site with the new software. They knew that there would be problems and got some of their more loyal users to test the new system and only switched over it after they were pretty confident that they had gotten the worst problems out of the way.

You can afford to take a few more risks as long as you keep a known working system around as a fallback.

Google

[Anonymous Coward]

Did a search on Google for this topic... Here is a link to the cache since the site seemed to be down.

code-retrofit

Document More!

[doorbot.com]

Actually, one thing that can be very very helpful is to also write down your assumptions for each function. What format are you assuming the variables will be in when passed to a function? What if they're passed incorrectly? What if someone tries to "hack" the function? How would they go about doing it via a form submittal (for example)?

This is a good exercise and should lead to much more secure code.

Unexploded bombs, not sleeping dogs

[ChaosDiscordSimple]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

Yes, and if they're cryptic and uncommented, they are worthless. Eventually one of these incomprehensible, magical fixes will stop working. Perhaps the bug it works around is fixed. Perhaps how the function is being used changes to previously unexpected behavior. Some poor engineer will look at the little big of magic, scratch his head, and be forced make a blind decision about how to fix it. Perhaps he can change the code while leaving the bit of magic in working, but he can't be certain, since he doens't understand it. If the collection of cryptic tweaks becomes dense enough, any attempt to fix a bug or add a feature becomes highly risky.

On a related note, don't let this happen to you. If you add one of these strange little fixes, for the sake of the programmer that follows you, document them. Just a little "Need to toggle the Foo because Qux 1.4 is correctly fails to do so" will bring tears of joy to the eyes of future programmers.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

You're wrong. The comments you recommend will only /add/ to the mass of code to be grokked. Don't forbid commenting; but don't consider them a solution.

Instead, refactor and unit test. Unlike comments inserted into the middle of the code, unit tests will fail and point you to the reason for the failure. In the above example, when we upgrade to Qux1.5, the unit test which asserts that the Foo is untoggled will fail, and will point right to the function which made the assumption. Bingo -- a quick fix.

-Billy

Re:Unexploded bombs, not sleeping dogs

[CodeRx]

Comments are fine as long as they explain why the code is doing what it is, not what the code does or how it does it (that should be obvious from just looking at the code).

Bad Comment:

/* Disable widget by calling disable() method */

Good Comment:

/* Disable widget so it can't be accidently clicked */

-sean

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Comments used that way are like perfume -- they cover the smell of stinky code. Perfume isn't bad when your code stinks, but better is to make your code not stink.

A better solution would be to extract the thing you're commenting into a function, and name the function so that the comment is unneeded. Don't forget to write a test which checks the purpose of the function. Then (and only then) delete the comment.

MakeNotAccidentallyClickable(widget);

See what I mean? Now it's impossible for your comments to become out of sync with your code -- your code IS your comments.

-Billy

Re:Unexploded bombs, not sleeping dogs

[Anonymous Coward]

My favorite comment to see in the code "This is a bit of a hack"

Re:Unexploded bombs, not sleeping dogs

[p3d0]

I think that the idea that code can somehow take the place of comments is bogus. Code, no matter how clearly written, only tells you what is happening. It can't tell you why.

For instance, why did you make that widget not accidentally clickable? Why not others? Is that widget in particular danger of being accidentally clicked? Are the consequences of accidental clickage very bad?

Code like this seems clearer to me:

/* Make sure user doesn't accidentally delete the document */
disable(widget);

Or:

/* This widget is very large, and liable to be clicked accidentally */
disable(widget);

Re:Unexploded bombs, not sleeping dogs

[medcalf]

If it's that important to you that "your code IS your comments", just write in COBOL. COBOL is documentation that runs.

Re:Unexploded bombs, not sleeping dogs

[ednopantz]

Mine too, and you know, it is the most helpful comment I have ever seen. Just when I am thking, "WTF??", I suddenly understand why the code looks like this, and yes, I feel their pain.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

I coded for the example I was given, at the level of detail I was given. You've added some more detailed scenarios; and I can code for those too.

The first comment should be a function name, not a comment. There's a LOT more you have to do to prevent document deletion! The comment, as it stands, is misleading, and later modification will possibly have to add more functionality. Just make it a function, so the functionality can be kept together.

The second comment says, "I, the programmer of this system, don't like the UI design." BAD SMELL. Use comments if you must, but a FAR FAR better solution is to fix the UI. Don't you agree?

Now, I'm sure you can invent other details under which those specific solutions are inapplicable -- but for every scenario you invent, I'll invent a way to code it so the comments aren't needed. The only exception is when time or lack of knowledge prevents removing the comments -- and the result is tragic.

-Billy

Re:Unexploded bombs, not sleeping dogs

[p3d0]

The second comment says, "I, the programmer of this system, don't like the UI design." BAD SMELL. Use comments if you must, but a FAR FAR better solution is to fix the UI. Don't you agree?

Hey, you're the one who gave the example, so you tell me why you would call the "MakeNotAccidentallyClickable" function. I can't tell from your code, and that's the exact problem.

It seems to me that my comments have helped you to diagnose problems in the code that were not obvious without those comments.

for every scenario you invent, I'll invent a way to code it so the comments aren't needed

Why would you want to do that? What is the advantage of your scheme over comments?

Your aim of having clearly-written code is admirable. I think code should explain very, very clearly what it is doing. However, explaining the "why" when it is not obvious from the "what" is exactly what comments are good at.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Hey, you're the one who gave the example,

I didn't give that example -- look at the history. That was given by someone who thought that the comment was needed (they provided a comment, too). I made a function which contained the code being commented, gave it a name which was essentially identical to the comment, and called it where the comment was. I also claimed that in any real work, I would have first built unit tests which confirmed that the function did what its name claimed.

so you tell me why you would call the "MakeNotAccidentallyClickable" function. I can't tell from your code, and that's the exact problem.

You use it when you want to make something not accidentally clickable. I haven't defined what "something" is, and I haven't defined in what context it's useful. The first job is for the type system, and the second job is for the object system.

It seems to me that my comments have helped you to diagnose problems in the code that were not obvious without those comments.

No! There were no problems in my code; there couldn't be, since I wrote no code. What I did was respond to someone else's problem sketch with a solution sketch. You responded not by pointing out omissions in my solution, but by inventing new problem details and situations. The fact that my example solution didn't cover them hardly matters; it was a solution to a /different/ problem.

What is the advantage of your scheme over comments?

A superb question.

Obviously, it can do everything comments can do except contain language syntax; so it's clear that for most purposes a good function name equals a comment. So why should we choose a function name _over_ a comment?

The first answer is that when more than one line is covered by a comment, those lines form a functional group and MUST be a function of their own. Once you have that, it's silly to not choose the new name to be as descriptive as the comment.

But there's more. If only a single line is covered by the comment, it's still beneficial to extract a function and delete the comment: by encapsulating the "how", you provide opportunity for later changes to the "what".

But the final BIG statement is this: comments can't be tested. Functions can, and a unit test will make it obvious when a function's name doesn't match its function.

However, explaining the "why" when it is not obvious from the "what" is exactly what comments are good at.

Please tell me SOME information in a comment which could not be conveyed in a function name. Fact is, you can't do it. Function names are exactly as expressive as comments.

There is some info that can't be conveyed by code; it's called "requirements". That info can't be conveyed by comments, though, either. It should be contained in use cases and acceptance tests, one acceptance test per use case. Acceptance tests should be automated, but that's a different story, and up to the customer, not the programmer.

-Billy

Re:Unexploded bombs, not sleeping dogs

[p3d0]

I didn't give that example -- look at the history.

I was referring to this code:

MakeNotAccidentallyClickable(widget);

That's what I meant by "example".

No! There were no problems in my code; there couldn't be, since I wrote no code. What I did was respond to someone else's problem sketch with a solution sketch. You responded not by pointing out omissions in my solution, but by inventing new problem details and situations.

Ok, perhaps I made a straw man. The point I was trying to make is that there must be more details to the situation, whatever they are; and because your code didn't specify them, the problems you found in my commented version of the code may or may not exist in yours. One can't know without more information.

Perhaps I can sum up my thoughts this way. Programmers make mistakes. It's hard to tell mistakes apart from deliberate decisions without some indication of the programmer's frame of mind. Thus, comments are only unnecessary in perfect code. If you choose to encode your comments using function names, rather than the features provided by the language specifically for the purpose, that's your choice; but it's not yet clear to me that it is a beneficial one.

A superb question.

And you have provided some superb answers that deserve some thought. I'm not convinced yet, but your position no longer seems ridiculous. :-)

Have you ever come across (or written) any material that discuss the pros and cons of your approach?

I do have a bit of a problem with the assertion that a block of comments should always become a separate function. There is a "sweet spot" when it comes to decomposing functions: with too few functions, they are too large to understand; with too many, algorithms can become scattered and fragmented, like a book with one word on each page. Splitting every function block into a new function seems to run a great risk of fragmenting the code in this way.

To me, function decomposition already serves three purposes: information hiding, code reuse, and divide-and-conquer problem solving. Adding a fourth criterion--of allowing comments to be encoded as function names--seems to raise the possibility that the criteria may conflict, and one may have to be sacrificed for another. Looking at the list, I know which is first on my chopping block.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

I was referring to this code:
MakeNotAccidentallyClickable(widget);
That's what I meant by "example".

Yes, I wrote that -- but it's a direct translation of the comment someone else told me couldn't be replaced with a function name.

Ok, perhaps I made a straw man. The point I was trying to make is that there must be more details to the situation, whatever they are; and because your code didn't specify them, the problems you found in my commented version of the code may or may not exist in yours. One can't know without more information.

You're absolutely correct. However, this situation MUST be left to the wisdom of the programmer. There's simply nobody else to trust! If the programmer's wrong, he'll write wrong comments as well as wrong function names, and most likely wrong code as well. (See "peer review" and "pair programming" for some solutions to this.)

Perhaps I can sum up my thoughts this way. Programmers make mistakes.

Just a quick interruption: so do maintainers. The programmers shouldn't assume the maintainers are going to make all the mistakes in the book, and neither should the maintainers assume that about the programmers. A lot of common programming practice seems to assume that a horde of idiots are going to take over the code tomorrow, so anything I don't finish today will get destroyed and perverted. This results in late hours, aggressive comments (like "DON'T TOUCH THIS!"), a lack of tests (because you don't think the later programmers will have the wisdom to improve you code), and so on. Some cynicism is appropriate; after all, the later programmers will have at best a foggy memory of what you were thinking when you wrote it; however, at the same time, it's just as likely that the later programmers will understand the overall problem BETTER than you do.

It's hard to tell mistakes apart from deliberate decisions without some indication of the programmer's frame of mind.

This is true, and it's why I want code to be self-documenting. The code has to wisely and correctly make comments unneeded; you can't just delete the information contained in a comment. The advantage is huge: you go from *two* sources if information which may contradict each other (don't tell me that you've never seen out-of-sync comments ;-) to one source which is automatically tested.

Thus, comments are only unnecessary in perfect code.

Not really true. There's no such thing as perfect code; comments are unneeded in code which tells the same story as the comments would. And such code (unlike perfect code) is not overly hard to construct.

If you choose to encode your comments using function names, rather than the features provided by the language specifically for the purpose, that's your choice;

Pardon, but function names are usually more of a language feature than comments; they're checked by the language, they're part of the syntax... Comments are just bolt-ons. And this IS the purpose of function names; but most people don't use them wisely.

but it's not yet clear to me that it is a beneficial one.

Well, I've given some reasons above. To me, the biggest two are as follows:

1. information can't get out of sync.
2. all information appears in two contexts: actual use (in the code), and prototypical use (in a unit test). Function names will look dramatically bad if they're chosen inappropriately. Comments can be ignored.

Comments can get out of sync with the code, in which case all of their "benefits" go to waste; and they can appear in only one place, and aren't checked there.

Have you ever come across (or written) any material that discuss the pros and cons of your approach?

It's a critical component of Extreme Programming, which is being discussed pro and con in a lot of places. I can also link to a few semi-balanced discussions (good paper, but biased toward the rather management-heavier SEI CMM model, which is needed for some projects). BTW, I was going to provide a con link, but the one I found is BAD -- hasty generalizations are just FLYING. Every single "danger" he cites is completely unjustified by experience and study, and many of the quotes he makes are out of context. It looks more like a Usenet post than a web article.

Oh, here's dmoz's directory page.

Anyhow, to implement what I'm describing you'll need at least Refactoring and Unit Testing from XP's arsenal. There's no reason you should have to adopt more, although once you have those in the others usually can be added at will. You also have to practice egoless programming -- don't be offended if someone changes the name of one of your functions; instead, make sure that the name is still appropriate to the use (if it's not, we have a bug being formed, since the person misunderstood the purpose of the function; you should show him a unit test in which the old name makes sense but the new one doesn't).

To me, function decomposition already serves three purposes: information hiding, code reuse, and divide-and-conquer problem solving. Adding a fourth criterion--of allowing comments to be encoded as function names--seems to raise the possibility that the criteria may conflict, and one may have to be sacrificed for another. Looking at the list, I know which is first on my chopping block.

Information hiding: true, but part of the purpose of IH is exposing only the correct information. That's part of the job of the function's name. So really, these two work together.

Code reuse: how can you reuse code if you don't know what it does? By choosing a good name, you are able to tell when things are being reused, and how appropriate they are to the situation. Note that the first time you extract a function, it'll have a very specific name (because you probably took it from a comment). The first time you reuse that function, you'll probably want to think of a little better name, and you'll probably also make some code changes -- the code worked in the one special case before, but now it has to work in two cases. As you keep reusing, your code keeps becoming easier to reuse; the name and the code keep becoming more general. So those two work together as well.

Divide-and-conquer: indeed, this is important -- but it's hard to argue that bad naming will help your strategy here.

-Billy

Re:Unexploded bombs, not sleeping dogs

[p3d0]

Alright, thanks for the links. I'll ponder that some more.

A few more small things...

Pardon, but function names are usually more of a language feature than comments; they're checked by the language, they're part of the syntax... Comments are just bolt-ons. And this IS the purpose of function names; but most people don't use them wisely.

A function's name is not checked against what it does, any more than a comment is. As for most people not using comments wisely, I think that relates to your point that one has to trust the programmer because there's nobody else to trust.

A dummy that lets comments get out of synch is just as likely to leave function names unchanged when their implementation changes. I'm not sure I buy your argument that poorly named functions stand out more than incorrect comments.

Code reuse: how can you reuse code if you don't know what it does?

I think now you're arguing my side. I have been advocating naming a function for what it does. You're the one arguing that it should be named for why it does it, remember? :-)

Divide-and-conquer: indeed, this is important -- but it's hard to argue that bad naming will help your strategy here.

I hope you don't actually think I'm advocating the use of bad names. I'm just advocating naming functions by "what" rather than "why".

Anyway, thanks for the discussion.

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Good points, again.

A function's name is not checked against what it does, any more than a comment is.

Well, yes; it is. A function's name appears in many places; if functions are supposed to have clear names, the lack of one will be very clear at a glance. A function which is being used for the wrong purpose should be obvious, and the best way to make it obvious is to make its name describe its purpose. Imagine seeing a function named 'SetFontAngle' in the middle of a vector rotation routine. You'd KNOW it was being used in a hackish way, and later font modifications could hammer it! How would you ever notice that if the function were instead named 'SetGraphicAngle' (i.e. it changed a cell named 'GraphicAngle' which happened to be shared by both vectors and fonts)?

Note that you're recommending naming functions for what they do; but the true story of what a function does is actually told by its implementation code. What a function does should be irrelevant to its use; for example, I may have a function which sets a specific bit in a specific I/O port, but that doesn't mean I name the function SetResetBitAndThenPollFor7854ChipReady. More likely, I name it for its intent: Reset7851Chip. Later on, when a reset bug is discovered, I implement the workaround without changing the name -- your function name would have to be changed to SetResetBitAndThenPollFor7854ChipReadyThenClearAll DMARegisters.

OTOH, suppose I'm writing some export unlocking code for the same chip, and step 5 of the unlocking procedure is to SetResetBitAndThenPollFor7854ChipReady. I would naturally notice your function name and would quite possibly use that function, because the name claims to tell what the function does -- but in your mind, that function is supposed to be a Reset7854Chip. What happens when the workaround has to be put in later? The only disciplined answer is that the person who originally wrote the function has to notice that it's being used elsewhere (hopefully), and rather than changing the old function, create a new function which calls the first and then zeroes the DMA registers. The new function would have the name we mentioned above, of course. Now what happens when a third programmer has to reset the chip? Which function does he choose? There's literally no way to know -- perhaps the comments in the calling functions will tell him, but that's a lot of source digging.

Code reuse: how can you reuse code if you don't know what it does?
I think now you're arguing my side. I have been advocating naming a function for what it does. You're the one arguing that it should be named for why it does it, remember? :-)

I think we're getting confused here between 'what', 'how', and 'why'. I'm using 'what' to mean 'what is intended'; you're using it, I suspect, to mean 'what is being done'. I would not use 'what' for that purpose; 'how' is traditionally used. However, I should not use 'what' for my purpose, since 'why' is less ambiguous.

So I want to name functions based on their 'why'; you want to put the why in a comment, and name functions based on their 'how'.

I hope you don't actually think I'm advocating the use of bad names.

I apologise abjectly. That was a terrible connotation on my part.

However, you were clearly saying that function naming was a constraint which might get in the way of writing good functions. If you're going to abandon my constraint because it hinders divide-and-conquer, then you should equally consider abandoning all other naming constraints.

I'm just advocating naming functions by "what" rather than "why".

I hope you see why I believe 'what' is the wrong thing for a function name to say -- at least if the 'what' is 'what this function does'.

-Billy

Re:Unexploded bombs, not sleeping dogs

[p3d0]

I think we're getting confused here between 'what', 'how', and 'why'. I'm using 'what' to mean 'what is intended'; you're using it, I suspect, to mean 'what is being done'.

I think this is indeed a source of confusion. When I say a function name should indicate "what" that function does, I don't mean just what it does now. I mean what its specification indicates that it is supposed to do. This, I think, covers your bit clearing example: if the functions's spec merely claims that it resets a thing, then it should just be called resetThing.

I had perceived that your claim was that a wrapper function called resetThingToIndicateSuccess should be added, for example, to convey not only the function's spec, but also the programmer's intent. My position was that the "to indicate success" part, being indicative of the programmer's motivation for calling the function, should be in a comment rather than in the function's name.

Perhaps we are both mistaken about each other's position? Anyway, any misunderstanding aside, I think we probably still disagree on how to name functions. :-)

Re:Unexploded bombs, not sleeping dogs

[William Tanksley]

Perhaps we are both mistaken about each other's position? Anyway, any misunderstanding aside, I think we probably still disagree on how to name functions. :-)

I think we're on the same page, actually. If you'd name a function resetThing then you're naming it according to its intent, not according to its contents.

However, you're also missing a point -- I'm not resetting the chip to indicate success; I'm simply setting the bit which usually means 'reset' and polling the device ready (which happens to coincide with the intended way of resetting the chip). The chip doesn't actually reset; it simply finishes unlocking its cryptographic functionality. So I don't call resetChip, because my purpose isn't to reset the chip, even though I think the procedure is the same. I would be wiser to call a function propagateUnlocking (or some such name). Therefore, when the workaround I mentioned is added to resetChip(), it breaks no code -- because everyone knows that a function named resetChip will be used only when a true full reset is needed.

My point is, as you discuss, that the programmer's intent in calling a function should be the main thing conveyed by the function's name. A function should never be called if its name fails to match your intent; if you examine the implementation, and it matches your needs exactly, you should at least change the function's name so that it fits into your intent as well as into the intent of its previous use. That way, when it comes time to modify the old function, the maintainer will know unmistakably that there's someone using the function in a way which was not originally anticipated.

Of course, if you choose to do that, you MUST also create some new unit tests for the renamed function which illustrate its new use. That way the maintainer will not only know _that_ there is a new use and intent for the function; he'll know also exactly what that new use it.

-Billy

Worse is Better?

[qbalus]

Here is an interesting link that leads to others links

http://www.dreamsongs.com/WorseIsBetter.html

If the project is in Java...

[Curt Cox]

I don't know how well this translates into Perl, but if the project is in Java, the steps are pretty straightforward.

1. Determine a clean architecture for how you would have solved the problem, given the benefit of hindsight.
2. Create new interfaces based on your new architecture.
3. Code facades to make your old code implement the new design.
4. One by one, re-implement your new interfaces until the new implementation has solved all of the problems that led you to re-implement in the fist place.

Of course, during development and maintenance, you should be doing all of this stuff everyday, anyway. Reimplementing everything just means the team has agreed on a global change in design, as opposed to the local design changes you don't need any co-operation to achieve.
If you follow this approach, you always can always have working code at the end of the day.

or...

[Plisken]

"The root of all evil is premature optimization" - Knuth

Similar Situation...

[Unreal One]

I've written and maintain a similarly large web application for my company, that was all done in Active Server Pages and SQL Server.

All reusable code, and authentication verification in my web app is in a single 'functions.asp' file in the root of my project. Every single page, page in the project has an include for this file which takes care of the authentication, and provides that ASP code to the rest of the app.

Also in the root, I keep a javascript-functions.asp which works the same way but is included only in pages that have html forms. This file contains common client side functions to validate forms and what-not...

On the same note, I usually use a single CSS file that is also called universally throughout the app to simplify maintenance of the look of the app.

On the database side I'd recommend using stored procedures as absolutely much as possible. Obviously try to make all your transactions atomic and reusable in stored procedures to optimize performance and simplicity.

That's my 2 cents...

buy a formatter

[GutterBunny]

We're going to have to make some hard decisions about code style.

The best thing to do about code style is find some method or function code formatter (built into your ide) that most everyone can agree on. Basically, it should be very easy to format your code after writing it. Then code formatting is a non-issue. Everyone lives (and dies?) by that formatting.

Re:buy a formatter

[Doctor Memory]

We used to use indent(1) as part of the front-end to our SCMS. Whenever you checked in a module, it was automatically reformatted to the "official" standard. When you checked something out, you had the option of reformatting it to your personal style, or just leaving it as-is. I modified the SCMS checkout command to search for a .indent.pro file in your home directory and automatically invoke indent(1) if it found one. This by and large made everyone (except some of the old guys) pretty happy.

Small steps

[ChaosDiscordSimple]

Large overhauls are usually mistakes. Details in the previous code are lost. If the overhaul takes non-trivial time, people become frustrated that two weeks ago they had a working (if problematic) system and today most of the system doesn't work.

Instead, make small incremental changes. Pick something lots of code is replicating and attempt to unify it into a shared code base. Spend some time documenting key parts of the code. Pick a particularlly hairy class or function and untangle some of the worst bits. These sorts of changes can reveal minor bugs, build up to significant improvements, and leave you satisfied at the end of the day that you improved things.

If a signficant overhaul is necessary, try to overhaul portions while maintaining the existing bits.

Retrofit Mercilessly But Quickly

[Mentifex]

When making a comprehensive retrofit of your AI Mind for Robots Code, you have to sacrifice the early instantiations for the sake of smooth sailing on The Road Ahead to the Technological Singularity.

A major Open Source AI initiative has no choice but to r00t out and correct mistakes ASAP as soon as they are found. History will judge us be how kind we were to our robot and cyborg mind-brethren, not holding them back but equipping them to meet the world. Therefore Mind.Forth AI is for the robots themselves, while the JavaScript AI (q.v.) caters to human users who need tutorial instruction. Welcome aboard, all AI coders of good will towards bots.

Re:Retrofit Mercilessly But Quickly

[Anonymous Coward]

just out of curiosity ...how bout combining your efforts with the dude at tunes.org ?? they seem to be on roughly the same track.
of course theres no reason to think a technological singularity will ever come about or that turing machines are capable of self sustaining sentience but whatever..

Compartmentalize & Destroy

[DotComVictim]

1) Identify common functionality.

2) Encapsulate in libraries

3) Be sure to extract enough generality that you don't have special case functions

4) Don't extract so much generality that functional interfaces become unwieldy.

5) Write everything in the same language.

6) Find any complex pieces or algorithms. If they can be simplified or re-written, do it. If not, save it so you don't need to debug it again.

7) Throw everything else away.

Joel is Right

[qbalus]

Check out this site. Lots of great advice

http://www.joelonsoftware.com/navLinks/fog000000 02 47.html

Regression test suit

[angel'o'sphere]

The first thing you definitly have to do is Setting up a test suit for regressoin testing.

For those not familair with the term 'regression test':

Program a set of so called "test drivers", programms calling your code: routines/scripts.

Define test data, either in a DB or in flat files, used by those driver programs.

The test programs and test data needs to work with the old code, of course.

As the new code should behave similar you only need to adjust PATH or script names to let the test programs work with the new code.

Plan your project by defining which test cases(test program plus test data) should work at a planned milestone with the changed code.

After making changes rerun all tests.

Well, there is a lot more you could do, but that above is minimum (basic software engineering, sorry no art involved here).

Regards,
angel'o'sphere

The Blind Leading the Blind

[Junks Jerzey]

"Do such and such because I read in an interview that..."

"There's this cool buzzword called refactoring that my professor mentioned and so you should..."

"[blah blah blah] object orientation [blah blah blah]"

"I saw a book about Extreme Programming at the bookstore and..."

Perl: another close-minded fundamentalist religion

[Anonymous Coward]

The short sighted Perl Fundamentalist fools have invested so much time and energy in Perl, that they're furious to hear how horrible it is, and how good the alternatives are, because they know it in their hearts and spend a lot of energy denying it to themselves.

This is your bible

[Anonymous Coward]

Read this over Christmas. It is worth its price ($45).
http://www.amazon.com/exec/obidos/ASIN/020148567 2/ qid=1008975687/sr=8-1/ref=sr_8_19_1/102-3322096-38 46523

Design

[Anonymous Coward]

If you really must rework the entire system, a solid design will go a long way towards keeping things consistent. Consider implementing something like the Rational Unified Process to make sure that your new system will meet the needs of your current buisness model while providing a clear path to where you want to be. Unfortunately Rational products come at a heafty price, but the ideology is free.

Little by little

[Mike Schiraldi]

Don't stop development on the old tree and shift all work to the restructuring project.

Instead, leave most of the manpower working on the old tree as always. Take a small team of your best people and have them gather input from the others, while the others keep working on the old tree. Then have the small team outline the changes that need to be made.

Work on the changes while simultaneously working on the usual stuff. Say, 90% of your manpower should do what they do now, and 10% should work on restructuring things. One mouthful at a time.

When you have a mouthful that you think is ready, branch the old tree. Merge your diffs into the branch, TEST IT, and if things seem to work, land the change onto the old tree.

Re:Little by little

[William Tanksley]

Wise, but may I suggest the obvious extension?

Don't play with code trees. Use only one "tree": the current, live one. Don't let one code base rot unmaintained while the other one is hacked to bits, untested.

Do take one mouthful at a time. Add unit tests, make sure they work. Now add a unit test for something you WANT to work, but which doesn't. Make that unit test work by implementing the feature. Release the result. Repeat.

-Billy

It is called Refactoring.

[burtonator]

Most of the computer industry now calls this Refactoring.

I would HIGHLY recommend the book "Refactoring" by Martin Fowler.

There is a number of things you should do here.

1. Document your plan and come up with an official PROPOSAL document. Allow others to comment on this document and incorporate fix all relevant issues.

I started using this under the Apache Jetspeed project and now a lot of other Apache projects are accepting this practice.

It really allows the community to become involved in your changes and encourages constructive feedback and involvement.

2. Break this into phases. You should NOT attempt to do this all at once. Each phase should be isolated and should consist of one unit of work.

Each phase should be branched off of CVS, worked on, stabilized, brought back into HEAD and tagged. You should then RUN this code in a semi-deployment role for a period of time to correct all issues which WILL arise with the updated code.

After this you can then start your next phase.

3. UNIT TESTS! If management (assuming you have management) has approved the time for this type of refactor then you need to take the time and write Unit Tests for each major component.

It is important that Unit Testing can sometimes be just as hard, if not harder, than the actual development itself.

In some situations you can avoid Unit Testing, some here are going to call me crazy for saying this but it is true. In a lot of high level applications, which are NOT used as libraries by other applications, you can bypass Unit Testing in order to increase development time. This is a dangerous practice but it is often outweighed by the extra functionality you will end up with in your product.

Anyway. Good luck!

Kevin

Re:It is called Refactoring.

[Coulson]

Just to emphasize the point burton made, the importance of unit testing cannot be overstated!

Consider setting up an automated testing process which runs your test packages every hour or two minutes.

When you write code, write tests. When you check in code, check in tests. It's truly liberating! Once your code is in and tested, it's not your fault if it breaks! (Well, most of the time. :>). It's the fault of whoever checked in after you without running the tests.

Comphrehensive tests let you know the exact state of your product at any time, and when you want to test on a new machine configuration, you can just run the test suite to get immediate feedback on what works and what's broken.

Refactoring and designing new architectures is one of the best parts of the job, so good luck and enjoy it!

Re:It is called Refactoring.

[Anonymous Coward]

Writing unit tests first (extreme programming) actually reduces development time. So you may not "need" them, but you'll go faster with them. At least that's been true in my experience.

Re:It is called Refactoring.

[gidds]

Break this into phases. You should NOT attempt to do this all at once. Each phase should be isolated and should consist of one unit of work.

Amen!!! The more you try to do at once, the less likely it is to ever see the light of day.

You must break this into small, manageable chunks, so that after each one you still have a working, maintainable, testable system. (If this means you have some duplicated functionality during the process, so be it.) If you have regular deliverables it keeps you focussed, reins in scope creep, demonstrates progress, and of course lets you Release Early And Release Often!

Re:It is called Refactoring.

[Xthlc]

> In some situations you can avoid Unit Testing,
> some here are going to call me crazy for saying
> this but it is true. In a lot of high level
> applications, which are NOT used as libraries
> by other applications, you can bypass Unit
> Testing in order to increase development time.
> This is a dangerous practice but it is often
> outweighed by the extra functionality you will
> end up with in your product.

Amen.

While unit testing is important for key libraries, it's not a panacea. Inexperienced programmers can parrot the maxims of XP all they like, however the fact is a unit testing policy can create a terrible drag on application development if poorly implemented.

A sensible, real-world policy DOES NOT operate on the scale that %90 of the XP acolytes out there recommend. I'm sick to death of smug self-styled gurus (who, mysteriously, don't seem to do much programming since they started writing articles for Javaworld) that admonish us sloppy developers for our unwillingness to sacrifice key features in order to make time to build unit tests for every radio button. One of the most important components of unit testing is understanding the difference between a unit test and a waste of time.

Oh, and don't bother with articles or books that discuss unit testing, unless you interpret them in the most general terms. I've never seen an example of a unit test that wasn't unbelievably trite. Unit tests for an imaginary number storage class? Please. What about when the code unit you need to test manages asynchronous input from five different sources, where that input consists of five different complex packages of information that must be unraveled, interpreted, and transduced based on the code unit's multi-tiered internal state?

Unit testing is an art, not a science -- so don't sacrifice your project and your customer on the altar of XP. Explore unit testing, but don't embark on a comprehensive policy until you understand what you're getting into. A half-assed unit-testing policy is much worse than none at all.

Re:It is called Refactoring.

[KyleCordes]

I found a couple of things strange in this post:

[unwillingness to sacrifice key features in order to make time to build unit tests for every radio button]

Personally, I do test-intensive coding because it helps me go faster (build more functionality per unit time). If it didn't, I wouldn't. I also don't see the point in unit testing a radio button.

Re:It is called Refactoring.

[KyleCordes]

Oops, posted early. Other strange things:

[asynchronous input from five different sources, where that input consists of five different complex packages]

I've found that it makes a lot more sense to focus on testing the hard stuff, like this, than simple things like the ones you mentioned. By the way, it's also quite useful (totally apart from XP) to decompose such a beast into pieces you can test effectively.

[so don't sacrifice your project and your customer on the altar of XP]

I've never heard of this happening. Can you point to any project which did XP (most or all of it, not just talked about it) and resulted in an unhappy customer for that reason?

I've mostly seen people do little bits of XP (ignoring most of it), fail, and blame XP. A typical story is "we did XP, but we didn't do much testing, we didn't pair program, we didn't have a customer on site, we didn't integrate very often, we didn't track our progress, and our project went badly, therefore XP is bad". Weird.

Re:It is called Refactoring.

[Xthlc]

Re-reading my post, I guess it came across that I think XP, per se, is bad.

On the contrary, I feel that many of the assertions XP makes are quite good software engineering practices. However, I also feel that many professed adherents of XP become fascinated with the methodology and forget that, in the end, software engineering is about building a product that makes the customer (not the fake customer on your XP team, the real customer holding the checkbook) happy. This means that XP methodologies must be applied with careful judgement and an eye towards flexibility -- the most elegantly organized project in the world will still net you zero income if it doesn't get out the door on time with all of the required features.

That said . . .

> I've found that it makes a lot more sense to
> focus on testing the hard stuff, like this,
> than simple things like the ones you mentioned.

Exactly my point.
However, I think very few proponents of XP actually understand the principle outlined in this article, namely that XP is a set of best practices and NOT a color-by-numbers guide to successful software engineering. There's a great deal of XP fundamentalism out there, mostly on the part of development managers who are excited to the point of bursting by the childishly simple ABCs of XP according to [insert snide columnist here].

> By the way, it's also quite useful (totally
> apart from XP) to decompose such a beast into
> pieces you can test effectively.

Unfortunately, the actual code that inspired my description is about as atomic as you can get in an asynchronous system. A few inputs, a few outputs, and a very well-defined domain of functionality. However, both internal state and external state, as well as the timing of incoming messages, are quite complex, and non-trivial to simulate as part of a unit test.

My point being: unit testing is Hard in many situations, and most XP literature I've read glosses over that fact and downplays the necessity of picking the right things to unit test.

> Can you point to any project which did XP (most
> or all of it, not just talked about it) and
> resulted in an unhappy customer for that reason?

Well, I can't point to any project that managed to successfully implement "pure" XP, the kind that I've only seen in a training center. Most of the ones I know of that tried it gave up on it pretty quickly, usually because either:
1) They didn't have enough staffing to constantly pair program AND get all the necessary features completed
2) Their time frame wasn't sufficiently flexible to allow for the additional time incurred by XP-style unit testing (as opposed to a more traditional test suite).
3) Their Real Customer (not the XP customer) was not a nice and reasonable person with an understanding of how software development ought to work. He kept demanding additional features, some of which were negotiated away; the remainder necessitated that process be streamlined in order to deliver them on time.

While only a couple of these projects failed, they all wasted effort and money on implementing a methodology of which they had little understanding, but were led to believe was Just That Simple.

Nobody can sensibly argue that the generalities espoused by XP are bad. Testing is good. Iterative development is good. Refactoring is good. Pair programming *can* be good, in the right situations. However, the manner in which XP's principles are presented and disseminated (i.e. "the cult of XP") is, for the most part, misleading. Like most cults, The Cult of XP claims to have the one right and true solution to every problem in existence, which can be yours if you just do A, B, and C. Like many cults, The Cult of XP has some good things to say; in fact, it DOES benefit you to join up if your problem fits certain constraints. Also, unfortunately, like many cults, the Cult of XP is more interested in casting aspersions on its detractors (who are "afraid of change" or "unenlightened") than it is in engaging in serious examination of situations in which its dogma doesn't exactly hold true.

Re:It is called Refactoring.

[KyleCordes]

[not the fake customer on your XP team, the real customer holding the checkbook]

XP says to have the real customer. If you have a fake customer, it's not "pure" XP. Many people who like XP but can't get a real customer use a "proxy customer", but they (hopefully) admit that they aren't really doing XP per se, just something similar to it.

You are of course 100% right that satisfying a fake customer is irrelevant if you don't satisfy the person with the checkbook! In my mind, customer equals check-writer.

[There's a great deal of XP fundamentalism out there, mostly on the part of development managers]

That actually doesn't match my experience at all. I see many developers who are very excited about XP, but many development managers seem to like some of the ideas but be *far* from "fundamentalists" about it. Rather they want to immediately disregard major portions of it, typically.

[about as atomic as you can get in an asynchronous system]

I've built a number of asyncronous systems, and agree they can be challenging. I do believe it makes testing even more important. If I was building the system you describe, I would probably insist on testing that piece.

Regarding choosings unit tests, the frequent advice is to test everything that could possibly break, which is pretty vague - but it definately includes this async beast.

[Well, I can't point to any project that managed to successfully implement "pure" XP]

[didn't have enough staffing to constantly pair program] etc.

My last project didn't do RUP. And it didn't do Scrum, although we no doubt did bits and pieces of each. I don't use it as an example of either of those methods succeeding or failing.

Your projects didn't do XP, it did bits and pieces of it, omitting large important portions. So why would it indicate anything about XP? Because the people talked about XP a lot while they did something different?

[The Cult of XP claims to have the one right and true solution]

I've read thousands of messages on the XP mailing list, I've been to an XP conference, I've read a few of the books, and I've seen presentations by most of "big names". In the process, I have not seen such claims.

Who is making this claim?

Complete rewrite necessary or a waste of resources

[beamz]

Not too long ago a link was posted to an interview with Joel Spolsky who used to work at Microsoft.

His comments about code reworking and rewriting have a lot of insight in them.

Here are some quotes from his article:

SMS: Joel, what, in your opinion, is the single greatest development sin a software company can commit?

Joel: Deciding to completely rewrite your product from scratch, on the theory that all your code is messy and bug prone and is bloated and needs to be completely rethought and rebuild from ground zero.

SMS: Uh, what's wrong with that?

Joel: Because it's almost never true. It's not like code rusts if it's not used. The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. There's nothing wrong with it.

The point is this... The benefits of spending time rewriting code completely may be a waste of the companies resources, this is for you to determine.

His interview is here:
http://www.softwaremarketsolution.com/index_2.ht m

and his site has more information about the concepts here:
http://joel.editthispage.com/articles/fog0000000 06 9.html

Fire the developers if all they know is Perl

[Anonymous Coward]

Morons like that who only know Perl and think it doesn't suck were responsible for the Internet bubble bursting. They don't deserve to have jobs. Fire them, and hire some real programmers.

modularity/incremental rollout/unit tests/iterate

[tim_maroney]

I second all that has been said about making sure that you really need to do this and that it is worth the time and risk. One sign that you may need to do so is an excessive reopened bug rate, where fixing one bug often creates another bug due to side effects and component interactions. If you decide that it is, then the three keys to success will be modularity, incremental rollout, and unit tests.

Modularity is probably what you're already thinking about. Go over the old code base, in a code review, and find where the same thing is done over and over either with copy-and-paste code -- the bane of crap engineers -- or with different code that serves the same ends. Look for repeated sequences in particular. Create a new library that encapsulates those pieces of code.

Incremental rollout is vital. Only replace small parts of your system at a time, doing complete retests frequently. Don't write a new encapsulated routine and then roll it out to each of the three dozen places in which it appears in the whole code base. Write the whole function library, with unit tests, and then start applying it to separable modules one by one, retesting as you go. Otherwise I guarantee the whole thing will fall apart and you won't be able to tell why. Ideally, you might set a threshold on the rate of replacement of old modules and work primarily on creating new modules with the abstracted logic.

Unit tests are crucial because, as noted, the messiness of your old code probably conceals a lot of necessary logic. We had this great phenomenon on Apple's Copland where people who had never used the old OS managers were rewriting them in C or C++ from the assembly source. When they saw something in the assembly they didn't understand, they just ignored it. Guess what -- the new managers didn't have any backwards compatibility. The only answer to this is to have a thorough unit test for any module that you replace, against which you can test the new version. This also confers other quality benefits, but during a rewrite it's critical.

Finally, once you have replaced a significant number of your modules, you will find that new levels of abstraction appear. The average size of each function or method will have shrunk considerably, and now it becomes possible to see new repeated code sequences that were not visible due to the old cruft. Move these into your new library modules and start using them in continuing replacement work. In addition, start going back -- slowly and incrementally -- through the already converted modules and replacing the repeated sequences with calls to the new abstractions.

Finally, figure out how you got into this mess in the first place. The worst programmer habit I know of is copy-and-paste coding instead of using subroutines. You can tell people not to do it, but some always will. Those people should be bid farewell -- you can't afford their overhead. Other common problems include lack of planning and review, a code first and think later mentality. Start moving your organization up the levels of the CMM and you may find that you wind up with fewer modules that need replacement.

Hope this helps.

Tim

I'd like some info on this as well...

[josepha48]

in particular how do you write a large program? I work on a program that has over 3000 files that are over 30 to 300 pages each. The software does alot for and has been around for over 30 years and has just had code sloppily added to it. So how does one go about rewriting an application that noone knows all the functionality of?

eXtreme Programming

[Anonymous Coward]

1. Look back and wish so badly that you'd written Unit Tests for all that code in the first place!

2. Wonder whether you really need to. Check out this slashdot article from a microsoft guy on why their competitors died when they attempted rewrites instead of making do with what they had.

Barf!

[Anonymous Coward]

I can't believe you put 'code style' and 'perl' in the same paragraph.

That's the beauty of Perl... you don't need coding standards. Easy to write, hard to read no matter what you do.

Go slowly young Jedi

[0xA]

This is a redundant post but I think it's imortant enough to be repeated.

Don't set out to refactor your codebase as one big project. Try and split up the code by functional areas and take them on one at a time. Now, this doesn't really work too well most of the time. You're going to run into way too many places where things are interdependant but try anyways.

If you can, resist changes to your database schema while you are refactoring code. Having both of these thing happen at the same time is pretty scary.

You say CGI so I asume you are using PERL, look into OO PERL, it's worth it. Even if you don't want to go OO all the way the 2 big things that can make your life easier are packages and layering.

Using packages, especially for things like DB access can save you tons of time and headaches. You have one place where you run a query and build a hash, all of your code calls it when it needs the data. HUGE adavantages here.

Layering your design is helpful ass well. I've found that you can do a lot of good if you have designed Data Access, Logic and Presentation layers seperately. All each one of these layers needs to do is take the hash ref passed by the other layer and do X with it. You can rebuild each layer at will as long as the data structures passed between them don't change.

Solicited suggestion

[ciurana]

chizor wrote:

My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code)... What suggestions might readers have about tackling a large-scale retrofit?

My advise to successfully accomplish the changes:

• Where possible, use Python or Java instead PERL
• If there is a line-of-business piece of code (i.e. if your company's bread is on the line when this fails) make sure you can roll the original code back if the new version doesn't perform
• Divide the programming team in groups of 2-4 people
• Use XP techniques for development
• Have an external group evaluate progress to keep people honest; this group is responsible for testing that functionality meets or exceeds that of the original system

I led the development and migration of some very large mission-critical systems in my career. Too many programmers making decisions on-the-fly, totally centralized management, or a "leave the technical folks alone until they're done" attitude are sure recipies for disaster.

Good luck with the changes.

Merry Christmas, and God bless us everyone!

E

Re:Solicited suggestion

[m_ilya]

Where possible, use Python or Java instead PERL.

Indead. I'd like to extend this advice:

• Where possible, use Perl instead of PERL.

DON'T DO IT!

[mcrbids]

As posted elsewhere, re-writing is painful, and generally NOT A GOOD IDEA!

I currently maintain a code base of around 120,000 lines of php and html (written by myself in a long, hard year) and have had to "retrofit" it a few times.

I find that when it's time to do an "over-haul" it's generally best to:

1) Pretend I know nothing - redesign from scratch. Write out a spec with flow charts, DB table definitions, etc. - make it VERY DETAILED. Spend lots of time at it. More time spent here saves even more time later.

2) Ignore your spec. (See step three)

3) When a bug comes up, or new functionality needs to be added to the codebase, refer to the spec built in 1, and build to it, and then put in compatability wrappers to work with the existing codebase.

Make these compatability wrappers log their calling in some way, based on a global variable. This allows you to see when they're no longer needed simply by defining a variable in a config file and waiting a while.

4) You'll be slowly bringing the application up to the new spec - eventually you'll reach a point where it's easier just to bring the remaining pieces up to snuff than to build more abstraction wrappers. When you get to that point, you'll find most of the work is already done, just finish it to the spec and remove the compatability wrappers.

This can still be a painful process, but at least it isn't a "gun to your head"! This allows you to regression test your work as it's done, resulting in a more stable deliverable, and you can still meet clients' needs in the meantime without making them wait 6 months while you re-write all your stuff.

Hope this helps...

Re:DON'T DO IT!

[esconsult1]

I know what you mean!

I've also been working on approx 7000+ lines of PHP code over the past several months http://makunu.com

After 3 months coming back to your code, even though it's well commented you start to wonder... "WTF was I doing!?". But guess what, it works! And it usually works well.

I've found with this project, its better to just go in and modify the code. Another thing... do it in with lots of little subs/functions that serve as black boxes. Because of this approach, when the time came for us to switch from Mysql to the more robust Postgresql, then we didn't have to change any code.

Another thing we did was to make the system to be like a "plug-in" type of architecture, so adding or getting rid of modules was childs play without having to rip out the heart of the application.

One drawback to our large library of functions, is that it took a fairly long time to load with each web request. This library contains massive quantities of small functions that do everthing for logging errors to database independent access. But was there a way around this besides senselessly duplicating code... I dont think so.

With such a large code base, I really have no plans to rewrite it any time soon. This system will end up being massively patched over the next few years.

Re:DON'T DO IT!

[Anonymous Coward]

120,000 lines in 1 year. that's a lot. thats one line every 4 minutes, 24 hours a day 7 days a week for 365 straight days.

I hope you cut n' pasted a lot.

Re:DON'T DO IT!

[cpparm]

One drawback to our large library of functions, is that it took a fairly long time to load with each web request. This library contains massive quantities of small functions that do everthing for logging errors to database independent access. But was there a way around this besides senselessly duplicating code... I dont think so.


How do you know that's the reason it's slow? Function calls are cheap, performance wise. If you profile your code and look at what is slowing it down, you might be surprised.

Re:DON'T DO IT!

[Anonymous Coward]

maybe if you read it, you would see that it took a long time for the library to load, not to call a function.

Step #1

[Sanity]

...don't use Perl ;-)

[-1 Flamebait]

Re:Step #1

[cscx]

I hear that C# and VB.NET can scale your code down a LOT ;-)...

sixth hex post

[Anonymous Coward]

a8a0a6d7a4f1c6e4c2c9a2e7f2

e8b2c3b6f4f2c4d5a1e8d8b2f4

e7d4e0a2e7a2e0f3d7d4c6e1e7

f8b8d2a5a2c6f2d9e2c2b6b7f9

c7e4a8e4c2a0e8d1d3e5e9b0e3

f3a4c4f8a0b2d2b3e3d5b3e9e2

a8c0e2f8b2c8e3d5b8c8e7a7e2

d3a7a7e6a9f3d9d5f1f0a8f0b3

a2a9c8e5b4c3e2d6b7e2d4f0d6

d1a7b0a6c9f2a6b4f4a7a7b1f5

e6d7e8d5a5f6a6d5b2f7c1e7f7

e7e4e7f6f6e5d2e2a1d3d6e8d1

a2c6a9f3f1c5d4c5f7d8c7d6a1

c0d1e9a7b2e1a0d3c4e3c1d1f2

b2c5d8a5a3f5d2e8f3f1d1a0c1

f3b0f4c6b5a1d2e0e3c8e2e9a8

d9f2a7c9a2f6d1d1b8e8b3e7a8

c8b1a5a6d7b1f7f8d4c6f8b4a3

b0b9a5b1d8e2e4d5f7c7d6e1b2

e0d2c7f6f7e7e2b1a1c7f5f3f6

d8f4e7c1c4f5e6a4e1a6d8c2f6

f4b3b8c5a2d3b3b7c8e2a2a2c4

b1e2a5a9b9f9c6e9c4d8a1a3a2

f8e4d7a4a4f1c1a4f4a2c1c6c9

b2a6f1f2a5b3d5d3a2d4d5f1b3

c4c4b5d0e8d3a0a5a7c3e4b4f5

d5e8a7b7d5c6f3b3f4f2b5b3c2

b2f1f8f2b6e3c3c8a1c3e3c5a3

c9f2f3f2f5e7c6b1f3a1b6d3c2

b4a9e4f2f8e7e2e7a7a6c3e8f1

b7e1d8b8f2f5f8d1f8f7d6f9f5

e3f0a8c1c3a2d4a1c7f4c2e5c1

e5d6c7b3d2d1c0f7d9d7d4d5d8

b8c4e1e6b4b4d9b0b8d8c7b2b1

d4b1a4c7a8e2f8d9e4f7e1c6d3

c8d4d8a5d6d9b2f8d9e1f8c1b2

e3d4a2d1c5b8a6a1b2c1e6b7a4

c5b4a8b8b5b6c3c8b8d4d0d5c4

a6e0f7d5a4c1c0f7f3e1f7d8f7

d1a2b9d5b7a4b8b4c3a7c8d1b3

f7d2a4f3d8e8e4a8b5f2f6a2e8

b6c8c7a7b1b5e4d4c2d2e8e2a3

d7a4a6c7b4f0c8b7d5f1a8d0b3

d7e7a5e7c2f5a1e8b6f7b4c2d6

a5a6f1b7d4c3c6e7d2f1d9f4a1

f5a8f8c4a2a1d0f7c8f3c3f3c5

e3b0a8f4a9f6d7d8b4c0f4f5e2

f4c6c9e1d2d6a2e9b5c8d5b3d2

e7d5f3b1c4c2e2f2e3d1b5d5a4

b1a6a5a4c5e5b2f9e5b4a7e0a3

b2e4c5c6a0d5b6f7e5f3c8f9a2

a6e7d8a8c4c4a8c3f9c7d2a2c9

a3c3f0f5e3c0d2c2e7b4f5a8a3

d8b4d8c0f4c0d3b9c7b7c6b8b4

a3c3f4c3c2a1e7e5c4f4c7f6d2

a2f4e4f5d8e5b4d4f1f9d4e6d5

e1a4b4d7f1b7d6e2b2a2d5f7a3

e1e4c2c7c9c8c3e2a3b4c7a8b4

a7d7b8d0f6b2b0e3c4f9e3a3c0

d6d4f9d7b3c9d3e8c1c4a0a9e8

f4d6a6a6a9e1f8b7b2a5b2b2b0

e5d2e2f6d4c8c1e7c8d5f7b4f8

a5f4e2e5b4a3a7e4a7d8e8d5a2

a5c0f3d1e1a3f2e6d2e8e8f2d5

f8a0f9c8e6f8d8e3f5e4d3a4c6

c1e8b7c8c6e7c1d1d1b0d2f2b5

f3e2e6b4b5e1e5a6b4c1b4a8c7

a6f7e2f9b5b8a4f7c6a3f6b2e8

b0e6f3b6d4f1d2c2d8a9d2c8b5

e1c7b2a4a4d5c1c0c4b0f5b5d2

e7b5e8a3f3b6a2a2f2f5a4f2a1

e8d1e2b1f3a6b0a1a2a4b0d4e8

e1e7b2c3f3d5c7f4c8f3e0c6a2

d7a6e9c5c1f1b4a3a4e1d4d7a7

c4f7f6f3b2c6b3c6a6d5b3c7f2

f6f1f2c3e6e8a8a7c3b5a3b2c6

c5c4e1f1d1b6f6a6b6d1a9c3b9

e1c8c1a3e0b2b9a7b8b4f4f5b8

f1c1c9c7f5a9c8d2a5e1c3a7b8

e5c5f4f6a7c4c2e6f3f8a2a4b1

a7d1a4a8b3a8b3e8b0c4d2b4a3

d3f1b5a7a7a5f3d1a9f7d6d6c5

a8a0d4d9a2f3a7a4e6d4a5c3f9

a3c4d4e7a5d4b0c5d9d1f5e5a6

c6f5e1a1e2a3e8f6b5e4c4d3a4

b4a3d6c4d9a4b3f3d3e2a8d5d1

b1a8e3e9d8d4d3d1a1c1e3f6f3

d9b9a4e5d6f5f8f1b9f6d0e6d8

e1a9d1e4e1c6a5e5a4f3a8d5e2

b7d7c5d1b4a1a5e3b5f5a6e3f5

f2e8d8d5d1c7c5d2d0a1f4a4d3

e3e8e9c7c7d8b4a1b7b8d6a7b5

f8c5d9a1a9b4b3b8f7b4f5e6d8

b2d4c7c2b6a5c5c1f3e5a1e5a9

d8c4c5d6b2f6f6d3a8d1b0a1e1

f2d5e0a0a6a2a2c5b8c6d7d7a7

f7b7c8a3d2d8f5f5f7e2d1a9f0

e7c4c1b1b6b1a8a5d1c4f1b1c7

b2c3e4c3e4b6d4b8a1c9d1c3c8

d3f2e3b7f0c0b6c7d7b2e1f8b9

c3d2b9d4a5c4b4b6f6a7c3f6c8

a5d3c5e3d0f1c2b1d2e4c6e8e8

d3f5f8a1e6f2c7a8a4a4e7b5b1

f6e8d4b8c8a7f8c2e2d8d6e7a9

a3c1e8a2b7a6c1c4e6d6a7f3a1

c1e9c7b5e9e4a4c4b8a4d7f3a5

c6c4f7b5f3f9b1e5b7a7d4a8e3

d9b5b3b9f2d5d4b5c2c9f3f2b2

c4c1f9e2c3e6c8a0c1c6f3e8b9

e6a2e7f2c1a4d6d9c4d8b1b4d6

c5d7e3d2c7e0c7e1b7d0c2c8e6

b4e2d7e1c8f7b2b2e3f1f8b4f6

b4f5a4b7f7c6b6f8a8f7f1a4d8

f5a5e3c3a0d6e6c4f8f2a1b1d1

f3f2f0a3d3b8f3e3c4b5b1f2e8

e6f1a2f5b7a7c8a9c6c5b5b9e3

d9b1c4e4e1e4d6d3b2e9e4d2e5

b7d7f8a9b5d1a6c1d6a7b0a5b4

a6c0e6d4d1d5e3d3b7c5d0e0f8

d6b4f7b6f0a3a3c3f4d1f8e2f8

a8d4e4a6e9c5d5d5f4d8f1d8e5

e1e3c8f9d4c1c8a6d8b5c4a5c7

e3b0f2b1f2e9d2f2b5f8d2a4c0

b5f8e5f6d6f5d2a9e9d1c9d3b0

a5b2a9c4e4f3d1f6e2e4b6c1e6

c4c4d7f7a2a5e8f5e0c9e1c2f3

a3c3a7a4d2a8e6e1f1a1b1e9c0

a2a6a5a7f7e1c3d0d3d2b5a5b7

a3f7f6d3a1e4e9e1d8d8f2f2d2

d6a7e0e8f7e4d6d6d7c9b5f2f7

b8c2a5e2c1e1a3f8c5a3c2b1a3

f6f2d4c2c7f0c4e4d6b6a7e4c6

a6d4e1a2a6b7e9c6b0f9d6c7a4

e9e7a6f3c1b9c5c2f8d6d8b3d8

d2d8b3d1c8d1b6b3e1b4d7c4c2

e4e9a4d5e5e9c0a6e1f7c4d3d1

f7b6c1c2d4e4c8a0f9f5c0b6c7

f2e7a7f4c1f5c0d1e7c4f7c8f0

e8d5b2c6d8f9b5c0c5e1f5b4c8

f4c8e3e8f4b8f2a8a3c1c5d8f6

d6a3a3d1f4e5f7b5b6e7c1f4d4

d3b2f6d6a9e9f6a8c7c7b3c9b8

a1e3d5e2d8f1c1b9f5c2f6e4f1

b2c7d8c2e6b1b4a5f5f8e6b2b0

a2f6a2f1d5e3e1f4f8f3f2d4c8

a2d9b2d3e4a8d9e2d6f5b6c0b9

a6d1b3a2f1d5a3d2a6e6d7b8b3

d7f7d2b0e6b1d8a7b8c5f1e8d7

f1f3a0b3f3a0b6d9f7d2c4f1f8

e4c0c4d7c9b1a2c2f9e6f1b8b3

a6c8e7c2f9a4a5c8a8f4b7c5f7

c7a7e3a5f0d3b2a7d1b1e5b6a7

e2d8c9c4f3e3b2e7b2f2b2f7e1

a9d3c8c3a1d8e0a6e0a6b6b4e5

d5e9d3a6d2f8d4c9e6a7c7b7a0

d0b1d6b7e4e4e6e1d3a4a5b3b3

f7c9f7b6e1d3c4d9b9b8f6e7d2

b6c2e1b6a6c1c1b9e3f6a4b0f2

a6e5b4d4e3a3a3e7f9f4b2f6a2

d8e7f6d8a1f6b5a4e6c7a5e3c8

a6d8c2a7b0c9e2c3f3e9d1b0c6

e7a1c4c9a3d7b5b4b4b2b0b4c0

c8c6e6d3e0a6a8e5d5f7f5a5d7

c1d1f8a4c5e7f1f5c4e4a8d7d2

f5d1f3b4c0d3f6c4d5b6c1f1a6

a0a6e4c6b6f7b2c7e3d7c7c8b0

d9f7b8a5e3a8e8e4c1f3e7b9a8

c3e3a2f9e6f7d7b2b6e5b6a5c6

b1d1e3d1c6d1c3a5f3a8a8c8f4

d6a8d7d2e6e0c2a1f7d8f5b4d0

a6f4b8a7c8f6f3d5e4d1c9c1b2

d9a6f6f8c1b1e4a7e4d0e0a8e6

d7a6a4c4b5b3a9f7c0f7b2d8a6

b6d4e6d8d7a7e6a9d4d0a5c4f5

f1e1a3f3a1d5f2e5a8d9b0a7d2

e3a6d0a2c7c7d2d8f0c9f6d3f0

b9c3b4c7b8a5c7a9e1d8d2e6f5

a7c7f1e3a6d7e4f1f1f7e3c5d8

d8c6f3d5b8e3b4b5a1b6d2f1b0

c6a7f7d2c4e2a6a6c8b2d2f2d9

e8b5d1a8b3b7f3c9c6f7b6d5b3

d2a3c2f6a3b5d2c3c6f8a6d1b8

e7e6a4e2a1f8d2e6d6a5c2f2d4

f9f5b6e2c3e7a1c3a5e9a4f6c7

c7d3f3c0b8c9d2c9e8f5d0d1e6

e8d1c8d3d6e6c2c6a1e8b3f3a5

c6b3b3e4f0f0f0c0b5b3f2f4c0

a7a5b5c3e4a3a8c2d2f2b6c4d4

f3c0a8f7a6b7e1c5f2c3b5a2f4

d1a9e7a7d0a8e7d5e4d5a3a4d9

d6c3d0f4c6d6f3e2e2c1c5c8c8

d1d6e0a7b6f3a9f1b7f9f5e1a3

b6c5c1f8f8e7c2f7c2f7c2b2f5

c2f6e1d2d7e2e8e4f4c8c3a7b9

c4b6c2a2c5a8a5e3d3e4b2b3c7

e5e2b9c2f1f9f8b3d8c6c3d5c1

d6e1f7c4e4e7f9c7d5e2a7d1b3

e5b9b5d7b0a7f0e1c7e1d4b3a6

f9c6c4a5e1c5d1b0e9c1a4c3a7

a5c2e2a1d8c6c2f1d8a0e8a2a8

f6d1f0d4f1e7e2b6a2b8e4e4b5

c6f6c1d3d5f1d3d7b3f7d5b5e2

c5c5d4b1c5d0c6a8d9b5e8b6b8

a9d4b8a3b9d7d2b6e1d7b4e9a4

a6e3b4b8f5d6e4f6c0e7a0b7e6

c2a2f6c9e0d2f2b3c7c7d5d0b8

c9c4a3e5f9f3b1a1f6f4c5f9b5

b6d2c8d5f5f6b8e2f1f2a7b3a7

f7f6c3a4f5f7a4e5b8e8f2c0a8

d6f0e3d7e0f1f1c1d1a5b0d6c4

f2f5c4b2f3a4a5f1f9c4b3b0a3

f3f2d7c3a3c7b9a5e6d9c7e9f7

d8d4a8a9a6f0c0e2a7a9b5f4b8

b0b1d3c1c1d3d5a0a9b3e8c1b8

e6a6e4e7e3c4f1f2c4b1b4c9e7

f8f2e4d1c1d0d1e8c5c8a5a3f0

b6d3d0c3f3d7a0e6d1d2f2e5d1

e3a4d5b5e5b2a2b2b3f6e0a1c4

e8d4a8d4d4a0b2c2b8c4e6c0a2

c2b1f8a4c5c5a8c6e8d1c5c8f8

e4b6f2a3d3b4d4f4f0a3a5f5f3

d1f7a7a6c7e4f5b9c3c3f6a3b3

c1f0c3d2a7a0b8a8d0d6c9b8c3

e0f2f2b4c9a3b7f6b8d8a2e1b2

f2e4e4c6f9c3c7c2d7f4e4b3f5

a1d7d6b1f1c3d1a6a4d8f2a2c9

f5c4e3e8e5e2a5b6f5a2a4f6d1

e3f6e9f7e0b9e6b8b7b6c6b5f5

a1a3a3e1e2d1b1a7a7f7c4b1c7

c2c5a7a4d1b4b9a6f8e2c5b9d4

f2d2b6f4e8f8c2c4b7e7e2c6e4

f0d3f5b2b2b7c2d1d9c6f6f6f6

c1c1a8d7d7a6d2a3f9c4d0d4c5

d4b8c8c3f5f2a5f3c7d0d4c0e1

d6c5a8e1e3f4b5e3c0d0a1a1b0

e5e2b6b3d9e7b9c9a1c8a3c3a1

f4f2f7b1b2b4d3d2e2e4a1d1e4

d3d1f4f5b1d2a4f7f2f8c1d8e9

f2e1d7e7b1d1e8e3f9d2a9d1d6

e9f2a6d8d1d2f6b2d2a1b7b5a6

c8f0c7a2f6b1b4e8c1b4c5d3e9

c5a5c7f4c3f6d7f1f7b0b5c5b2

b5a7c4c6e3f8c3f8a1f1d8d9c2

c5b2c6e3a1b3e5d1c5b6f5e3f9

e3e5f3d4a9a6c8f0d7f4c6c6f7

b6d7d5a6c4c8c8e2f0b8c3c8d7

c6c2a1a2a8c8f8a0d6e7c2d5a6

f8f3d8f3f1a9a0e2e4d3a1e2a8

a5b2e0e9b3f1c7e2b7b0d6d7a4

c5b4e8a8d0a3d1d3c1a5e8e9e0

a6e6d7d4b2a1b1d4d4f4c2a2b4

e4e4a1a0a0f4b0e8d4c5e2c2a2

d1a4f2f4d7c5e3c0b4d7a7a3a0

f6c2f4b3d7d9e6b2c3b5c7d2c1

a6c8e7c1f8b6a1e6e3b2d2a4b7

e6b3e1a3e4b2e3a3f9a3e1d6b4

d9c4f8e0e2b4e0a8d9b7d1a7a2

a1e2e5d5d1b1e8d1c1d4a2c9d7

c8a5b5c5d0f1e2f0e1a2b4a2d0

f2f3c2c3a4b1c1e6f6e3e8f9c3

a4e9d4d1d7c7c3b1b3a2c3b0e8

c5b2f2f4b3b6b4b7c8c3a7e4b4

b3b6a6e2d8f3f6b4c2b6b3b1e0

a0d1d3a4f2d7a9f7e4a9a3e8b4

d8f6b6a3c8d5f6a3a5a0f8f7b1

c3b3b2a8a2d5e0e7c1d6c0b0b6

d1f1e4e8f8a3f5a9e3c2a6b6c5

c1b3a8b4f2e6d7a4b8d3a7c7c5

e9f7a4d4c3b3e3b3d2a4f5a2d8

f7f0b4c9a8c6a7e2f4f7c0a1c2

d9b8f5e7c7b6e1f4e0c3f3a4a2

a6d7a9c4d0c1c5f6e9c5f2e4a0

c8a5b3f9a7b2e4e6c3c2a5e6b5

f5b7e2c3c7c3d3a3f8c6a7b7d1

f5e1b0e7e7e7d4b2f4c3f8b2f6

a9a8d2e3a1a1d2a5e5b3b7f7a2

f4f6f4a1e5e5c2c3b8c4f7b1c6

e3b7a4a3b6e6f3e5c4c7f7d2d9

f4b2e4d1d0a7d3b9e1b3c9b2e7

b2e1c5a3e7d6f4a7e5a4e8e4a6

e5b5d2b2a0a1c5c1c5e3a1e3c2

d1d7d1e1d2e1c4a1b5a8d2c6d8

c1e8a3d5e6c3a9a3c8d9c6a5d2

a6a7a4b9f8f2d8b9e3c3a3d5c2

a8f2c9d5b8b5e0a7a4c6d8d8b8

a2c2f7e3b1a2b1b3f2a6a3f9d9

c3b6d2b1a6e5e7e6b1e5d2b9b2

a8a5b2d5d5f5a8d3e1a7b3f4d5

b1e0c6c3e8c1b6c5e0f1b9b5f1

a6a4c6e8d5a5f8a1a2d7b7e5a8

b2c1f4c1e6e3d1d9b2d1d6e4d2

c4c5c5c8f3c7b2d0d0a1f1b8d7

c7d7e5f3f3d5a7a7a1f4f7d2f5

f6a4e5d6c4b4b7b4c2c8d3b0a5

c5c1f4a9f6a6e3e8e9d9b8f6b2

a2a9d8b4a9b4e1f3e0c3a1b0a1

c5a7c8a1a5e3a6e6a4e4d0a2d1

e1e2f7a5b3f5c7d2f0e8a7f6a9

a4a3f5a6d9d5b5b3e7b3a8f6f7

a7f6c6c5c4c7a8d4c7a3c1c3b5

c4f8e2d6e2d5e4d6f0d5b6f8a7

c7b1e5e3b2a9a1c8c6d4c3c2f8

e8a7f8e2f2f4e6e0a1f2c7b5d9

d1a1b7e4c2c7e8e1a9e5d5b4f3

d1d1c4a2e1e2b5c7a2d9f6b1d2

a6c8a8b3b8a5e6a4c6f6c8a1c2

d3d0b4b4c8c9b7f3d4f3f3f4a4

a4c3d1e6e6f1b3f6f4d2e2e6c5

c8c5c1b3c0a1a8a8f0a2a7d2a3

d8d1b4b1a7f7e3d4a6e4c1f5e7

c4b3e0b1b4e1c5d1d0f5a8f1e8

c7e1e5a8a8c7c3f8b8b5f5b1f9

c2b2c2d8f3d7e1d3b5f2f5b5a7

d3a9b2c5e1a1e1a6a1b5f8c6c4

f2f0a6b5a0f5f5c6a6d6f1d5d4

b1d8e5d1c3e9c9e4e4b1d5c3f4

f5a1f6d1c8f7c1d5d3e8f4b7b4

a7c5f5f3d7a3b1d2b6a1a3a8d1

c8f7e2d8e8c7c6f4f1f2d8d5b2

f2d2d5c5b3f8a3a5a7f4f2a9a5

e3c3b4c8b4e1e8d0d1f5b9b2c5

e7b7b0e9d3f8f0a5b4c3e5b7e5

d1f2c0b1d2c4f8d3f4d2e6e2b1

e9f4f2b4f8d7c6b3f4e1e4a1d1

a8e9a8b4d3b2d7e3e6e2e5d3b0

b5e8d2d3e1b1d8e2f1b7d2e3b4

d6e4e2f6e4c5a3e4a5f9b1e6f3

d9b8e5b5c1a6a3a6a5f9c5d4e4

d3d6f2f3c8c8f6b7c8a2f4b6d5

b2f2a9e7e5d8f2a9d4c8b2a7b6

d8c4b7b7e3c8f9a5a5e4d4d2b4

a3f5a1f6c9f1c8a2d2a5e1e6b8

e0c6b1c4c2f4b0b9d5f5c0b1c4

b0d1e6b4e8b8e5b0a6f8d4a7b0

c4b8e7e4d4a5c8d5d9b1a6a4b5

c4b2a3e3b7a5c8e4b5b2f3f1a7

e4f7b7c0d6c4f8b6c0a2c9d6d3

e8e7b4f2d7e9f0f8e6a2a4b0d3

d4d0d0e8c4a4d6f9f2a9e5d4a7

d6c8b6f4a1c7d6f4c5d4c4e3e5

c9d7f7c2b3f5d3d7e6d3c2a0c2

c3e5d7e0b0f4b1c8b0f3c1b4d2

f1b0e3b7b2d6e1a5b8e4f8f3b1

c3d1c6b1f2f1a1e3c2a5e8d8d8

e2b4d0e2b7a5d3b6e7b2f6a5c2

c4f6d4f5d1c8c6d0d2d2d3b7e7

a9b4e8c4d4d2c8a7c4e8b6d2f3

f7b2f8e8d1f2a8c4b8b3d9e1a7

f9e7c5d8a9b6f3c7b1b2e9a9c7

d3a4d6a4c4d3e0f2b9b0e7a7e4

b0d4f0a5c5b3b6c7d6c4a3d6e7

b1f1f4f8e8e4a1c1f2d1b2e1e8

b5d3a3e2a2c7a7c2c8f6d1e7e1

e3e6c6a5e7f6b4b4a7f9b3a7a4

c1c7a7c6d5b7f8c6f6a2d2d0a8

d0d9f2c3f3b8b7a7b5f8c1b9c0

b7c7b3a7f6d6a1b1a6e5a7b5f5

e7f6a3f1d3f3c6d6f6e8f0d7b4

b0e8d1f2e8c9f2b1f1a7a4a1b8

b4f1a8d4a4e1b5c1c3f2f2a9e7

b8f4c7d5b6c0c6c7b4f8c2f8f5

d6a6a1f2d5e5d8b3d1f2a2d6f7

e9e7d5e7d5f8a8c2d8c7f9e3b8

b7b4f6a2b6f1a7d1a5d6f8d8d9

d0d7a6b1a3f7b7e5d7e5d3f9b8

c3a5b9e5a5b3d2c1a0e4b7f7d4

d7b5e1a7c8f7a8d4f1e1b7d1f8

d5d3d5b6e9c6f4d7b2a2b4a7e6

b4c6c0d2b5f2a2c8a3e1e1e8c1

e2c6e7a3f7b7c5e8b5a1b8b5c6

b0b2a5c6e5e0b6d3e3f5b5d4d8

f6f4f5b3c0b2d4a6f2d6d6e9f8

e4e9f7f0d2a7a3f7e5f5b0f2c3

e5b6c6e0f7d7a3d1d5d3a6f4f7

d2f7c1a2e4b8d4f1c4c5d7c2c7

d8c6c2e3d4e7a1c8a1c8e4e7d2

d3d3e8b0e4c6b7b2e1b6b2e5a5

a8d6b1e1b3f4b7d8f7e6e6e3b8

f4b8a3c8b9d0c3a8a3b9d3d2a9

a5d7a6d6f2d1f8e9f1e4d3a6e3

f8d8f3b7e4f0c0f2d5d2b6c8f7

d4d1d7b9e8f5a6b8a7f7c5a7a4

c4b1e5f5c4f3c7f1b1c3f0b7b1

a8c1f5b7d5a4f2a4f8f8d4f2c6

b1f4f5a3f6e3e7f0e4b6e5e2a4

a1b5b1d0a1d7b4b7a9b4f4e4e4

c8a7f5f2e3d5e7c1d6f8e5f5c4

d8e3e4a2a1d1d1d0b4d3d2a8f3

e4e7b4a5b3c6f4a7b0b3f6b6f3

a6d8b2f7a1f8f4f8e6e8c6d8d6

d8c6f4e4d3c1f3b5f3d1f5f2b5

c7a8a0e6a1b7e2f8e5e7e5e7b8

e1e3e8d1f1f6c1a4f7a0a5a8b5

f1f1c6b2e1d5e6f6b7f8d7c5d7

f6d2f6b5c9c3d4f7d3c7c7c2f5

c2a0c3d5f2e1e8d1f0f4f5c0b8

c7a3b8f4d7d7a4c4c8d5b5e7e1

a3b1b8a2f6e5c9c3a4b4d7a1d0

e1e2e5e2c4b7b1e1b2f2b0e1f2

d5e9b3e4c6b4e7a4e0d7d3d3b8

c8a6d3f3b5f4d4b6c1b5b2e5b0

c7e8a5d7e6a2a7c7c7b7f0e5d1

f2c8d0c6d4a2e0c4e3c9b2f7a5

f8b2a1e2f2e9b2b8b0b5c8e6f6

d5d2d7d0d6d8b2e8e4c8c5b7c5

b2a1f6b8a5c2c0c9d2a1b8a6e2

c3f6c3e4e2d5d7d5f5b7a1c4d0

c3c8f8d3a4c1d5c4f0d3a6f1f5

d2e2f7d4f3f9f8a5b9f1f5e3e4

b9d7e1c3b0f1c6a2b4a4c9b5e5

d2b1e4f6a1e4c7a5a3f4f1d3a4

a6c6f2e1f7d2a7f4f4d2f3d3a0

a8c7b4d3a5c1c5f6d8b0d4b8f4

e4d5e3e4a8d2b6d2b9f1b4d4b2

e1b5b7c2c5d6b3f4f6e2c4a3b4

c3d8e1b7d3e2d8b2e5f1e7a1e7

f0b1d0d0f4b6f7c8d6c2f1f3b1

b5b3a8b2a1b6d3c0b2d1e3e7e2

d3c3d0b7d7d7a7d0e1d4c4a7b9

f3e9a1f0c6d4d4a5b6b5a1e4b0

a5c8d6e1a3c2e9c6e3d5e1f2d6

e5f8b9e2e1d7a3b9f4d6b2e2c6

f8f7e7b4f5b3a8e9c1f2b9c3f4

f8b6f9c7d3b1a3c0c4f8c0f7e1

f1c9e3f2d1f3f6a5a5e7a6d5c8

a4f0e7f1c7c4e4e3d2e8a3d1c1

a6f2e1d7e3e6f8d5c8d5d2c3c5

f0b2e1b1c3e5c3d6d1a0b8f1e9

f4d7a8a8b8d6b0f7b2c4d3c1c4

c3a6e4c9f6b5e5e4b8e8a1b7f3

e1f5b8e5d1d2b1a0d8a8c2d6c2

e1d0d2c6a3d5a9e2b4a3c5c8b2

a7b2c1f7e8c3b2d2b2d4d4d1e8

f6a2b6f3b4f5d4e7d3c9d3e7c1

a2b6a8e3a4c8b5a1a6c4a7e6c4

b7e7c2c0e1d9a5d0d3c7f6f5a7

d1d2f3b8d5c6f7d2c5d7d6b4d9

a1f7d1d2a2b7f7f7d7a8a5e2c5

f4f4a8a1c1b7f4f4a4b1d9e5c3

f1c3b7c3b4b7f4a6e8d1f2e3f8

e6d9f6c4c6b9f8d0b5f7b8e1b1

f3a3a6f1c2a0a3c8f6b4d7a2c4

d6b3b5f8c9b3d2a8e1e3a9f0e0

b4b8a7f8d7c0c0d5c8f8f9a8f9

f3f2c2e6e4f6c1b4a4d7d8e8d6

f8f5f1a4b6b3b4d7b8f5b4d1d5

f1b8c1c8b9c8e6b2a2e0c4b7d2

b6f9a8b8c2f8e8e3c5b4e3f3d7

e2e3a1c5d6a8f9c4b5e2f1d8f3

b4e8c4a1c8f1f4f5b2c2d1c3e8

c8a8d0e3f1a1d1f2d7e6d5f4d6

c6d5f8f3d3b4a6a9d1c6f4a5c1

e8b0d5c5a5b3d2b6a3f3c9c4c6

b2b3f2e9c7d1f5c5d8c5a4f6a7

e0a7c6b3c9d3a3e8b1f4f5c0e4

c6a7a9f6a1e7d6a4b4f4e3d4a4

b8c3a9f5e7f4f8e4c3d2f6c3c7

f5f3a6b7a8c2e4c2b5d5d5f1a4
Share twitter facebook linkedin

You have a Big Ball of Mud

[mahlen]

First, read this essay: The Big Ball of Mud. It is an interesting look at why, when we all know that spaghetti, gnarly, twisted code is bad, that it happens anyway (hint: it may mirror your understanding of the problem).

Ignore the "don't touch it" naysayers. Even before it's done, it'll be much nicer code to deal with. You can make decisions with less nagging doubts. You'll code onward with gusto. You'll be able to accurately predict the names of methods without looking them up.

Test the current state of things at all points through the process. I'm hoping that you have lots of automated tests you can run everytime code is checked in; if not, make them FIRST, before the overhaul. You are majorly diverting the intent of the code at hundreds of points; you can run astray in so many places that the above naysayers would be correct. Constantly assure yourseleves that the code is working. Go out of your way to ensure that the code is buildable and runnable, even to the point of writing scaffolding you know will be soon thrown away.

Burning a little incense every day in obesience to the Gods can't hurt, and will make the room smell nice.

mahlen

Shantytowns are usually built from common, inexpensive materials and simple tools. Shantytowns can be built using relatively unskilled labor. Even though the labor force is "unskilled" in the customary sense, the construction and maintenance of this sort of housing can be quite labor intensive. There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth. Shantytowns emerge where there is a need for housing, a surplus of unskilled labor, and a dearth of capital investment. Shantytowns fulfill an immediate, local need for housing by bringing available resources to bear on the problem. Loftier architectural goals are a luxury that has to wait. -- from "The Big Ball of Mud"

Quick suggestions...not so quick solution

[runswithd6s]

As everyone else should be telling you, your existing codebase is a very valuable resource. It's been tested and debugged a number of times. It's mature, it's stable, it's there. Don't throw out the baby with the bathwater.

Now, on coding standards and how to incorporate them into a legacy project. Your concern is NOT format. The format of your code, such as indentation, spacing, etc, should be the least of yoru concerns. Everyone has their own style, but there are wonderful tools that you can use to force everyone to a single style, ident and astyle just to name a couple. Use a wrapper script to the CVS (or similar system) on checkins. Force the code through an automated cleanup. check the code back out and make sure it compiles/runs as expected.

What you should worry about is how much your design team has embraced the "black box" design principal. Parameters go in, results come out with no "side-effects" that impact the remainder of the code. Make your code re-entry safe, i.e. stay away from globally scoped variables as much as possible.

Someone's going to give you the whole OO-Design sales pitch. Yes it's nice on paper, but don't sell out because something looks nice on paper. I learned this the hard way. I have a tendency to overdesign things. When OO, this gets to be really scary. I waste my time writing object classes for "everything" instead of simply designing the software to its functionality spec. Make things more "object oriented", "functional", or "blackboxed" when you find yourself repeating code elsewhere in the application.

Don't spend a lot of time with naming standards such as Hungarian, Modified Hungarian, etc. Find a style that you and your team is comfortable with for the Interface API level. Below the Interface API, be more lenient. It's likely that portion of the code will undergo many changes anyway.

And most importantly, document! This is the singlemost important issue of any coding project. Either force your developers to write docs as they go, use embedded documentation solutions, or hire a techwriter to follow you and your team around for a few months. Documented API is the quickest way to start someone off in the project, and a great way to keep track of the flow of the program.

Re:Quick suggestions...not so quick solution

[Fubari]

About over design in OO - check out YAGNI (You Ain't Gonna Need It) and other XP principles here. Or, take a look at this article, Is Design Dead?.

Suggestion for rewrite method

[mcpkaaos]

When you look ahead to a large scale rewrite, there are few things you must keep in mind before even attempting such a task. If you expect to jump in and hope to tread water, think again. Unless you sit down before you start dividing the coding tasks up amongst the engineers, you are simply begging to go down the same road twice.

Before you begin, make sure you weight the cost of a rewrite against the cost of maintenance. Many engineers (myself included) feel so ashamed for poor code that they want to dive right in and fix it. Whereas this could potentially solve the problem as far as you can see it, it won't make the financial people happy after watching your team spend 4 times the money than they would just dealing with maintenance. In other words, make sure you are adding a hell of a lot of value with such a huge task. This is a good time to analyze the risk of a rewrite as well. How critical is the system? How many users? How will this affect any users that have built systems around your interfaces? Will they need to recode portions of their systems as well? There are many more questions to ask yourself, but these are generally the ones that will come from people outside of your immediate group (and usually right away.)

If you do decide to rewrite, there are a few things you should do before analyzing the current code base. Review the original requirements and compare them with any new requirements you may have now (or later, if you have any insight into the future, such as rumors of change in business practice, etc.) Once you have worked out any inconsistencies or pitfalls, it's time to compare the original design against the original requirements. Look for inconsistencies between the two. Usually you will come across a few requirements that didn't quite make it into the design as expected (if at all.) If the two are so far apart that there is no feasible reconciliation, your hopes of a rewrite are pretty much dashed - it's time to start looking at a fresh start. Hopefully, however, the design met the requirements adequately, with no "creative designing" put in place to get around poorly thought out requirements. Note that you should do all of this before you start going through the code. Leave the code out of it for now.

Once you have established that you can safely move forward (i.e., the design followed the requirements), hold code review sessions with your entire team, targeted at the original code. Ensure that the design was followed properly and that any bug fixes/workarounds/etc are clearly documented and easy to spot. Usually this is not the case, else you wouldn't be contemplating a full rewrite. This is by far the most painful (and time consuming) task, and is usually the point of failure in rewrites. If not enough attention is paid to the caveats of what you have now (and why they are there and not resolved), you will paint yourself into the same corner with the next go 'round.

Without spending too much time on that topic (which is easy to do, believe me, I've done my fair share of code reviews with large teams), I'll assume that you are planning on continuing with the rewrite.

Once you've made it past the reviews, it's time for a new design. You won't necessarily need to completely redesign the system, but you will have to (at the very least) document a plan that will provide a solid path to the new system. Even if your functionality isn't expected to change, it will, so a mild to moderate design document will be your life-jacket. This will be difficult, as it's no easy thing to feel like you're simply writing the same words over again, but stick with it: you will find new methods in this design that will help to avoid the traps you were caught in the first time around. Have the utmost diligence with this step. Review the design often, and brutally. Bring in members from other teams for sanity-checks - a fresh set of eyes can do wonders, even if they are unfamiliar with your product.

Now, onto the coding!

Whatever you do, do NOT use the original code for anything more than a guideline. Create new projects, start with blank sources, whatever it takes to start anew. When you rewrite a system, you are actually writing a new system that simply mimics the funtionality of the old. Your performance is going to change, your stability is going to change. Don't think for a moment that you will end up with exactly what you had before, only prettier. This is a pitfall that fools many an engineer, especially the engineer who trumpeted the idea for the change. ;)

Finally, more than just about anything else, remember this: when rewriting a system, be sure to adopt a coding style that will prevent spaghetti code. Be true to a modular design and cleanly separate the functionality into logical groups. This is always easier in an object oriented language, but very achievable in the procedural world as well. Make sure that this time you document your code inline. In other words, instead of simply commenting your code to say, "This function does this, then that", say something like, "This function performs the following business rules, as dictated by requirement #x, conforming to the component design section 5.1." Tedious at best, but far more valuable than most people realize. Make sure you also document the bug fixes that were migrated into the new code base, also. Eventually, someone will want to review the code to make sure the same bugs that were once in the old system aren't going to rear their ugly little heads in the new system. Finally, unit test the living hell out of your code as you write it. This is the last ditch effort (before QA and integration testing) for ensuring stability (which is the main justification in the end for the rewrite.)

Anyway, sorry for posting a novel, I hope something I had to say here can be of some use. Best of luck should you tackle the rewrite! :)

-------
[McP]KAAOS

Tablize It!

[Tablizer]

Put as much in possible into tables. That makes it easier to find, view, add new pages, menus, product categories, etc.

Of course there will be exceptionss that need code instead of tables. Have special "override" points that allow you to override stuff as needed. Make the "overrides" their own function. (Example: sub foo(row); if row.clientID=7, then color3 = blue;.....). The override points are a lot like "on_x" IED-GUI or database triggers/events.

If you store SQL in tables for a report generator for instance, there may be situations that tablized SQL cannot handle very well (or not worth adding new columns if only one or two instances differ from the majority). So make a routine to override each SQL section (SELECT, WHERE, ORDER BY, etc.), and another to override the whole SQL for the really complex stuff like correlated subqueries.

I have successfully tablized HTML forms also. It is a lot of up-front work, but is better than coding each from scratch if you have hundreds.

Don't touch the semantics

[JojoLinkyBob]

Noticed you have a lot of perl scripts, which is similar to the situation I've been in before. Come up with some agreeable style (i.e. indenting) but stay away from the semantic issues, because a perl script speaks identity loudly. Case in point, how do you reference an item in a list: @list[0] or $list[0], Both are accepted, and in way, both are intuitive, it just depends on how you're learned the ropes. End Transmission.

Re:Don't touch the semantics

[chromatic]

Sorry, that's wrong. The correct way to refer to a single item in a list is $list[0]. @list[0] refers to a list of elements. (An list of one item is still a list.) See this Slashdot article for more.

You're asking for subtle bugs by conflating them. (Whether or not this was a good design decision, it'll be fixed in Perl 6.)

Re:Don't touch the semantics

[JojoLinkyBob]

I'm talking about simple cases such as:

@list = ("apple","banana","chihuahua");
print "\n@list[0] @list[1] @list[2]"; print "\n$list[0] $list[1] $list[2]";

Both of these print statements have the same results. My point is that every programmer has their own special coding nuances. Taking these away to adopt to some standard might be counter-productive, because in most cases the original developer will probably be making revisions later, and some of us probably know how hard it can be to read someone else's perl :) Speaking of semantics, maybe my original comment was worded precisely enough, and therefore misinterpreted...

Re:Don't touch the semantics

[chromatic]

I still disagree. If I came across @list[0] in a code review, I'd mark it as wrong. There's a reason Perl warns about it. Any Perl developer who doesn't yet understand the idea of context needs to know it to do anything serious with the language. Any Perl developer who knows better and uses it in production code just to be cute needs a better outlet for his creativity.

If your organization eschews coding standards (especially those that follow compiler warnings!), you'll be very lucky to avoid several common pitfalls. I'd rather have every developer on my team willing and able to modify any part of the code. If that means I have to use 2-space tabs, so be it. If that means I have to teach other people how to use anonymous subroutines and the map operator, so be it.

I trust my fellow developers to follow best practices for a reason -- to make development (including maintenance) easier for everyone. They're smart enough to adapt to intelligent standards.

Re:Don't touch the semantics

[thogard]

I would back that up as being wrong...
$list[0] is a single element and @list[0] is a list with a single element.

This is a common problem when coders don't understand what is going on. In this case perl will build a new list of one item and then deal with it in a list context when all that needed (and wanted) is to deal with one element. This type of problem isn't perl specifc (but the syntax is) as these types of ineffiencies show up in things like OOD data access methods and many over designed programs.

Help decipher a comment from 1982

[Anonymous Coward]

Searching for the first documented case of the word "fart" on Usenet I stumbled upon this "joke." http://groups.google.com/groups?selm=anews.Aihuxn. 107&output=gplain

I don't seem to understand the punchline, if anyone can explain it to me I would really appreciate it.

Izzy was getting on in years, and had a terrible digestive problem. Every night when he got in bed with his wife, Bernice, he would let out an enormous amount of gas, accompanied by the sound: *HONDA*!

As their fiftieth wedding aniversary approached, Bernice said to Izzy, "Izzy, soon we'll be married fifty years, and I still can't smoke in bed. Won't you see a doctor about your problem?" Izzy was reluctant, but bowed to the inevitable.

His family physician was baffled, however, and said, "Look, Izzy, I could tell you I know what the problem is, but I'd be a liar. I'm sending you to a specialist, Dr. Yakamoto. He's the best."

Izzy did as he was told, and went to see Dr. Yakamoto. After Izzy introduced himself and explained his problem, Yakamoto nodded sagely and asked Izzy to step into the examining room. Their dialogue follows:

Yakamoto: Please to open mouth.
Izzy: But the problem is with my... uh... other end, doctor.
Yakamoto: I know. Please to open mouth.
(Izzy complies.)
Yakamoto: Ah so, as I thought. You have abscess.
Izzy: But doctor, what does that have to do with gas?
Yakamoto: Well known medical fact, abscess make the fart go honda.

-- DJ Molny

(ucbvax!ihnss!ihuxn!djmolny)

Re:Help decipher a comment from 1982

[Pinball Wizard]

Its a pun of the phrase: "Absence makes the heart grow fonder".

Actually that was kind of funny, but you would have to be familiar with said phrase.

Re:Help decipher a comment from 1982

[dgroskind]

Absence makes the heart grow fonder:
Isle of Beauty, fare thee well!
Isle of Beauty. , Thomas Haynes Bayly. (1797-1839)

Also the first line in an anonymous poem in Davidson' Poetical Rhapsody (1602) but probably popularized by Bayly.

See also:

Absinthe makes the heart grow fonder.

Abstinence makes the heart grow fonder

Maintenance makes the heart grow fonder

Obsolesence makes the heart grow fonder

Absynth Makes the Heart Grow Fonder

etc.

Can we change the word "Developers"?

[IRNI]

Ever since I saw that horrible, nightmarish Steve Ballmer video.... "DEVELOPERS DEVELOPERS DEVELOPERS DEVELOPERS DEVELOPERS DEVELOPERS... DEVELOPERS DEVELOPERS DEVELOPERS!!!!!!!!!!!!" Oh god lets call it anything but Developers! The image of a big, fat, bald, sweaty man keeps entering my head when I hear that word.

Rewrite?

[stevewz]

Treat your existing product as one big, well developed prototype. Not many people have that advantage in a large-scale development effort. If you can somehow put a function-freeze on the existing product, that will enable to you to develop the new product to match the functionality of the old product without trying to hit a moving target. When possible, scope out and implement your improvements and efficiency improvements at the same time. In many ways, this is the best way to do a large-scale project.

Consistency

[Malc]

Unless you want to go and reformat all of your old code, leave it as it is. Apply coding standards to all new code. With old code, use whatever style is already present in the file. If you think that somebody can automate a way of reformating all of the old code in a half a day, do it... but don't bother wasting anymore time on it. Only considering reformating old code if somebody does a lot of work (e.g. bug fixing or functionality changes) in the future on the file anyway.

Yes, I know, it is nice to have a clean slate. But nice is all it is... it's not really necessary, especially for parts of the old code that doesn't get touched very often anyway.

For the Love of all that is Holy!

[955301]

Don't do it Man!

action:Leaps behind the nearest dune and holds head low in the sand:action

Just let it go! You've laid down your mine field (developed) and activated the detinators (deployed). Don't go walking back through and "sweeping" anything!

Kaaaabooooooommmmmmm! I expect to come back to your page to see camel parts everywhere and your little Nomad guy hanging from the cool purple font!

Let the code evolve

[Anonymous Coward]

If it is a royal pain in the ass to maintain your existing code *and* the code can't evolve the way it is, then rewrite.

"150+ perl CGIs, over a meg of code" in reality is a small project. If you had 8,000 files, then I would call it a "large scale project".

A lot of rewrites are a result of ego. Everyone can "always do it better", so it goes. Let the code evolve.

Whoever is the main author of the code will know if it needs rewriting or not.

Peace

No such thing

[Neubie again]

R U Serious!?!

First, read all previous comments.

Second, realize that a "sweeping rewrite" entails just as much work as the first write. The only thing you will (should) get from current code is a functionalty demonstration. (I.e., the current version is a "throwaway" prototype.)

Then, take everything you know about software development, learn a lot more, write the requirements specification, get it approved, then start thinking about technology, languages, and design. Coding should be about a third of the effort - the middle third. (Final third is test/debug/test cycle. If you didn't know that already, then don't even attempt this.)

A free solution

[Anonymous Coward]

Why not just open source it and let the community "fix" your code base..

I think he can be the most helpful.

[nkabbara]

Go to joel.editthispage.com. He has years of hands on experiance and when it comes to software management his advice is right on.

Re:Use framework, seperate code form display.. and

[cwinters]

Perl has plenty of web application frameworks, and they're all approach the problem differently. See the mod_perl page for starters. <plug>OpenInteract is probably one of the most complete of them as it includes a framework for data access, security, user and group management -- all the fun stuff.</plug>

Chris

Does it work?

[litewoheat]

If it works don't break it. Pointless rewrites do nothing but feed the programmers ego and knock the company out of business. Its happened over and over and over and over again. Progress, not regress.

the obvious

[archen]

If you're sticking with Perl make sure that all the code runs flawlessly with warnings.

#!/usr/bin/perl -wT

and

use strict;

I've actually done this on 2 large projects..

[Beatlebum]

..both projects were >250,000 lines of code. Assuming you don't have the resources to redesign from the ground up, my advice would be to retrofit in a piece-meal way. Try and factor out key pieces of code and rewrite if necessary. Whenever you make a change, run the old and changed system side-by-side to verify you have not introduced bugs. As you progress, it makes sense to write scripts which can automatically perform this side-by-side QA. As you rewrite sections of code, use the Facade Pattern to maintain compatibility with old code you don't have time to rewrite. If you do things right, the Facade can be stripped out later if you do decide to change internal API's.

First Modularize, Then rewrite?

[nonlucent]

This is just an idea, I have no actual education on this subject. I do have a good amout of experience working with old, uncommented code- some of which didn't even have an initial design ( verbally gave some contractors 'a good idea' of 'what it should do') ! Anyway- here it is; first break up your EXISTING ugly code into a proper grouping of modules and functions ( your new design, maybe with some extra small functions for easy debugging ), keeping all the nasty little tricks added to fix the nasty little(big) bugs. Then, rewrite each little function one by one; after each fix, checking the output of a good automated tester is handy! Otherwise a whole QA cycle is required, but this IMHO is a _safe_ , if somewhat painful method, that gives you better options in terms of bugfixing/adding features at the same time( bullshit? maybe! ) since the 'in-between' code will (often/sometimes/probablynot) still work.

Re:Extreme Programming - IT WORKS!

[unixpro]

I've used this exact method (XP) for cleaning up a project as well as with new projects. It takes some work, but the rewards are far more than you have any right to expect.

Re: Design from scratch

[Spud the Ninja]

This is, I think, crucial. If you're looking at retro-fitting your project, then I'd assume that it doesn't work the way you'd like it to.

If it's really just a case of redundant functionality, then it wasn't designed well in the first place, or later add-ons/bug-fixes broke the original design. That should be fixed. If you need to add more functionality, then design in important in avoiding breaking the current design.

It seemed like a pain in the ass when I first started, but a good design job save so much development time, time that could be used later for documentation!

Stay focused

[Anonymous Coward]

1. Figure out what you want to achieve (make a very specific list)
2. Anything that doesn't fit in that list absolutely don't do it. No matter how tempting. Beat people down who do try to stray.
3. Make many phases (checkpoints, milestones, etc.)
4. Check in often.

The list should not be:

1. refactor code
2. make it easier to add new functionality

The list should items like:

1. Make it easier to replace the right side of templates
2. Make it more efficient to use the database
3. Create some layer between some top and some bottom.

My suggestions

[Pinball Wizard]

My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code

First of all, I think its important to realize that you have a medium-sized website and not a big software project. Therefore, some of the above comments recommending refactoring, UML, and eXtreme programming may be a bit overkill.

Web programming != software development! Its usually done at a much faster pace. Even if an object-oriented approach is taken, you are still probably talking about simple function libraries rather than complex C++ or Java classes. Again, overkill.

150 files is still a small enough project to be managed by one or two decent coders. Actually, I just looked at the amount of stuff I've written over the years for my online bookstore and its more like 500 files and over 4 megs of code. I don't feel like its too much of a job to manage this codebase by myself.

So, here are my recommendations.

You probably have gotten better at programming since the time you started your project. Take a few of the most recent CGIs you have written and compare them to the first ones you wrote. You just might notice a glaring difference in the quality. Also, the first pages you wrote are likely to be among the most important in your project, yet they are also likely the worst quality-wise.

Regardless of what language you program in, I think its important that you can tell whats going on in the program by reading the comments. If a manager can understand what a program does by reading the English bit, there's a good chance other programmers will be able to jump in and help as well. One specific rule I also follow: if you do regexes, say IN ENGLISH what those regexes do. I say this because regexes are one of the hardest things to read.

Look for any code that can be "factored out" of your scripts and put those into function libraries. Then include those in your program. The only problem with this occurs when you have huge function libraries that slow down your scripts when you include them. In that case you would logically separate your functions into different files. I have included very common functions in different include files, so I can make the actual code compiled or interpreted as small as possible.

Consider using a flowcharting tool as an aid to programming and/or documenting your code.

Standardize how you name variables and functions, write comments, identation, and spacing.

Be sure and include the date you write your scripts in the comments, in case the filesystem wipes this out.

I'm sure theres other things I've left out, but following the above guidelines have helped me do exactly what you are trying to do: manage a growing codebase. But don't forget, this is web programming, not rocket science, and some of the above suggestions may be more trouble than they are worth. Keep it simple.

Re:My suggestions

[angel'o'sphere]

Hm,

The coment is valid in some points but not so good in others.

<i>
Therefore, some of the above comments recommending refactoring, UML, and eXtreme programming may be a bit overkill.
</i>

XP is a very good method for this project size.

Also it *IS* a big project. According to Brown et al. "Anti Patterns" the perfect project size is 4 month done by 4 people. Is any property greater, then the project can be considered big, or wrong approached (1 man working 2 years is WRONG, except you did it in 6 month and then maintaned it for 18)

In conjunction with the allready mentioned book by Fowler: "Refactoring" XP would give you:
o Pair Programming
4 eyes glancing over the code while making changes(applying refactorings)
o Unit or Regression Testing, allready mentioned by a lot of posters
o Other XP properties you get for free(customer in your house, you are your own customer)

The poster above, is also contradicting himself: No UML but later he suggests making flow charts.
Erm ... one diagram type used in UML *is* simply a flow chart, albeit its called different: Activity Diagram.

His suggestion to factor code out in libraries is of course a very good one. (Besides he thinks its overkill :-) that is called: Refactoring ...)

However, why not documenting that with UML?

A simply dependency between packages or components would tell you: this package uses that one (code in this package uses code in that package).

You do not need to use a UML tool, you can do that on paper, photocopy it and distribute it.

Nothing is faster to grap then a visual description of how your code depends on each other and how your code interacts with each other.

If you know your code very well, *YOU* do not need that. But during reworking the code, it will change faster than the team can follow (keeping the mental model in sync with the software build).

So having a picture of what your goal is and also one describing how it currently looks is very good idea.

Finaly I dissagree that there is a hughe difference in "Web Programming" and other programmings.

Depending on the risks involved you might choose different software development processes.

However XP is one of the simplest processes available at all. If you are not even able to perform that one, you surely do not perform any process. (That does not mean that switching to any process takes some time, and patience to do it!! And it does not mean that everything to be done while switching is EASY)

Instead on relying on english comments another poster pointed out, the code should be the comment, speaking function names like:
makeBig(x);
instead of:
f(x); // this makes x big

Thats perfectly right!
For regexp, you only have english comments of course!!

If your manger can not read that, he will also not understand plain written english comments.

Another poster also pointed out: If the management agreed on unit tests(regression tests) you should use the time planned/alloacated for that for describing tests and conducting them.

Hu homm ....

Somehow you will test your software. The only point is software engineers might it not call testing what you do while you 'do testing'.

There is nothing a manager has to descide and shedule or to account on in a special way.

If you do planned testing, the management CAN account.
If you do not plan, it can't.

If you test your web application with a dummy users clicking in a browser on pages ... you waste FAR more money and resources then if you do standard software practice and write unit tests.

Sidenote: about 10% of your effords should go into those unit tests(writing and conducting!).

Its realy strage to me that a lot of posters (I remember about 8) said: no UML!

I can not imagine to start a project without spending at least a man day for absolut minimum designing.

If you use UML and a CASE system you spend 60% of the whole project time with making and refactroing your designs.

But your whole project time might be 50% to 80% shorter than doing it without CASE and UML.

As there are not many CASE tools generatng perl code its of course not worth to spend that amount of time in a PERL environment.

However a day for makng some descriptive diagrams is allways worth it.

angel'o'sphere

IBM's (re-) write of OS/400

[Degrees]

IBM was faced with a simlar problem when they came out with the RISC version of the AS/400 hardware. They needed to re-write, but they had to maintain absolute compatibility. I think the articles describing this were in Dr. Dobb's Journal (although I might be wrong on this). Unfortunately, I do not have URL's for this.

One of the success factors they found was documenting the interfaces for each and every call between modules. The documentation turned out to be excruciatingly precise - but this led to zero ambiguity (and thus 100% interoperability). It also required meetings (sometimes arguments) between programmers to hash out what was actually going to happen. Another factor was that they decided to allow zero 'overloading' of functions by different modules. A programmer was not allowed to duplicate someone else's work, nor create a second, incompatible version of a function provided in a different module. If the function was provided by someone else's module, the programmer had to call it (properly). The result was that they reaped the benefit of object oriented programming - reuse and refinement of modular libraries.

It would be better if you could get the real scoop from the real programmers - but this might give you something to think about.

Set up a message board

[dilute]

I would set up a Slashdot-like Weblog in order to collect people's learning during code review, and be able to browse and search the results later.

Re:Perl: another close-minded fundamentalist relig

[Anonymous Coward]

maybe it's because good programmers can write good perl code. just because you can't does not mean someone else can't.

first GOOD candy cane !!

[Anonymous Coward]

1fb
4b1f
4b6bb
4a5e5d
dc2f1f5
bf3d4b3
bd9a6f8
ad8b5f3
dd5e3e1
eb4f6b3
ad4e2b1
db6b0e6
fd1f6f3
ce4c2a2
86a7c1
73a2b
52b2
39b
4ed
6c0a
9a7fb
8e6d2f
fb7e2f8
be4e4f4
ef8d8a8
ae1e1b3
fb8e8d1
ca0f1d2
bb2b4c7
ce7a2f1
cd7c2a2
cc7c8e4
23e8f6
21b2e
73a2
63b
0bc
8d6d
8f8cb
5b5f7f
fd2b6b1
bd9b5a4
ca8c0c9
fa7b7a3
ba2d5b8
ec9e6d8
ef4c1e7
ee5d4a4
ad7b6d9
aa6d3f5
24c6a2
67b2b
69b4
46b
4de
4e3e
2a1bc
1b7e0d
df3c5a5
af5c6e7
cc5e4b8
cb8a7c0
ae5c3f3
da3b6b6
db7a3a9
ca6c1b7
ec5b2e6
ba2e5f3
47c7d1
26c6d
90e8
61b
0de
2d1b
5a8ab
7a9a1e
cd4d8a3
dd7e0b2
be2d3d5
bb3c2b4
ea3e5a8
ac0a8c8
ea2d2e8
cf3a4d5
ee5c7d8
cd8c8e6
02b8f8
17f4a
51e1
91e
1de
4b2d
4c8af
3a6c5b
ec1d2a8
ca2e7f5
cb1d0f0
ea3e2b1
ff4e0a7
cf5b3d8
ca1a7b9
fe5e5c7
df8c5d3
ba0f8e5
38e3e7
24a2c
97a3
54d
5dc
4d9c
8f1ea
2b5c5f
fa3b4d9
dc0b2d4
de1d4c3
ad8e5c3
de7b2a3
fa4d8f8
ae7b3c8
bd8c5f4
bf6f2a4
ef9c4a0
53b8f7
41d1b
36b7
56f
7ea
7e2f
7e4bb
8e4d3b
ab7a4a8
bf5b1a1
fb9f5a7
ad5f5a3
af2f9e4
dd8d6f8
cf0d5f7
fd5a3c3
fe7f2a7
ee5f8d7
06f7c4
69d2f
88e6
50e
2cd
7c6e
7b6ca
2d9f5d
fd1b3b4
ad1c3f9
ca8c9c2
bb4a8e9
da6c3c8
ae2c9b2
ad8f1f3
dc4f3e2
cb5a2a0
ac6c6e6
90c4f7
48a6e
06d2
12d
1ba
2f0d
5d6ca
6c3c3e
ba3d4c4
da5a8b5
ca4e2b8
cc5b5f7
da3f5f4
bd8c1f5
dd2c2c4
aa3b2d7
fc8b5b4
be7b3b9
97b2a2
36f2f
79a8
62e
4ee
3e1c
8c1ca
6f8e6a
aa5b3e0
de4b9d4
de5c1b7
ef0d2c8
fa5e2e9
cd3a4f4
cf9d9d5
ee2f7b6
ac1a1a7
da3a6c5
17e3d1
88d0f
57c6
02f
1db
4d6a
4e3dd
3b2c1b
fd1e1c1
fb8c5a3
ab5b1f5
cf1f4c4
dd4f4d3
fc4a5e3
df5e4d3
ee6f5d1
be3e0e6
cd4a7f5
34a4d2
96f6c
47e4
49b
3eb
9b8f
7b5dd
3a0b5b
ef4b4f8
ea8d4e3
cd0f2a3
ba3e3d9
ae4b4a6
ac6e8a8
fb3a6b1
bf4b7a3
fe6c5c8
ac0a1b7
57c1d4
87d5c
86f9
13b
3ac
8c3c
2a1be
9d6b8f
ff1b6b8
ee1c7b9
bf7d4a6
af2d8f7
fa4a2c1
ca8f6b8
df2c3f3
df3d1a2
bc7e3a4
ef2d4d4
35f2d6
33b0b
66a0
87e
2ee
5a4a
3c4ac
8d3c3c
ca8c0b3
cf6c3a7
dd6c4f8
ea2f9c3
da7d5a4
ff5e7b7
ee3a8a1
aa9a3b4
df1c4b5
af2d5c7
61d4b2
38b4f
84e4
76d
3ac
1b1f
7b6ad
3b6b3b
fa6b2a8
bd1e3a4
fb7a4c0
cc2b5d1
af6b1b7
ca8d2e6
dd5a3d9
dd0b4d4
df9f9a7
fb5c7f7
65d4f4
34f2a
98b9
31c
7fa
2b1e
4c1de
2e4a3c
cf5f3d3
ba1c8f4
cb8a5e6
db4c1b5
be7c2d1
cd2a5b0
db4c4d4
da1b4d3
ea5b7e5
ca8d3f1
12e1a8
36d8d
31a3
31f
7bf
3f2a
9b4af
8b9e4b
ca1b5b8
ec8c5a1
da4b8d6
ca3a8a6
ef3d5b4
db5d0e1
ed9e9e4
ce4c1d8
da4d1c4
fa3c0c1
60d5c9
00b5b
41e1
48c
3cb
2d3b
2e2cf
7b1c8b
ea5c8b6
fc2d6d4
fb1a2f2
bc2f7d3
bc7e2d9
fb6c1e6
fa3f2c4
ac4f4f1
bb7a8b2
bd7b6d3
20e5b7
14f6b
78b5
78e
9db
3b2f
2b8df
2e1f5e
ee0b7c5
fe3b4e2
ac2e9b8
bf2b8e2
af6c3d7
bb1d2a3
ab7a1f4
bf5e2c5
ea9d1f3
ee6e8b7
14b3c9
35c0b
57d3
14b

Microsoft, paragon of quality code

[dmoen]

This advice makes sense if you are embedded in a culture like Microsoft, where you have a huge quantity of legacy code, and a really shitty software process. Microsoft has no record of what design decisions were responsible for making their code the way it is; they just have the code itself. They can't change it because they don't know what they might break by doing this.

If you enforce a good software process for every line of code that is written, then you have more flexibility. For example, suppose that you use Extreme Programming. Then you will have a unit tests for every bug that was fixed. Each time you re-run the unit tests, you find out if any of your previous bug fixes have become undone. This gives you the freedom to refactor your code whenever it needs to be refactored.

Doug Moen.

Re:Microsoft, paragon of quality code

[William Tanksley]

Microsoft doesn't have quality code, but it DOES have reams of documentation. They have, and follow sporadically, a really strong process -- in fact, a huge number of really strong processes.

In many of their products, the design decisions are documented, and documented well.

As with all documentation at that level of detail, good luck finding the right page...

Other than that, you're right.

-Billy

Re:Microsoft, paragon of quality code

[dmoen]

Interesting. I think that a software process can be "strong" (tedious, bureaucratic) without necessary being "good" (ie, effective). Does Microsoft's current process actually deal with the problem of not being able to refactor your code without losing all of your bug fixes? Keeping all of your design history in labrynthine documentation that is difficult to find anything in is expensive, but doesn't solve this particular problem.

Doug Moen.

Re:Microsoft, paragon of quality code

[William Tanksley]

Well said, on all points. Microsoft's problem is that they have too many processes, and IMO most of them are too heavy (and therefore they aren't really being well-applied). So they don't have A current process; they have many of them.

I personally have a bias against strong processes. I do admit that they can work, and work well; but they also cost a LOT, in every way; it's very hard to tell whether you're applying enough of the process to make it work. I'm definitely an XP fan; there, your process is light (although it IS hard), and feedback is immediate and automatic.

-Billy

Perl is a *tool*

[tmoertel]

Perl is for system administrators and system administrators-cum-developers, not real software development.

Baloney.

Contrary to what lots of people around here seem to think, especially those who like to make wide-sweeping declarations about what things are and what they are not, Perl is a tool. Nothing less, nothing more. Like all tools, it can be used to create well-engineered systems, and it can be used to create crap.

The community that grew up around Perl is all-welcoming and generally free of elitism. That's why a lot of newbie programmers and "system administrators-cum-developers" use Perl -- because they can without getting crapped on by others who think they know better. As a result, there is a lot of ametaurish-looking Perl code out there, but that's not a result of the language, that's a result of the all-inclusive set of people who use Perl.

Let's be clear: If you write code in any language and the code sucks, it's your fault, not the language's -- the language is just a tool.

Don't blame the problems of programmers on their tools.

Re:Perl is a *tool*

[markj02]

Let's be clear: If you write code in any language and the code sucks, it's your fault, not the language's -- the language is just a tool. Don't blame the problems of programmers on their tools.

There is no problem. Good programmers can produce good code in any language. However, even good programmers take longer to produce good code in some languages than in others. Perl is a language in which, in my experience, it takes a lot of time to create good code. Perl's strength is that producing quick one-of solutions is fast, which is also good, which isn't bad either.

Re:Perl is a *tool*

[derobert]

Use the correct tool for the job. Perl, for example, would not be my tool of choice for a compiler. INTERCAL would not be my tool of choice for anything except torture, and job security.

The important thing is to use the right tool for the job. If you use the wrong tool, the code will suck more than it otherwise would. And that, of course, is the programmers fault for choosing the wrong tool.

Learn from those who have succeeded at this...

[William Tanksley]

Some here are warning you that major changes always require a total rewrite; yet in real life, total rewrites result in inability to compete (look how long the Netscape rewrite paralysed Netscape, unable to meet Microsoft's challenge!). There's some good discussion of the danger of rewriting at a former MS software engineer's site, and some limited advice about how to get away without doing it.

But you've decided to rework rather than rewrite, you say, so I have no doubt you'll ignore the naysayers here. So what CAN you do? After all, as you recognise, reworking is dangerous!

The following rules have worked for me; I've refined my own experience with advice from Fowler's Refactoring, a book as useful as Design Patterns, and with study of Extreme Programming, a design methodology forged in the traditions of Smalltalk, and in the knowledge that maintainance, the most important and expensive part of software engineering, is also the least studied.

First, do the simplest thing that could possibly work. Don't EVER take your program out of commission for more than a day; make sure it runs at the end of each day. If you're doing something and at the end of the day your code base is broken, STRONGLY consider throwing away your changes and going back to the design stages.

Second, rely on unit tests extensively. Start every change by writing as extensive of a unit test as possible. Unit test every function you touch, BEFORE you touch it, and after. Unit test every change you make, and run the unit test BEFORE you make the change to ensure that it fails (i.e. it detects the change). Write your unit tests BEFORE you write code, whenever possible; you'll objectively know your code is done when your unit tests pass.

Third, don't design too far ahead; you don't know what tricks the old code is going to throw at you. Implement one feature at a time, bringing the code into compliance. Once everything has a unit test (thanks to your following the above principles), THEN you can safely embark on larger design changes -- and in the meantime, you have working code with new features, a win even if your customer/boss/manager decides not to continue.

Fourth, don't be afraid to redesign your own code. The stuff you wrote has more tests, so it's safer to change, but it's more likely than the old code to lack some critical understanding only age can give.

Fifth, use the principles of refactoring. Whenever possible split each code change into two parts: first, a part which changes the structure of the code without changing its function (and which therefore allows you to run the same unit tests); and second, a part which uses the new structure to perform a new function (thereby requiring new unit tests).

Good luck. If you want more advice, read up on Extreme Programming.

-Billy

Re:Learn from those who have succeeded at this...

[startled]

Much agreed. We've found, though, that sometimes it's tough to write as many tests as you really should. Before you even start changing code, write a pile of tests-- it's best if you can just allocate a week or three for it.

Beyond that, it's tough to stay motivated about writing tests; people want to write new code, not test the old. So mix it in-- we've decided to require a test to be checked in with every bug fix, figuring that if it broke once, it'll break again.

While I'm at it, I totally disagree that a total rewrite is always the wrong choice-- you just have to keep significant resources going ahead on the old stuff. If your old stuff really was that bad, the new will catch up with it without too much trouble. If it doesn't, well, you didn't really need the rewrite.

Re:Learn from those who have succeeded at this...

[William Tanksley]

Well said. It really, really helps, in my experience, to develop the habit of coding to the tests.

That is, you first write the tests; do the design while you're writing the tests (and have the tests express the design). Then write the code, trying constantly to run it against the tests. AS SOON as all your tests pass, commit the code; you're done. Don't refine anything until you have written a test which fails under the current code.

You see what happens? Pretty soon, instead of tests being your enemies which break your code, they're your friends which tell you when your job is done. You'll quickly find that they're reliable friends -- and long-lasting ones, too.

-Billy

Re:modularity/incremental rollout/unit tests/itera

[Tomster]

Me too!

Kidding aside, I wish I could mod the parent up to 6. Tim's right on. Probably the biggest mistake you can make in this kind of task is to bite off more than you can chew. And the more crud there is in the codebase, the easier it is to do! So take it slow. Refactoring is like orbital mechanics; you slow down to speed up :).

I recommend Martin Fowler's "Refactoring" book. It's geared towards object-oriented Java development, but the basic principles will apply across languages.

My company made the mistake of trying to rewrite our product from scratch. The "Tiger Team" effort got about halfway completed. Nobody knows if it would have actually resulted in a better product.

-Thomas

Re:modularity/incremental rollout/unit tests/itera

[Plisken]

The worst programmer habit I know of is copy-and-paste coding instead of using subroutines. You can tell people not to do it, but some always will. Those people should be bid farewell -- you can't afford their overhead

This drives me absolutely crazy. I understand it's tempting to copy-n-paste, considering the deadlines that many programmers are on(I've done it on occasion), but in the long run code readability and the ability to _easily_ modify code suffers.

For example, we have a binary file format for our configuration files. In order to check to see if a particular data type is a byte array, you have to mask off some bits and do a bitwise or. What if we change the format for byte array? Well, you have to go through umpteen config readers and change all the copy-n-pasting that has been done. Just a few weeks ago, I blew up and said that I wouldn't touch any new code that hadn't used the functions that I had written.

Perlmonks.org

[consumer]

You might want to reconsider perl as the language of choice for a large scale application. I realize I'm posting this comment to a Perl system, but Perl hangs together like an immense kludge of a language.

What a monstrous Christmas troll you are. What qualifies you to make this judgement? Perl, like any other mature language, has people who write kludges with it and people who write clean, elegant code with it. Your lousy Perl code is not indicative of a language problem.

That said, you're probably stuck with it, and AFAIK, you may be forging new paths in programming for reusability by applying the above concepts to Perl.

And this shows how much you know, since the Perl community is full of activity around design patterns, refactoring tools, unit-testing, and other practices which are in favor among experienced people trying to write solid, maintainable code.

My suggestion for those who are looking for actual useful advice rather than this kind of "throw away all your work and learn Java" crap, would be to head straight for http://perlmonks.org/ and read up. There's tons of advice there for serious Perl coders. You would also do well to start reading the mod_perl mailing list, which often has informative discussions about these issues.

Re:Perlmonks.org

[benedict]

I know some serious perl heads who say they
wouldn't suggest using it for a large project.

Re:Perlmonks.org

[Anonymous Coward]

What a monstrous Christmas troll you are. What qualifies you to make this judgement? Perl, like any other mature language, has people who write kludges with it and people who write clean, elegant code with it. Your lousy Perl code is not indicative of a language problem.

Perl has bad and good coders, right. The point is that *Perl* itself as a language is a kludge.

Very trivial example: Perl has not a clean argument passing. It's a kludge. Any professionnal language designer would be instantly fired for designing argument passing in a way so obviously wrong. Now repeat for 100s other small UNNECESSARY design blunders.

Re:Perlmonks.org

[Anonymous Coward]

The problem is that "clean, elegant code" (if you can find it) reads pretty much just like Visual Basic. In which case you could have saved yourself the trouble.

Minimise Untested Documentation!

[William Tanksley]

Depend on tests, not documents. Documents lie. Tests don't.

Use as little documentation as possible, BUT NO LESS. (In other words, for heaven's sake don't ever try to get away without any documentation.) Documentation should state fundamental premises -- things like "The customer wants X." and "This code checks that I'm fulfilling the customer's requirement of X." Documentation should not state intrinsic properties -- the statement "this code does X" should be made as a test, not a document.

-Billy

Re:Minimise Untested Documentation!

[Arandir]

I'll have to disagree. Document as much as possible, BUT NO MORE! But this documentation must be meaningful and relevant. Otherwise it is worse than useless.

Document every function, listing the purpose of every parameter and the meaning of the return value. Document why you are doing something if there is more than one way to do it. If a section of code fixes a bug, document what it does and do not just document a bug number. Use self-documenting code whenever possible (ei. name your variables and functions meaningfully). Use a document generation tool if possible (javadoc, doxygen, etc). Write the user docs at the same time you're writing the code. Incorporate the user docs into the code if at all possible.

Here is a bad comment: // store x2 in fu
Here is a good comment: // save the index because we will use it later
Here is a bad comment: // this is not meant for you to understand
Here is a good comment: // please see Smith's "The Black Magic of Filesystems" for details on this algorithm

The most important part of commenting is realizing who is going to read it. It may be you. But in all likelihood it will be someone you never met long after you have left the project or even the company. It may be code or design reviewers who don't know programming but do know how to block projects they can't understand. If it's Open Source, it may be some brilliant programmer wanting to fix a bug but without the time to puzzle over your constructs.

In every code review I have ever been in, someone has made some silly assumption about the code, with the final recommendation that that section of code be commented better to avoid future silly assumptions.

Re:Minimise Untested Documentation!

[Anonymous Coward]

Here is a good comment: // save the index because we will use it later

Or use a good name: indexForAlgoStep4 = index;

And in this case the programmer will be reminded by force not to change it, or not to confuse with another index.

In every code review I have ever been in, someone has made some silly assumption about the code, with the final recommendation that that section of code be commented better to avoid future silly assumptions

Right.

Re:Minimise Untested Documentation!

[Anonymous Coward]

Personally, I'd have to recommend about one in 1000 times you do the same boring old thing ("return x"), add something like:

// Now steal the poor sucker's soul

If someone's gonna be slaving over your code, it doesn't hurt to make them smile...

A Code Nazi Disagrees

[dmoen]

Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.

If you need to make big changes to the code, then you are already hosed. If you have no record of what bugs you fixed, and no way of testing if those bug fixes are still working after you make code changes, then keeping all of your spagetti in place is no guarantee that you won't re-introduce bugs later on.

The Code Nazi approach is to write a unit test for each bug you fix. And you have an easy way to re-run all of your unit tests. After you make a big change, you can test if all of your bug fixes are still working. And now you have the flexibility to refactor your code whenever you want. Which means you can keep your code base clean and elegant, and you'll never reach the crisis point that this group has reached. This is the approach advocated by Extreme Programming, as well as by other software disciplines.

Doug Moen

Re:A Code Nazi Disagrees

[xah]

Great. Now "Nazi" becomes a good word in some cases. Just wonderful.

Re:A Code Nazi Disagrees

[Anonymous Coward]

Great. Now "Nazi" becomes a good word in some cases. Just wonderful.

No soup for you!

-kwijibo

PerlDoc anyone?

[Bat_Masterson]

If you're going to refactor your Perl code, I would suggest pushing to have the code well documented with a tool like PerlDoc. One of the key problems that new people have with code is trying to understand the decisions that previous people made in the design of the code. Pushing more documentation into the code will help that in the long run. Plus, making documentation a standard part of your code will help make the code better as the developers will tend to throw out and/or rewrite code that they can't explain in the documentation.

Hardware isn't cheap

[drig]

I agree with your point and I'd like to expand on it.

Yeah, adding a gig of ram or increasing the CPU from 800Mhz to 1.8Ghz isn't expensive. But, if you go beyond what a singl-processor machine can handle, you run into another host of problems.

Adding a second CPU means *MUCH* higher chances of race conditions and other threading bugs. If you know you're coding for a single processor, you can often use a single-threaded model which makes life so much easier.

Adding clustering brings a whole hoist of data synchronization problems. It's *ALWAYS* easier to code for a single-machine than to code for a cluster. There are tools you can use to make shared memory easier, but those often flood the network.

Re:Hardware isn't cheap

[Arandir]

Yeah, adding a gig of ram or increasing the CPU from 800Mhz to 1.8Ghz isn't expensive.

Pardon me, I just sprayed Jolt Cola out my nose. Next time we need to upgrade 500 machines, I'll come to you for the cash.

Re:Hardware isn't cheap

[augustz]

Idiot.

If you have any reasonable kind of rotation for hardware (ie 25% per year) you would realize what everyone else has. Hardware has gotten enourmously cheaper, and the development time and money and investment you spend developing for that hardware now makes up a greater proportion of cost on a % basis then it has in the past.

So while you drip cola out your nose and try and clean your keyboard, remember: If you have 500 machines and can't support the cost of an upgrade as small as this guy is talking about, you are just another .com that overexpanded == .bomb

Re:Hardware isn't cheap

[Arandir]

If the machine is obsolete, I'll upgrade it. If it can't meet the demand that the clients put on it, I'll upgrade.

But I don't upgrade the machine just because I'm a lazy programmer.

Now on to the rant: I've been with the PC since day one. I learned how to program on a machine with only 8K memory. I remember when if you couldn't fit a program into 64K you were a wus. But over the years we have been taken over by lazy programmers. Their programs get larger and larger. Bloat is king. And the needs of a GUI is no excuse. I remember running a full Motif based GUI desktop on 640K (Geoworks).

And don't talk to me about speed. You're obsolete 800Mhz machine was state-of-the-art the last time you changed your underwear. You don't throw away last year's machine just because there's a new review at Tom's Hardware.

While we're busy upgrading our machines like there's no tomorrow, our landfills are filling up with perfectly functional machines, and leeching enough toxic heavy metals into the ground water to make Love Canal jealous. People like you are the vanguard of the throw-away society.

If you can't dish out webpages on a 800Mhz machine, it's time you looked for a career more suited to your abilities.

Re:Hardware isn't cheap

[augustz]

Yes... while I run my stuff on fewer machines that use a finer mfg processs (.13) and consume less power, you keep on running your 500 power hungry machines that consume enough power to light a small city, and require enough cooling to freeze a volcano.

I seen your "old time" racks and cabinates, and the amount of energy alone these things waste makes me shudder. I highly recommend folks walk through a datacenter and actually look at the 500 machines this guy is in love with, and realize what a fantastic waste it is.

The real cost is not the cost of the hardware, which is cheap as I have been saying. Look at your rack fees for *500* machines. Multiply that by 12 months/year * 5 year lifespan of the machines. What fits in a 4U 2 years ago fits in a 1U today.

Before talking about the careers others should be taking, I'd take a long hard long at the one you are following.

Those old computers can go to places that don't have to worry about the fact that something takes 4U instead of 1U, or 900W vs 100W. Things like your office, schools etc.

Listen, this is a pretty stupid conversation. EVERY single CIO you talk to at any level will tell you the cost of hardware relative to even the TCO is tiny. That has only gotten smaller recently. On a % basis to IT budgets it is smaller yet. I mean some of this is ancient news.

The original poster asked a question about throwing out his entire program and starting from scratch. I said hardware is cheap in context... So flame away, but you're looking the idiot.

Consider bringing in outside help

[Oroborus]

As someone who's been through what you're describing more often than I would have liked, I'll strongly advise you to bring in a consultant to oversee and direct the refactoring process.

The reasons for this are many.

You bring in a fresh viewpoint, someone who isn't married to any particular design decision or strategy. This is one of the biggest stumbling blocks in re-implementing existing code.

You bring in rationalized realism. A consultant to oversee such a process is able to make rational and reasoned projections about timeframes and milestones, and can be held to them. Without a voice of authority it will often end up being a battle of wills between you and your manager(s), each side having it's own ideas about what is, and isn't, possible. You might tell your manager something will take 2 months, and you may or may not be right, and he may or may not believe you. But if a percieved "expert" says it will take 2 months, the manager will have no choice but to agree, and he's likely right to boot.

You'll learn something. You really will. My first interactions with consultants tended to be dismissive, after all, who knew my code better than I? But what they have to teach isn't technical so much as it is organizational and structural. I learned a great deal about how to get things done, skills which served me very well as an employee and now as a consultant myself.

Inviting in an outside professional is a difficult thing to do, it requires admitting that you need help, and freeing up additional funds to pay for it. But in a situation such as yours it will not only pay for itself, but will make your job much easier and stress-free. The only real problem is the difficulty of finding a truly qualified consultant with experience and insight into your particular situation. But that's a topic for a different reply. ;)

Don't rewrite, and don't refactor

[splattertrousers]

Don't rewrite your code. That's a waste of time.

Don't spend a big chunk of time refactoring it either. Waste of time too.

Instead, make slight refactorings as you go. But make sure you are doing what you are really being paid for: implementing business value.

And you'll find that you'll have much more courage to refactor if you have a full set of automated tests, so maybe you should work on tests first.

Having done complete redesign of dynamic sites

[f00zbll]

I've rebuilt dynamic sites from scratch twice. First and foremost, if you're rewriting because of serious scalability or design weakneses, then it is unavoidable. If it's just to get rid of annoying things, then I would say don't even try it. I consulted at a fairly big E-Commerce site that was crawling and couldn't handle the traffic. The original site was built by a programmer who scaled examples provided by MS. After it was done the whole site was a dog and would crash constantly. They finally brought in a programmer who was able to rewrite parts of it and make it work. After 7 months of intensive work, they 2 people stabilized the site. They decided to completely rewrite the site and I was contracted to help.

In this particular case, it was necessary because the site was right at the max. If the traffic increased, it would kill the site. Since it was an E-Commerce site, rewriting it was fairly straight forward. The old code kept running, until we were able to finish the new system and make sure it was stable and ready.

As a consultant, one of the most important aspects is detailed documentation that explains both the high and low level details. Often I will include very specific details about why a design was chosen and what limitations it has. When applicable, I will also describe how to extend, or modify the code to support additional features. This means you spend a lot of time doing documentation, but it forces you to think about a design more thoroughly and will expose weaknesses. Always keep an open mind and never fall in love with your design. There is no right way to build something, only right for the situation you are given.

look forward

[djarum72]

You're probably planning on working solidly for a couple of weeks or months and coming out with the same product, but cleaner.

Well, consider all the (big or little) features you might like to add, and when you do your rewrite, consider HOW those features would work. That will improve your refactoring and also keep some fun in the project.

I think you should concentrate on improving your processes so that you never have to do this again. So, basically, catch up on all the unit tests and reviews you've never done!

Others talk about tests. That is a valuable practice, but keep it limited to things that are easy to test. If your tests are ten times flakier than the code because its really really hard to write a good test, well, fuckit!

I'd recommend a checkin-review process. Every should be reviewed for:
1. comments-missing-from-unclear-code,
2. violations-of-coding-standard,
3. corner-cases-not-covered.

Those things are pretty easy to discuss with developers, they wont be taken as criticism.

Checkin review REALLY DOES improve your code and it DOESN'T have to be onerous!

Don't break the #1 Cardinal Rule

[Pedrito]

Obviously you didn't read this slashdot article about Joel on Software.

What is the #1 thing he says causes software companies to fail? Rewriting from scratch!!!

I won't deny that there are times it has to be done. Joel points out some of those time (and yours isn't one of them even from the little you've written). Ours happened to fit everything he said, and we did rewrite, and not only was it the right thing to do, it was the only option for us, but the company just barely survived the process.

Don't take that lightly. I speak with experience and Joel is right. You don't rewrite from scratch unless there is no conceivable alternative. Joel describes it well, so I won't "rewrite" it. You can just click the link and read it yourself.

Re:Perl: another close-minded fundamentalist relig

[Anonymous Coward]

maybe it's because good programmers can write good perl code. just because you can't does not mean someone else can't.

Typical Perl denial: "you can write good perl program if you are not an idiot". Problem is that Perl a worse language than many others. So the point here, is that *you* could write (much) better code doing the same thing in another language.

Re:Perl: another close-minded fundamentalist relig

[Anonymous Coward]

it's not denial. and i don't disagree with you all that much. it's just i'm as sick of anti-perl people and i am on perl zealots.

Re:Perl: another close-minded fundamentalist relig

[Anonymous Coward]

maybe i should quit drinking and use preview.

Paradigm Caution

[Tablizer]

There is NO decent evidence that OOP factors better, or does *anything* else better than procedural/realational programming (or other non-OO paradigms).

IOW, don't switch paradigms just because something is in fashion right now.

Look carefully before leaping into something that may just turn out to be a fad or only shines in specific domains.

Re:Paradigm Caution

[jslag]

There is NO decent evidence that OOP factors better, or does *anything* else better than procedural/realational programming (or other non-OO paradigms).

On the contrary, there is plenty of anecdotal evidence that relentlessly hyping OO has the effect of getting Joe Average Programmer to start thinking a little about modularity and abstraction.

The best way to clean the stove ...

[cpparm]

is to move every other year. When you find yourself in the position where you need to rewrite a whole system, get a new job.

php project mngt app

[gol64738]

a good project management application is important for any development team. usually, these are hard to come by unless you plunk down $10,000 or more, although these come with a gazillion features that you probably won't end up using.

i discovered a new tool on sourceforge which is an open project written in php.

i'm impressed with it. the code is also well documented.
the homepage can be found here.
i recommend checking out the screenshots as well.

Perl vs. PHP ?

[Tablizer]

(* I would never use PHP for the kind of work we do. PHP just isn't the kind of powerful, flexible, complete environment that Perl is. *)

Could I possibly ask for a fairly realistic little example of something that Perl does well that PHP can't?

Re:Perl vs. PHP ?

[tmoertel]

Could I possibly ask for a fairly realistic little example of something that Perl does well that PHP can't?

Middleware.

For example, most enterprise applications do something important -- trade shares, manage accounts, track patient records, etc. How it's done is governed by business logic -- the company rules, policies, regulations, procedures, and so on. Now, you can spread this logic across your web site (as one might do using PHP, which is tied to the web site). Or you can bundle it up into an independent application, keeping all of the business logic in sensible, cohesive compartments that run on an application server (e.g., using one of the existing Perl app servers or one you've rolled yourself via, say, POE). This not only makes the business logic easier to understand and manage but also makes the logic independent of and accesssible from any number of front ends that you might need. Simultaneous web, client-server, and even command-line interfaces become possible, and for enterprise projects, multiple simultaneous interfaces is often a requirement for backward compatibility with older interfaces.

In summary, for shallow web-only stuff, PHP is a reasonable tool. For stuff beyond web work, PHP is out of its design envelope. However, Perl works here just as well as it does for web work.

99% Planning, 1% Coding

[freality]

When you first design and implement some module, a
lot of time is involved in cycling between "ok, I
know what to do" and "huh, maybe not". I've found
this crucial, esp. in team work, in order to gain
a good conception of the scope of the task. Also,
many external issues, e.g. how the module interacts
with the system, efficiency, etc. that aren't pure
functional issues, are first grappled with here.

Refactoring is different from this, in that you're
probably very comfortable with the "state of mind"
of the code. Instead of creating, you'll be
clarifying. So, most of the refactoring is in
your head (99%). All the external issues have been
addressed before (or else this probably isn't really
refactoring), so just work at a white board with
your team until writing the code will basically
be transcription (1%).

I've found this to yield the best code.

Quake 2 source code released

[MisterBlister]

Yeah yeah this is completely off topic to the current thread, but many developers may be interested in this and no Slashdot story yet.

The full Quake2 source has been released under the GPL, similar to the previous Doom and Quake1 releases.

Grab it here:

ftp://ftp.idsoftware.com/idstuff/source/quake2.z ip

Don't do it at all!

[Anonymous Coward]

Per my professional experience with an obsolete, out-of-date PASCAL on DOS system that keeps beeting the daylights out of would-be replacement systems, leave the convoluted, undocumented pile of spaghetti as-is. If your people have too much time and money, they could productively spend it figuring out what everything does and documenting it properly, but the mess that works is priceless compared to the alternative!

Re:Complete rewrite necessary or a waste of resour

[informer]

Certainly one of the keys to Microsoft products "working" is when they make well defined interfaces and well documented interfaces. In my experience, if you define your interfaces well and document exactly what they are supposed to do, no matter how terrible your code is, you can make it work, and make it flexible to later changes.

This is where a language like perl is not extremely conducive to well designed systems, however you can of course design a system well in perl.

- Adam

Joel Spolsky's Take...

[ijx]

Joel Spolsky has an article up on his blog site that speaks to this point.

He uses Netscape's decision to rewrite Netscape 6 from scratch as an example, and expands upon many of the points mentioned above.

Re:fp

[Anonymous Coward]

right, and that's why bill clinton was the first nigger president (minus the big dick part).

Retrofit? Redesign?

[Anonymous Coward]

Do it better! Do it faster! Do it cheaper!

Just don't forget to convert from inches to centimeters, and vice versa. (:

Nasa jokes aside, reconsider doing it in Perl. Perl, while a jewelled spectacle upon the crown of Language, is glue. It sticks things together. It fills in cracks and niches. But, like glue, if you just sit there with a handful of it, it's going to be sticky.

Before you touch *any* code, make sure you read the existing code. Go over it. For weeks, or even months. Understand why programmer X added a comment saying, /* Jeebus H. Christ, I quit! */ on top of an odd looking hack. Understand why things were done they way they were done, because, no matter how well your design, you too, will have to make clever hacks and changes upon encountering real world impasses.

Designs always look great on paper. It's when they hit the real world that they go to hell. :)

Why are you rewriting?

[Anonymous Coward]

You should never do something that will require a long time/resource investment unless you have a good reason. Is this just some clever way for management to justify firing the developers because they won't have accomplished anything at the end of this rewrite?

If you don't have a reason, you can try to find one. You might identify things that are truly broken about the system, and try to fix them. A typical problem with a lot of code is that its not as general as it could be (not portable, or you can only run one instance at a time because of some common resource or something like that.) By finding a concrete reason you will be able to direct your efforts and have a way of assessing your results when you are done.

Tablize It using macros

[bcaulf]

Well of course you are quite right: it is a catastrophe to hide tens or hundreds of cases in a nest of code. The logic of a program has to be visible to the maintainer and this means using a language that is appropriate to the problem at hand. A tabular language might be the right one for a particular problem.

It should be observed that there are unpleasant performance implications to doing things this way. For performance reasons it would be nice to have that stuff hard coded! But how do you resolve these two wishes?

You use a code generator that builds code (which is later compiled) using the tables. Now everyone is happy. Speed plus maintainability. What approach should be used for code generation?

You could build your own code generator by hand, spitting out program text by generating strings. But that is not very nice. There are many ways to screw that up both in terms of correctness and in terms of the power you provide to the programmer who uses the code generator.

And so we arrive at the point of this little screed: a macro system is the correct approach for creating a little language inside of a large program. This is one reason why programmers should care about the macro facilities of a language. Perl and Java have none built in. C has the unpleasant cpp. Lisps have lovely macros, making it absurdly easy to do tabular programming, complete with breaking out special cases easily.

Rewriting from scratch

[horza]

I have just carried out a similar project, and one thing I learned to beware of is don't suggest any timelines until you know *all* the functionality. You may think you've explored the site but there may be a lot of hidden functionality you don't know about, eg xml exports to affiliates.

I disagree with the "code doesn't get rusty" comment. Often there is a lot of pressure to get a project out the door and the initial launch version contains many compromises the designers didn't want to make. I don't think there is any project I could look back on and think "hey, the next time I would do it *this* way". People learn from other people's (or their own) mistakes, it's how we move forwards.

In response to the person that said don't rewrite as you will have to rediscover all the little quirks that the old site fixed: by doing a rewrite and documenting the little quirks as you find them means you will have a record of them rather than staring blankly at a screen of code wondering why on *earth* the previous coder did *that*. By rewriting from scratch you won't miss any quirks, which then come back and bite you inevitably at the least convenient time.

As for the Perl vs Java vs PHP debate, I'd say use the one you all know best. No language is any better or worse than any other. Some have libraries to make life more convenient, others have cleaner OO support, but in the end if you all stick to the accepted coding standards then half decent programmers can write maintainable code in any of the languages. Just bear in mind that your replacement will need to to know the same language so don't pick something obscure or dying where there will be no pool of available talent.

Phillip.

Consistancy is the key...

[CTalkobt]

The term "re-write" scared me when reading your query. In large scale systems, especially of the size you're talking about it's best to re-implement certain portions / areas that can be broken apart and handled independantly.

If you wish to convert into a differant environment then treat it as a seperate project from the re-write. If the conversion to a differant environment is not straight-forward ( I don't have a lot of experiance with CGI / Perl.. ) then write implementation wrappers for the missing / alternate functionality so it's merely a simplified change in the function naming rather than a re-implementation.

If you need to do both - handle each independantly. You'll thank yourself when an important requirement that marketing promised pops up when you're in the middle of it.

Seperate, Seperate, Seperate - There are no huge projects - only people who are too all-encompassing to break them down into achievable sub-goals.

Keep breaking them down until each is achievable in a months, 2 months time. None should go over 3 months even for the worst case. Ideal time-spans would be 1 month periods.

Reducing redundancy?

[Anonymous Coward]

You want to go through the code to help in 'reducing redundancy'?

Jeez, you lazy bum programmers. Do something useful instead of inventing new projects to keep your ass off of the welfare line.

Unnecessarily long variable names

[i_am_nitrogen]

I saw a quote once that said something to the effect of you can always tell a computer scientist's code because he won't have any comments: his variables are his comments (something like that, not sure of the exact words). While descriptive variable naming is important, excessive variable naming is annoying. For one thing, it's a lot easier to type no_acc_click() than it is MakeNotAccidentallyClickable(). Comments are necessary. When code does anything of even a slightly complex nature, there's no such thing as intuitive code. There will always be a need for comments in anything beyond the simplest programs. The best code is properly indented, case is consistent, variables have descriptive but not lenghty names (i is sufficient for an iteration variable, malloc is sufficient for a memory allocation function, etc.), a description at the beginning of the function of all expected uses and known bugs, and last but not least, comments of each change in the code, ChangeLog, and CVS logs.

Re:Unnecessarily long variable names

[Dwonis]

The best code is properly indented

I think this point needs clarifying: If a project is using one indentation style (this includes tabs vs spaces), for $DIETY's sake, use it!

Also, if you are selecting the coding style style yourself, use K&R style. I don't care if you think it's not 100% logical -- it's readable, and every programmer worth his/her stuff is familiar with it. The only deviation that is really tolerable is one particular hack on function declarations (i.e. some projects use the hack so the regexp /^funcname/ will work, and this is usually OK with most people).

Yes, I'm an asshole, but if you follow these rules, you'll find that your successors will curse your name much less.

Re:Unnecessarily long variable names

[i_am_nitrogen]

I think K&R style is pretty good. Personally I code to the Linux kernel standard, which is quite similar to K&R.

Speaking as one who is about to undergo a major rewrite of a minor program (actually, not even a program.. [shameless plug]Palace is my XMMS plugin), which was based on XPLSISNJASP, I can say I understand and appreciate all the comments shared in this discussion. If the XPLSISNJASP source code had been cleaner, non-leaky, commented, etc., it wouldn't have taken me a month to figure out how it works, just so I could pick up where the authors left off over a year ago. The code in releases up to 0.1.3 is in likely worse state, but 0.2.0 should be the "killer app" of parallel port light shows. I plan to incorporate all the applicable advice from this article. Anyone who's interested can e-mail me (my address is in the Palace readme) and see how my rewrite works, suggest features for the new Palace, and learn from my mistakes.

What a waste of space that post was...

Re:Unnecessarily long variable names

[Anonymous Brave Guy]

For one thing, it's a lot easier to type no_acc_click() than it is MakeNotAccidentallyClickable().

For every time you type a variable name, it will be read hundreds of times. MakeNotAccidentallyClickable() isn't a great name for a function, but its meaning is clear and unambiguous. It's infinitely better than the meaningless no_acc_click().

Comments are necessary.

Yes, but far less often if your other names are well thought out.

Re:Unnecessarily long variable names

[p3d0]

Here are my laws of naming:

1. Names should be as long as necessary to describe what a thing is.
2. The shorter a name is, the better.

The ability to satisfy these two rules simultaneously is a symptom of simple, clear abstractions.

Re:modularity/incremental rollout/unit tests/itera

[bcaulf]

Word to your mother. Code reuse via copy-n-paste is one of many basic software engineering blunders that is committed constantly in, I would guess, most software development organizations.

Ultimately it's a culture problem. The more junior software engineers in a lot of places just don't care about quality, so they don't pay attention and learn from each other and from more senior people. They don't criticize the flaws that they are no doubt observing in each other's work. A few senior staff can't do all the teaching and critiquing; it has to be an effort from all the developers.

At some point you have to blame management for creating perverse incentives that ignore or discourage high quality results from developers.

Ahhh Refactoring the Sport of Kings...

[j-jahnke]

I want to start by saying, before you do ANYTHING make it a priority for your team to start writing test cases and implementing them. If you are going to change your system you want your tests to find the problems not your users. Once you have a test suite that effectively tests your system (you will be surpised how quickly these tests will lead to enhancements in your software BTW) then you need to decide if you really want to maintain two seperate teams.

One team will be doing maintainence on your old system that is in use. You can't not keep this team in place, you have to keep customers happy while you do the new thang. You need the second team developing the new sytem (their job will be difficult with good tests, but impossible with out them.) Without tests they will have something that looks kinda like the old thing, but behave differently in a million small ways, some without a doubt will be better, many will not.

The other thing, and more sensible IMHO, is to tell pairs of programmars, to spend some hours each week working on the system together. Pairs becuase this is an excellent opportunity for people with deep knowledge of their systems to share it with other memebers of the team as they ferret out ways to refactor the code so more of it can be shared.

One day a week you can have a team working on some part, documenting spaghetti code, or pulling out examples of where code was cut and pasted and create a function which is called from both places. The stated goal should be fore each team to remove some number of lines of overall code from the project each week.

Remember the metric is the more lines of code you have the more errors you have, make it a mandate for your team to be removing code as well as adding, actually on our team we don't brag about how many lines we added in a week, we brag about how many we removed.
Anyway, the idea is by getting teams to do pair programming looking a specfic areas with good tests you can whip your code into shape pretty fast, and spread the knowledge of the system around as well. AND instead of a jarring change when you unveil the new system you can have gradual changes as fixes are tested and added.

Obviously if your overall architecture is broken and can't move forward this won't work. But it seems to me you are not fully convinced of that fact yet. At the very least by having teams look at code they may not be overaly famililar with you might find that team members can cross pollinate and share good ideas.

Jer,

It's called depreciation

[JohnsonWax]

There's a concept in the world of stuff you can touch called 'depreciation'. Why the software world hasn't caught onto this is beyond me, but from my experience it seems to apply reasonably well.

The value of your code goes down in value over time. Now, don't confuse that with the value of your design - your design could be grand, but the code is part of your equipment, like your hardware. Depreciate it over time to reflect the increasing cost of maintenance and integration costs in a migrating business.

Reworking your code allows you to make adjustments to your design to reflect a new environment, or to move away from languages/APIs/toolkits that might be hard to maintain.

I depreciate my code over about 3 years. It's all modular, and I replace code about as frequently as I add. A number of years ago, I had tons of bandwidth but not much CPU power, so I tended to push data rather than compute. The reverse is now true, so I made some design changes as part of a standard rewrite - no need to wait until it broke. Overall, most of my code hasn't radically changed in it's design, but it has been rewritten several times. I've had code cut back to 10% of it's original size by adopting a new toolkit, etc. I've made it more robust, faster, cleaner, better documented. I can't think of a case where it's gotten worse, and I can't think of a case where the rewrite took much longer to write than the original code - and more often than not took much less time. It's worked well enough that I've added a considerable amount of functionality, but spend no more time reworking code because it becomes increasingly efficient and is never too far removed from future additions.

Many people suggest that code rewrites are a waste of time, but it's a maintenance function. People that only budget time to write new code often find those extra work hours devoted to maintenance. Budget it in - and the best way is through rewrites.

Re:It's called depreciation

[rabidcow]

It's all modular

I think this is the most important thing. Design your code to be modular so you can rewrite bits without it breaking.

Massive retrofit?

[chaeron]

Why?

What is pushing you towards such a drastic move, and what do you hope to gain from the effort?

The answers to these questions might make answering Chizor's inquiry easier.

Re:Complete rewrite necessary or a waste of resour

[pturley]

I entirely agree with Joel's claim and I think the poster was correct to quote him. But notice that Joel doesn't say it's always a bad thing to rewrite your system - it's only bad to do so if your motivation arises from the specific reasons he mentions. IMHO, the original poster has more fundamental reasons for wanting to get rid of his current code base.

RYou can tell he works at microsoft

[kronsrepus]

If the orginal developers of the code (be it yourself with less knowledge than you have
now or someone else) wrote the programme in such a way that its absurbly bloated, poor designed and
all the rest of it then theres nothing wrong
with the 'start from scratch' mentaility. If the
people at microsoft decided to redo dos in the early days and get rid of that 640k limit, if they
redid a lot of there less successful product lines
(there is a lot of them, like ms publisher, etc)
then these products wouldn't be useless.

The entire prototype theory comes from starting
from scatch. Its important cycle in software development

Some advice:

[Anonymous Coward]

I have been through this before - with a different language, but the same principles apply. Here is some adivce, and some ideas:

1. Don't do a 'big bang' rewrite. A Big Bang rewrite is one where you totally rewrite all the code, but have no functional - or very little functional difference.

2. Do refactor or rework small portions of the code. Work on the code as you implement new functionality that the system needs. In other words don't waste time on rewriting working code unless it needs additional features or is broken.

3. Do Introduce Unit Tests and Acceptance Tests. This means you will know quickly when changes you make break code. Don't do a massive "unit test" building exercise however. Write a Unit Test for every peice of new code you do, and every peice of code that needs fixing.

4. Identify "Problem Code". Problem code is the bits which seem to take up most of the maintenance time. Identify that code, and try to work out ways to fix it.

Note : Some people think they are 'clever' by writing complex code which is hard to understand, but may perform better etc. This is the kind of code which usually ends up being a problem. As a professional programmer I try to be clear to understand - not only for the benefit of others, but for my benefit when trying to maintain it.

5. Read about how to manage projects. Extreme Programming is popular now, and I can say that it has some very good ideas. However don't become religious about it, and take a look at some of the 'classics'. I can recommend:

- The Mythical Man Month (tad dated, but a classic).
- Code Complete (also a little dated, but still good advice)
- Extreme Programming Explained

To take a look at my Open Source Resource, which has many articles around the subject of project management, take a look at:

Devcentre

I have to agree with the concensus...

[Codifex Maximus]

if it works... don't fix it.

If you feel you have to fix it, then prioritise the most problematic parts and fix them according to a set plan/policy. Use a naming and calling convention. Break functions that do more than one thing up into component functions that can be tested, verified and reused by other parts of your program. Fix it incrementally not all at once. Try using an interface contract when you make objects; that way, your new functions can call new methods and the old code can depend on old methods to be there. Deprecate the old methods when there is no code that depends on it. Don't forget to comment - comment the code then come back the next day and read your own comments. Make changes to the comments so they make sense today.

Blah... blah... blah... Etc... etc... etc...

I think this speaks for itself...

[markj02]

Not too long ago a link was posted to an interview with Joel Spolsky who used to work at Microsoft.

Yes, so if you want to write code like Microsoft, you follow their principles. However, unless you have the legal team and monopoly position to go with it, that strategy will probably not work as well for you as it does for Microsoft.

Deciding to completely rewrite your product from scratch, on the theory that all your code is messy and bug prone and is bloated and needs to be completely rethought and rebuild from ground zero.

Well, he is right to the extent that if you use the same tools and the same skills, you are going to end up with the same mess. Given that Microsoft has been using the same primitive tools for the last decade, it's not surprising that rewriting doesn't work for them.

It's not like code rusts if it's not used. The idea that new code is better than old is patently absurd.

Again, that's Microsoft-think. Sure, for Microsoft, things don't change unless they change them. For the rest of us, code does "rust", because the environment changes. And for the rest of us, there are new and useful tools and programming languages that we can use.

None of us know your situation

[LoveMe2Times]

There's a few things that I'll say. One, not a single person posting here really knows what your situation is (well, unless your coworkers read /. and are posting advice for you...). I don't want to be mean, but most of what's been posted here is likely not applicable to your situation. For starters, ignore all the threads arguing over languages: they'll never agree, and in the meantime you have work to get done. Next, I would suggest ignore all of the posts from people who aren't talking about web sites. There's a lot of methodology recommendations there, and it takes a long time to really understand *any* methodology. So if you don't already know it, it won't help for *this* project. If you're interested in that kind of stuff, read up on it; maybe it'll be useful on a future project.
Next, no matter what you decide, from your current position, there's substantial risk involved. If you don't have a good way of estimating the costs and benefits of your alternatives, then you will wind up shooting in the dark no matter what you choose. This isn't really unusual, but it can be stressful! Somebody in management has to decide if they want to play high stakes poker with this project. This will establish how many nice risk mitigating activities you can budget (like unit tests, code reviews, documentation, etc). One thing a lot of technology people have difficulty grasping (this isn't directed at anybody in particular), is that if your management decides on a high risk course of action, it's not WRONG because it's technically inferior. Now, if management doesn't understand that it's a high risk proposition, then there's a communication failure somewhere that will screw you no matter what gets chosen.
Ok, now that you've decided where you're going, I've got 3 pieces of useless advice (all advice is useless advice, btw...)

1) Know your priorities. This is the most important thing to getting ANY solution.
2) Understand your requirements(I don't care how you do this). This is the most important thing to getting (close to) a CORRECT solution.
3) Remember it's only a job. This can be very important for retaining sanity :)

Re: Unexploded bombs, never truer words...

[kuma]

when you inherit 70000 lines of code that do not work and the lead developer was logging only one system update per month but actually changing production code every few hours...

those pearls of wisdom will really hit you.

practical matters, bullet points and examples:

1) write code to be read

and if means going against the culture, fuck the culture of perl or c or whatever... believe in karma--got that urge to be clever, you'll get yours someday, trust in the dark side.

2) comment your readable code

but strip out old, boring comments (after they are petrified, before they are eroded by time)

3) put interface documentation in the interface

4) minimize documentation

face it, documentation is great for training new users and system roll-out, but it falls out of step fast and will be of minimal help to coders when the shit hits the fan.

if you cannot afford to have developers who think they should be managers sitting around keeping the documents ship-shape, or (miracle-of-miracles) have a writer on staff, your docs will be a joke.

LAMENESS FILTER AREA

** big fuck you to everyone who thinks shit like this is too cool... (declarations omitted)

$done:=False

Repeat

$position:=Position($3;$4->)
If ($position#0)
$2->:=Substring($4->;1;$position-1)
$4->:=Substring($4->;($position+1))
$done:=True
$0:=1
Else

RECEIVE PACKET($1;$receive;(32000-Length($4->)))
If (ok=0)
If (Length($4->)=0)
$2->:=""
$0:=0
Else
$2->:=$4->
$0:=1
End if
$done:=True
$4->:=""
Else
$4->:=$4->+$receive
End if
End if
Until ($done)

** refactoring can be fun, let everyone in on the party. very few people will know this syntax, but nobody should be mystified...

` spb20011109 standardize on vl_JRDC_position (kill vl_JRDC_Fposition)
` spb20011109 old cycles (2001) gone, remove $simplified_crossouts
` ... switches and vastly simplify logic of price line assembly
` spb20011112 add Crossout Mode and Format interface menus
` spb20011119 fix bug where $second_price not initialized
` ... and ensure no crossout when Just Reduced third price
` ... is blank (no sale beyond discount)
` spb20011127 add redtag format menu
` sbp20011127 update redtag logic to consider case of price
` ... modification pull number being reverted by offer or
` ... exporting option (via check box)
` spb20011128 simplification of sale price method resulted
` ... in stripping some strings from psuedo-price-line, so go
` ... back to promotion record for Just Reduced flag
` spb20011212 fix tagging issue with Just Reduced where
` ... Discount price does not get proper style sheet, remove
` ... placement of unnecessary tabs, and kill $space_char
` spb20011218 fix tab placement for Just Reduced (good
` ... for other types, but mistakenly put two tabs after
` ... full price and none after discount, oops)
C_BOOLEAN($discount_or_sale_exists)
C_BOOLEAN($discount_and_sale_exist)
C_BOOLEAN($crossout_required)
C_BOOLEAN($just_reduced_selection)
C_BOOLEAN($redtag_selection)
C_STRING(16;$crossout_mode)
C_BOOLEAN($crossout_under)
C_BOOLEAN($redtag_graphic_required)
$redtag_selection:=(Position("RTAG";[Promotion]A va ilable)>0)
$redtag_selection:=$redtag_selection&(Not(Offer_ Override_Price_Mod))
$just_reduced_selection:=(Position("JRDC";[Promo ti on]Available)>0)
$discount_or_sale_exists:=(vp_Priceline_Second_P ri ce#"")|(vp_Priceline_Third_Price#"")
$discount_and_sale_exist:=(vp_Priceline_Second_P ri ce#"")&(vp_Priceline_Third_Price#"")
$crossout_mode:=vDigs_Crossout_Mode{vDigs_Crosso ut _Mode}
$crossout_required:=(($crossout_mode="Redtags Only")&$redtag_selection)
$crossout_required:=$crossout_required|($crossou t_mode="All Selections")
$crossout_required:=$crossout_required|($just_re duced_selection)
$crossout_required:=$crossout_required&$discount _or_sale_exists
$crossout_under:=(vDigs_Crossout_Format{vDigs_Cr os sout_Format}="Under Pricing")

Reminds me of something...

[jadavis]

Hmmm... reminded me of an article about code rewriting that slashdot posted a while back.

Slashdot Story: How To Make Software Projects Fail

Relevant interview link

Re:modularity/incremental rollout/unit tests/itera

[Fubari]

You have been getting some really good advice from other people here. You have a huge amount of existing code. How do you choose the first thing to handle?

Let your next feature show you where.

The project I'm on now is doing exactly the same thing you are contemplating.

We are letting new features drive the cleanup. When it is time to add something else, change the existing system to make it easy to add the new piece. Maybe this means touching a substantial number of modules to tweak them to use a new helper function. Fine, you're not changing routines for the fun of it - you have a definate goal in mind. Maybe everyone needs to use a new html formatter. Maybe everyone needs to use a common databse interface. Whatever. When consolidate something, really consolidate - hunt down every thing that is like it and adapt the existing code to use your consolidated routine.

I would strongly encourage you to do it this way, because if you embark to the Great Cleanup it might be a long time (many many months) before you can put new business value on the table.

If you do it this way, it can become part of your team's culture - you should always stop to clean up code every now and then. It is an ongoing process, and it gets easier the more you do it (kind of like laundry or dishes... why wait until life is unbearable to clean up a little?).

Letting features drive your clean up does the following for you:

helps keep you focused (reduces the chance you'll get lost in the woods and wonder around forever).

cleans up areas of the system that are likely to change again soon

finds the most important code

makes it easier to change next time

make it easy for you management to have faith in you.

This last one is really important. If they get regular improvements to the system, they stop worrying/fearing what you are doing and start fighting to shield you so you can keep doing it. At least this matters in a for-profit environment. Make it easy for you manager to believe in you and you will make your jobs easier.

And scope it small - small bites, easy steps, and release often.

And buy the Fowler Refactoring book, and the DesignPatterns book wouldn't hurt... but that helps C++ and Java people cope with brittle type systems... this isn't such a problem for you in Perl. You would benefit more from the principles in the Refactoring book, even though it is in Java.

my suggestions

[medcalf]

1. review all of your existing code and list the things that each script does

2. decide how to consolidate and divide out these tasks, and make sure that any functions or data that needs to be shared, or may in the future need to be shared, is separated out into modules

3. document all of the programs to be written, their interfaces and their effects

4. then, and only then, begin coding

-jeff

Having attempted this before...

[druxton]

The most important factor is having a ruthless project director. You don't accept slippage, you don't allow mistakes.

However, a good project director understands the project and the time it will take, and doesn't curry favour with management by suggesting impossible time-lines in the first place.

Beware the project director who comes in in mid-stream.

Duncan (not the MacBeth one)

some friends

[jslag]

I know some serious perl heads who say they
wouldn't suggest using it for a large project.

And I know some serious NT system administrators who ftp from arbitrary terminals to their production servers, using their Administrator accounts.

Re:some friends

[benedict]

When I said "serious perl heads", I didn't mean
"drunks I met on the street". I meant "people
involved with the perl design process".

Cheap Hardware != Incremental Replacement

[Anonymous Coward]

I disagree, many places do not replace 25% of their hardware in a year. For large installations (e.g. major retailer's point of sale terminals for example) a complete phased roll out replacing all infrastructure in one year every few years is more likely. You might be able to push a software upgrade out in such situation, but getting hardware upgrades (no matter how cheap) is way harder.

Rewrite!!

[Scott Bender]

Sounds to me like someone did a bad job of designing your system. Although I obviously don't know the details of your project, you may want to consider starting over. If you need to do a retrofit, then some bad mistakes were made at the start of this project. Doing any kind of retrofit can only make things worse. If you have time to do a retrofit, you'd be better off taking what you've learned from this failed project and doing it right this time. I'd start with losing perl. I wouldn't touch it with a thousand foot pole for a large project. Then go back to the design phase, and do it right!

Java

[Anonymous Coward]

If you are having concerns about maintaining 1MB worth of perl, and you are thinking about re-writing anyways, you should think about Java. Java's OO design lends itself to code reuse and is very easy manage in a large system.

WebObjects, JSP, and a few others are something to think about...

Re:Java

[Anonymous Coward]

And watch you site get really fucking slow!

real world vs cs101

[thogard]

One example is one client wants a column in a report added up. fine. no problem. It was desigened to do that. Problem is that column is user supplied and not numeric. So the rule is something like an orange counts one and an apple counts 2 if its one type and three if its another.
Your saying you can plan for this kind of crap and design it in? This is a one off and perl makes that easy. In fact the orginal code spawns a 'report' and if checks in ~luser/bin/final_report and runs that if its needed.

You can never underestimate (or plan for) the lame request that some paying customers will want.

Re:real world vs cs101

[DrSkwid]

buy you can plan for the fact that paying customers will make lame requests

Re:real world vs cs101

[Anonymous Coward]

How about doing some product marketing and requirements gathering? I feel sorry for you dude. It sounds like your company did not take the time or give you the time to explore the possibilities.

I've seen two large services unable to scale their operations in either function or capacity because they failed to do the analysis necessary to create a high-throughput system with extensibility considered as a primary design point. Instead, both took the approach of using Perl scripts for their back ends because "the real world demands one-offs".

One of the systems choked on the 4th customer. It took the usual one-month of hacking to produce a set of one-off Perl scripts modified from bits and pieces of the scripts used for the other customers. It then took another month for the inefficient system of Perl scripts to process a quarter of the data the customer wanted loaded into the system. It was a disaster. All the while, the chief "architect" of the system was trying to hide just how screwed up the scripts were by hacking over them at night and restarting processes without telling people. This "architect" and major proponent of XP at the company had, over the course of one year, created a convoluted mess of scripts by doing incremental changes and one-offs. What seemed to work for the first couple of customers -- and made management extremely happy because of the quick and flexible way the system could be modified -- wound up killing the system.

The problem was serious enough that the Board of Directors considered shutting down the company. It turned out a little calculation showed it would take 1 to 4 weeks of hacking Perl scripts for each new prospective customer. And then, acquisition of the customer was not certain because of the poor performance. The founder/CEO of the company seriously said it was okay to require 1 man month of engineering time to bring up a prospective customer. He planned on hiring a staff of 50 programmers to hack those Perl scripts in an attempt to ramp up sales. This was pure dot-bomb lunacy.

To fix the mess, the company fired both the CEO and the "architect", who to the moment he was walking out the door was still swearing that Perl and XP were the only ways to respond to customer demands in the real world. The company then spent a month doing requirements gathering and analysis. It scrapped the old system and designed a new framework based system that meets the needs of performance, flexibility and extensibility. It took probably two months for a group of 5 experienced engineers to architect, design, document, and iterate the design through reviews. It was painful for management, but in the end the system was fixed and the company was not shutdown. However, due to the inability of the company to do business during that time, the company had to be sold. The only reason the company was purchased was due to the capability of the new system.

Oddly enough, that company was purchased by another company struggling to replace its mish-mash of Perl scripts in its back end. The acquirer had a similar system and made the same mistake of basing their entire back end on Perl because the real world demanded the flexibility. At least that company had scaled a little better and gotten up to 100 customers. The problem is they cannot scale further and no one really knows what all those 100 one-off Perl scripts do.

Re:Coding standards are important

[Nyh]

But de important part isn't the indentation style. The important is that it describes the prefered constructions to solve certaint types of problems. It describes the rules for a good, uniform implementation.

And DO NOT make it a book of rules. For every rule there has to be argumentation why it has to be this way, including examples and they main advantages of the rule (improves robustness, maintainability, clarity, portability, uniformity).

The coding standard will help new and unexperienced to build good solid code. The professionals will like it because after reading the coding standard they know what is kind of code is expected from them.

And, maybe most important, will keep the hackers under control. Hackers won't adhere to rules just because they are the rules. But if they know why the rules are as they are they will use them.

I was a hacker who was coding only his way too, I didn't like rules because they made no sense to me. Right until I read Patrick Savalle's (Synalyse (sorry guys, dutch only)) C coding standard, that one made sense.

I am still using it, both for private and professional work (on most jobs there isn't a coding standard, the one I bring is very welcome and if there is an coding standard it is mostley only about identation so those are replaced just as easy).

Just my EUR 0.02, Hans

Re:Quake III Arena reveals ANOTHER linux security

[Anonymous Coward]

So, where is your implementation of memset()? Clearly you're a master programmer who can only humble us.

PS: You have the gayest username Slashdot has ever seen. Do you suck little boys dicks and feel ashamed afterwards? Go on, just admit it.

comment unneeded?

[volpe]

...and name the function so that the comment is unneeded


The only function whose name makes the comment unneeded is "exit()".

Re:comment unneeded?

[William Tanksley]

The only function whose name makes the comment unneeded is "exit()".

That's a stupid universal; I've already shown one example, which therefore disproves it. (exit() sometimes still needs a clear name, anyhow -- you may need to make it clear why you're exiting.)

But is there a form of your statement which has value? I say NO. Comments are only needed when clear code is impossible. Clear code is ALWAYS better. Sometimes it's unattainable, in which case comments are the only thing that can be done.

-Billy

Dont forget rquirements

[Anonymous Coward]

It's all well and good to discuss languages/technologies/methodologies but, at the end of the day, you have to ask if you are delivering on requirements,

From my experience, if the thing does what the customer (ever)wants - leave it be.If it's likely to change within 6 to 12 months - rethink it.

In order to continue delivering what the customer wants, you have to make sure you can continue to make it - and that means ensuring new hands understand exactly, not only what is wanted, but what has already been done.

can you:

a: be sure you understand the requirements?

b: communicate said requirements and existing deliverables?
c: ensure you have or can obtain resources that can deliver on the customers requirements?

You didn't read his pages did you...

[alfadir]

"my first software released under GPL (November 1988) -- now the Slashdot kiddies have to shut up; I was writing GPLed software before they were born, bwa ha ha!"

Posted Thursday, December 13, 2001 on his site..

He has alot of good thing to say about software management and programming.. read and learn instead of getting caught in Microsoft bashing..

Re:You didn't read his pages did you...

[markj02]

He has alot of good thing to say about software management and programming..

I disagree.

read and learn instead of getting caught in Microsoft bashing..

Instead of repeating what other people tell you, turn on your brain and think for yourself. Most of the programming and management advice that self-proclaimed software experts give you is junk.

"my first software released under GPL (November 1988) -- now the Slashdot kiddies have to shut up; I was writing GPLed software before they were born, bwa ha ha!"

That makes him pretty young and inexperienced in my book.

My analogous situation

[Chris Johnson]

I'm currently doing something similar- the program I maintain is Mastering Tools, which has long been an over-featured, densely packed program operating on a text input basis with a certain amount of visual feedback on the text input.

However, I've long known that there are two things my real users (ideal users? the serious mastering engineers) want: familiar interface and realtime processing. I can't deliver realtime processing without literally doing the whole thing over again in a language I don't know (it's done in REALbasic, which is little better than a scripting language for speed, though it's got really, really nice prototyping abilities and GUI support.

However, the time's come to completely overhaul the interface, partly because I have some ideas for mid/side processing and don't have any room in the current layout to fit them! The ideas have to do with rectifying the side channel and using it to either enhance or remove signals that aren't in both channels equally- where regular mono is a 'node' that totally eliminates out-of phase content, it's also possible to completely eliminate R and L-only content and keep only R+L and also the out-of-phase content.

That part's the easy part- it's simply signal processing (and will be duly released under the GPL as soon as it's done, as always). However, the interface is asking for a total, complete overhaul in several ways, and that's what's taking all my effort currently. Here's the situation and how I'm handling it...

Layering. It's no longer possible to fit the whole app in a 640x480 area, even with small print. There are several possible answers to this: one would be having separate windows. This would be more adaptable to larger screens, but it's untidy and there are issues with closing windows and still referring to controls they contain- so that's out. What's looking more reasonable is tabbed panels- RB implements a nice little drag and droppable tabbed panel control that appears quite easy (though I had some trouble attempting to do nested tabbed panels- after one experience of having all the nested panels (at identical coordinates) switch to the top panel of their parent panel, I quickly gave up on that concept. Instead I'm using more panel real estate and trying to divide the controls into logical categories. That is, of course, a real headache- doing interface properly is hard! (says 'interface is hard Barbie') It's only somewhat easier with the additional space. Complicating matters further, is the expectation of the intended audience here. It has to both be organized and look and feel like a mixing board, amplifier or rackmount box of some sort.

Solution: implementing controls like knobs and meters. It's actually quite fun to code a knob appearance out of graphics primitives- and surprisingly hard to get mouse gestures to work on the damn thing- using a two-arg arctangent routine in RB that I don't fully understand. I also have a single meter control already implemented for azimuth display- think that it might be best to tear that apart and re-implement it in a more general sense.

That's because the Knob class turned out to be the right thing to do- it's implemented almost totally separate from the main body of the program, as if it were a RB control or something. Knowing I was going to be using it in different sizes, possibly different colors etc, I wrote the knob code as completely scalable- from maybe 20 pixels to over 100. It reads its size from the control width and height, runs itself as far as handling mouse input and storing a control value, and does not inherit comparable interfaces to the controls it will be replacing- so when it's time to plug 'em in I can run the program and see what routines crash and burn (and need to be rewritten for the new control interface). I'm thinking meters need to also be handled in the same way- somehow- not clear on the form yet.

So: dunno what else to tell you, but it seems like the things I'm doing that are helpful are: compartmentalize new interfaces, make them adaptable and get them working independently of the existing code, while leaving the existing code in working condition. Then when the new stuff is brought online do it in such a way that you could do it one control at a time or in small batches.

Dunno if that's relevant to where you're coming from- but it's what I've found necessary when facing a major re-implementation.

My solid advice.

[mindstrm]

First, decide WHY you want to re-write all of it.
Second, document, in detail, what it is you want to accomplish (what should each script do, etc).
Thirdly, write up a project plan, IN DETAIL, so you have something to work from
Lastly, actually translate that into code. Do not start writing code until you have the modules fully documented.

Programmer buy-in

[arabbit]

There are lots of good reasons to refactor, but "consistency and reducing redundancy" doesn't seem very compelling. Anyway.

A lot of folks have talked about process and tools, and those are important and I assume you know what you're doing. The real issue is morale. Let's face it, at this point you're requirements are probably stable and your team is familiar with the system and everyone's probably been thinking about how they might do differently in the back of their head. And code is easy to write.

If you have people on your team see what they've done and know they can do better and want to give it a try, things will go smoothly. If you have people on your team who regard this as a needless rewite, on the other hand, morale will suffer and production will be surprisingly low.

If you have some of both, take the people who want to rewrite it and do it from scratch in another branch. If you don't have buy-in from all of the developers and try to fold changes into the original product and it doesn't go smoothly, company politics will turn against you and you'll probably be looking for a new job.

My advice: Start Entirely Over

[Myoot]

Yes, it'll probably be much easier if you just hash out a set of rules and style guidelines and then rewrite everything from scratch, to those guidelines.

You're going to end up rewriting everything along the way anyway, if you're doing a coding-style-level rewrite rather than just a bugfix, and it'll be a lot easier if you start from scratch.

Josh

Perl.... coding style???

[Glabrezu]

bwaaaahahahaha!!!

Even if its probably possible, its a bit hard to define a coding style for a language whose motto is "There's more than one way to do it". Probably the amount of things you should define to have code that looks similar its BIG (where should you use fors instead of maps? should i use $_ or assign it to something else? etc., etc., etc.)...

Its easy to define *your* coding style, but imposse it to your fellow coders might be harder (always speaking about perl). Anyway, you might like to read the "Common practices" chapter in Programming in Perl where Larry gives some good advices about coding style in perl....

My suggestion, if you think the code might start to get bigger... start slowly to switch to another language, 30K lines of perl code just gives me the creeps...

finding and keeping a competant staff

[KyleCordes]

[There's only so much a methodology can bring to the table -- after that it's about finding and keeping a competant staff who understand the value of what they're doing.]

You are so utterly right. The key idea is "people over process". I've rather have 5 great people with any old process, than 25 mediocre people with the perfect process.

[look at the console gaming industry for tips]

I have always been impressed with the gaming industry for this reason. Sometime people erroneously think of game programming as being somehow less "heavy duty" than business apps, etc. But games are usually under incredible schedule pressure (for vital marketing reasons) yet also need to ship very few bugs, because buyers are prone to return a game that doesn't work, not become a source of upgrade revenues.

Re:finding and keeping a competant staff

[Anonymous Coward]

On the other hand, few console games are forced to run in a constantly shifting soup of third party software, OS updates, etc. etc.

This is not to dismiss the stellar work that goes into a console game. But Square doesn't have to worry about the Playstation having the software environment change underneath its games after release.

It's a different set of challenges.

Re:finding and keeping a competant staff

[Anonymous Brave Guy]

The key idea is "people over process". I've rather have 5 great people with any old process, than 25 mediocre people with the perfect process.

Absolutely. 5 great people will look at the process they've started with, and if it's ****, they'll throw it out, come up with a new one that works, and then after the five minutes that took, they'll get on with the job.

Analyze first!

[Flu]

I am not a perl programmer (just C and Java), but these are the steps I try to take whenever I need to rearrange my code:

First of all; make an analyzis of the code; Is it modular? What dependencies are there within the code? Was it thought to be modular from the start, and in that case, has the modules started to deteriate? Are there functions/variables that are just positioned at the wrong places? Functions/variables only used by one or two other functions? Functions/variables used by way too many functions?

Based on those questions; start moveing functions and variables between modules. Consider that a module may be more than a single file, but probably not more than 10 files. Try to keep the number of functions per file to about 10 or less. Also, try to keep the number of files per module to 10 or less. Since you have more than 100 files, try grouping them in at least two blocks. Why not try to use the model-view-control or (interface-control-entity if you're into UML) model?

All this can be done without endangering the code, since you do not rename anything; and thus, you do not risk to suddenly start using some other variable/function than you intended. But, remember not to change more than one thing at at time without testing!

Also, if the language can't help you protecting (file-, module- or block-) private functions/variables, use comments to mark them as non-accesible from functions not in that file, module or block.

Now that the code is logically arranged, it is easy to see if there are functions that perfom a similar thing, since they (probably) would have ended up in the same module - or even the same file.

At this stage, you might want to actually start re-writing one or several functions in order to get rid of those functions that does almost the same thing. Also, you might want to re-name some functions to better represent their actual use.

However, if you do that, first think if you want to enforce any coding-standard starting with new functions? If so, create a coding-standard that most closely matches the code written so far, thus only a smaller portion of the code will fail to comply with the standard.

/Fredrik

Consider keeping programmers too.

[guitarrista]

What about personnel? That strange looking code that embodies solutions to real-world problems had to be figured out by SOMEBODY. Such people, as much as working code, are a valuable resource that should not be abandoned lightly.

Additionally, there are difficulties adding new people to a team. Brooks' book "The Mythical Man Month" describes some of the pitfalls.

Re:Minimize Untested Documentation!

[wik]

If you are going to use descriptive names (which I think is a fine idea), particularly like the one you mentioned, make sure the steps of the algorithm are clearly denoted. When one programmer sees an algorithm (or a maintainer looks it up in a book), they may be looking at the same algorithm, broken into different steps. There is nothing more frustrating than seeing the right thing (if you look in book A), labeled in a completely different way (because you learned it from book B).

I think one of the biggest problems is that people believe that because they named something clearly, it must automatically be clear and logical to others. More generally than just naming steps of an algorithm, this is a problem with naming commands, functions, variables, etc. I think this paper makes a strong case http://citeseer.nj.nec.com/furnas87vocabulary.html for the naming problem (yes, this is even a problem with experts in a particular field). Names are great, they don't always stand on their own. I'd highly suggest that people read the short section on "armchair" naming. I haven't seen a programmer who wasn't tempted to use this at one point or another.

Don't make the same mistake twice

[dgroskind]

The main reason for redevelopment is when the underlying data structures change and the legacy code won't work with the new data base product or schema.

Typical reasons for changing the data base are:

the platform is obsolete, e.g, the vendor abandoned the product or went out of business; it doesn't run on a new operating system.

some critical feature is only available on a new platform

the original data schema is inadquate or poorly designed

In general, if the underlying data design remains the same, then you don't need to do a massive redevelopment.

Complete redevelopment to make maintenance easier, that is, for consistency and reducing redundancy, is rarely cost-effective. It's better to begin adding new features in a more robust or easier-to-maintain platform and convert legacy systems as time permits according to some set of priorities.

If you do redevelop, make note of why and make sure you don't make the same mistake again. Often a poorly designed system is the result of some business requirement or limited resources. If those pressures remain, the new system will have the same problems as the old one.

K&R style

[Anonymous Brave Guy]

Also, if you are selecting the coding style style yourself, use K&R style. I don't care if you think it's not 100% logical -- it's readable, and every programmer worth his/her stuff is familiar with it.

K&R style was developed for presenting C code in a compact format for publication. It has popularity going for it, but there are better styles available for everyday code, some at least as common. K&R style becomes hard to read far more often in C++, where you have things like try-catch blocks around, in-line function definitions in class declarations, and so on. As always, of course, it's more important to choose a consistent style and stick to it anyway.

Screw Comments DO DESIGN DOCS

[Anonymous Coward]

For christs sake do design docs so the person you're designing the system for can understand the logic. Otherwise you will end up programming what he said he wants instead of what he thought he said he wanted. Big fucking problems.
Now if I could only convince my boss designing before coding is worthwhile.

Re:Screw Comments DO DESIGN DOCS

[William Tanksley]

Design documentation is highly overrated (after all, it's a duplication of the code itself, which is the most detailed possible design document). Requirement documentation is essential. I suspect that this is what you're asking for, anyhow, since you're talking about showing it to the person who asked for it.

I think that requirement documentation is the only universal documentation requirement. Everything else can be ignored in some (special) situations.

-Billy

Re:Screw Comments DO DESIGN DOCS

[Anonymous Brave Guy]

Design documentation is highly overrated (after all, it's a duplication of the code itself, which is the most detailed possible design document).

Not if it's done properly, it's not. Attempting to duplicate your code is actually damaging, because your docs will become out of date and misleading. However, maintaining a 10 page summary of the basic foundations of your system is far more helpful when bringing a new team member up to speed than sending them away to look at your "most detailed possible" 500,000 lines of existing code.

Re:Screw Comments DO DESIGN DOCS

[William Tanksley]

And having him look at the requirements, and then work on the unit tests, and finally pair with a more experienced programmer, is better still -- and this way there's no extra paperwork to maintain.

-Billy

Re:Screw Comments DO DESIGN DOCS

[Anonymous Brave Guy]

Do you have any evidence to support that, or is it just a little gratuitous XP evangelism? ;-)

But seriously, on a large project, you can't always afford a sufficiently experienced programmer to work as a pair with all the newbies. For example, on the project I'm working on at the moment, there are only two people with anything like complete knowledge of the project. This is a serious liability, and we're trying to spread the knowledge out to the rest of the team. However, there are several more than two junior guys. It's far more efficient for those who know to document the basic knowledge as a starting point, than it is to try to convey it to each junior team member directly during day-to-day development.

Re:Screw Comments DO DESIGN DOCS

[William Tanksley]

Do you have any evidence to support that, or is it just a little gratuitous XP evangelism? ;-)

The second. ;-)

I seriously do recognise some need for "get up to speed" documentation; it's just that it's almost impossible to write enough of it, and not write far too much. Of course, if you're doing pair programming regularly you won't need any (and you will have ALL team members knowledgable about the code in general), but if you can't do that (and politically it's almost impossible) you'll HAVE to write that getting-started documentation.

A pity that almost nobody will read it; the only exception will be the people so concientious that they hardly need it. Of course, you could be the exceptional project in which the people who know the code are also talented writers. But I doubt it.

-Billy

Re:Screw Comments DO DESIGN DOCS

[Anonymous Brave Guy]

I seriously do recognise some need for "get up to speed" documentation; it's just that it's almost impossible to write enough of it, and not write far too much.

Sad, but true. :-)

A pity that almost nobody will read it; the only exception will be the people so concientious that they hardly need it. Of course, you could be the exceptional project in which the people who know the code are also talented writers. But I doubt it.

Probably not exceptional at all, actually. I'm just noticing that around 18 months ago, as I came onto this project, I started a steep uphill learning curve to get up to speed. So far, about four people have made it to anywhere near the top in seven years of the project's history, not counting the guys who designed it all in the first place. These are the only people who can do serious maintenance or development on a project critical to the company. Everyone else on the project does their bit, but there is a certain point when they can't do any more to help.

Now, a year and a half later, I'm responsible for training up the next guy to join the team. He's enthusiastic, and has a great attitude, and you can work with someone like that. But he has the same problem I did back then, and I have the same one my own team leader had back then: there's no basic docs to teach him the ropes, so I have to spend large amounts of my valuable time -- like my team leader before me -- training him up instead. He may or may not ever grok the really key stuff at the heart of the system, but I hope that he will. However, if someone would authorise the time for just one person who knew the stuff to document the basics, it would literally save weeks of effort in future on the part of senior team members, particularly in training up those who are only ever going to use the basics.

And then I woke up. :-)

console games, PC games

[KyleCordes]

That's a very good point about console games.

I hadn't thought of that, I was thinking of games that run on a PC. PC games get the "best of both worlds", so to speak.

Re:modularity/incremental rollout/unit tests/itera

[Anonymous Coward]

Excellent points.

I agree completely. Make the code more robust as you maintain it and add features instead of trying to rewrite it.

I'm currently working on a palm application written in C. I took the code over from a programmer that practiced the copyNpaste method of programming. What is worse is that the programmer didn't understand to separate the data from the screens.

Stupid things such as
if (getCheckbox(checkboxptr) == true)
checkboxvalue = true;
else
checkboxvalue = false;

Even worse is that the code was designed to write the data to the file between screens. So without tapping on "Save", the application would be updating data. When the user tapped on "Cancel", the application would have to go UNDO all the writes.

I've been using the Big Ball Of Mud theory of code redesign. Take the big ball of mud and fix it over time.

Fighting redundancy by repeating yourself?

[Zero__Kelvin]

"My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code) in the interest of consistency and reducing redundancy."

Do you mean that you have consistently repeated the act of re-writing your code in an effort to reduce redundancy? Or perhaps you are seeking to reduce redundancy by consistently repeating yourself? 8^}

Inner loop?

[yerricde]

The first answer is that when more than one line is covered by a comment, those lines form a functional group and MUST be a function of their own.

When you scream "must," you seem to claim that "there are no exceptions." What if this new function needs to use a lot of a function's local variables without passing them as parameters (expensive for stack memory bandwidth, especially in the inner loop)?

There is some info that can't be conveyed by code; it's called "requirements". That info can't be conveyed by comments, though, either.

However, you can refer to and briefly quote the use cases in the comments, to give future maintainers a bit of context.

Re:Inner loop?

[William Tanksley]

When you scream "must," you seem to claim that "there are no exceptions." What if this new function needs to use a lot of a function's local variables without passing them as parameters (expensive for stack memory bandwidth, especially in the inner loop)?

Optimization /is/ a special case, I admit. For optimised code, expect readability to go out the window, in general. You have to give up something. Thus, you use comments rather than proper coding. However, keep in mind that micro-optimizations like that are dwarfed by better algorithms -- and sometimes (rarely) they're even trumped by a decent compiler. So keep those comments ready, but TRY not to use them.

However, in the general case, there's an alternative you aren't considering. Rather than fiddling with the stack or with messy hand-optimization, put the variables and the two methods into a class of their own. It's a standard, documented refactoring.

There is some info that can't be conveyed by code; it's called "requirements". That info can't be conveyed by comments, though, either.

However, you can refer to and briefly quote the use cases in the comments, to give future maintainers a bit of context.

Most requirements can't be expressed that way; any that can be, should be expressed as function names, as with all the examples which have been given.

-Billy

Perl has macros out the goat ass

[yerricde]

programmers should care about the macro facilities of a language. Perl and Java have none built in.

Nonsense. Perl's macro system can even translate Latin into Perl at runtime.

Re:Perl has macros out the goat ass

[bcaulf]

I stand corrected; thanks, yerricde. It is indeed possible to mess with your Perl during the compilation process. You can define your own little language. But if I'm reading the man page you referenced correctly, this module includes its own custom parser and translator. At the end of the parsing-translating process it spits out a string of text.

That is a psychotic, albeit powerful, approach to extending a language's syntax. And it is not a macro system. Wouldn't you agree?

Netscape 4.x; mod_gzip

[yerricde]

Take a look at their HTML/CSS and see it is HUGLY bloated

It's bloated because the webmaster tries to support Netscape 4.x, and doing CSS inheritance (which would unbloat many sites' CSS) will crash Netscape 4.x.

and not gzipped

Is there a Free module for Apache HTTP Server to do this? (In other words, is mod_gzip Free?)

Lack of predication on some archs

[yerricde]

The worst programmer habit I know of is copy-and-paste coding instead of using subroutines.

Sometimes, you don't want a lot of if-then's or subroutine calls in an inner loop, especially if software is supposed to run in real time (less than 50ms latency from key input to video and audio output) on fixed hardware (such as a game console) with all required features.

You can tell people not to do it, but some always will.

And some never will. I understand that, as a prominent CS scholar has stated, "premature optimization is the root of all evil," but, once we have a working sub-real-time implementation, some corner cases in software that processes digital signals really need code duplication if we expect them to run efficiently on architectures that lack predication. (Predication refers to machine instructions that say "execute this instruction if and only if the given bit of flags is set.")

Then there's the problem of not being able to quickly pass a function's local variables to another function because C, C++, and the Java language do not allow functions to "own" other functions, a construction that was quite handy in Pascal.

Re:Lack of predication on some archs

[tim_maroney]

I agree completely. Performance considerations trump elegance considerations, provided they are driven by empirical measurement rather than speculation.

However, this does not necessarily mean throwing all abstraction out the window. Much can be inlined or macro'd effectively.

But yes, needs must when the devil drives. However, this is a very different case from the usual lazy copy-and-paste scenario, which seems to be driven by the strange idea that adding the one line of code to make something a method or subroutine takes longer than copying the entire segment and tailoring it to a local need. I've really never understood how this was supposed to save coding time even in the short term, and it's clearly a huge loss over the middle and long term.

Tim

Function calls are not always cheap

[yerricde]

Function calls are cheap, performance wise.

Not in an inner loop, on fixed hardware, in real time signal processing. And yes, I have profiled my code; for some functions that have four separate cases, inlining each case separately improves performance. However, using compiler features to inline a piece of code tends to destroy portability (say if you have one version that runs on PC and another that runs on GBA) if you use anything but GCC-across-the-board.

Documentation not comments.

[MikeFM]

I think an important thing to clarify is that documentation is good and comments usually mean your code is bad. Comments are not documentation.

I attempt to keep a documentation/history somewhere for each function, class, or any other major component. In this documentation I typically explain what the code is for, it's inputs and outputs, and a history of why certain things were done in the way they were. This is often just one or two lines but despite good naming practices and good coding there are times where it just isn't obvious that on platform Foo you have to do it this way or you see Bar bug. I don't however go through code line by line commenting what it's doing and why. That should be perfectly obvious if your code is any good.

Hardware is cheap?

[Anonymous Brave Guy]

The "hardware is cheap" mantra is exactly why we have crapy code and bloatware. "I dont have to optimize, hardware is cheap." "Why make it efficient? hardware's cheap!" "I dont care that it is slow, hardware is so cheap these days"

I agree wholeheartedly. I think most of the problem is McManagers failing to understand what "hardware is cheap" means.

Hardware is cheap, but this is a relative statement. Shipping your kit on a platform with a 1.5GHz processor instead of 1GHz is often a better choice than having your whole development team spend six weeks profiling and optimising the code to get a 50% speed improvement. This is very expensive in terms of development time (= time to market hit and extra wages). Worse, it tends to lead to obscure optimisations in otherwise clean code that rarely get documented properly and thus hinder maintainability. So, in this respect, hardware is cheap.

However, this is not an excuse not to write good code in the first place. Don't prematurely optimize, sure, but for goodness' sake get the basics right.

• Use suitably efficient data structures and algorithms.
• Design your code in a modular way, so that if you really do need to apply optimizations, you can do so in an isolated, controlled and easily testable way.

The cheapness of hardware does not justify getting sloppy and ignoring the above. These are the major causes of bloatware and the reason why I could start typing faster with Word 5 for DOS on an 8086 more than a decade ago than I can with Word 2002 on a 2GHz Athlon running WinXP.

I have to disagree (Re: Perl vs. PHP)

[Tablizer]

(* keeping all of the business logic in sensible, cohesive compartments that run on an application server.......In summary, for shallow web-only stuff, PHP is a reasonable tool. For stuff beyond web work, PHP is out of its design envelope. However, Perl works here just as well as it does for web work. *)

I never suggested that PHP do *everything*. The trick IMO is to use RDBMS as the "transport and messaging mechanism". All kinds of different langs and apps can communicate thru it. Even IBM mainframes partitioned stuff into interface-driven stuff (TSO) and batch jobs.

You are describing 30-year-old technology.

Also, some people just don't like Perl. Don't take it personally. Everydody's neurons just happen to be wired different due to Darwinian divsersity (or a flayful Allah).

Re:Complete rewrite necessary or a waste of resour

[Martin S.]

>Not too long ago a link was posted to an interview with Joel Spolsky who used to work at Microsoft.

WOW, you are easily impressed. Just because somebody used to work for Microsoft does not make them an expert software engineer. In fact judging from poor quality of most Microsoft software we can safely assume the absolute opposite.

Use XP and Refactoring Techniques

[juanco]

This is a bit late, but...

You don't have to refactor your code in one swoop. You can do it gradually, like when you open up a piece of code to add functionality. Then, you can make changes that make the code cleaner, and make it better suited to support the new features. As long as you write tests for the existing and new functionality first, there should be few surprizes.

I recommend Martin Fowler's book on "Refactoring". The classification and analysis of typical code transformations makes it much easier to think about ways to improve code without braking it.