Slashdot Mirror
Justifying Code Rewrites?
snow70 asks: "It seems that it is getting harder and harder to justify the decision to re-write rather than hack/patch away on a flawed code base, given the economic situation and the drive to squeeze every last drop out of an existing code base. Are the readers aware of any research articles or reports that extoll the benefits of re-writing in terms of the cost savings in maintenance and support ?" This topic was discussed two years ago, but I think many people who'll find themselves in this situation would also benefit from any research articles or publications as snow70 requests.
But at least they're admitting it this time ;)
I can't say that I don't give a fuck. I've just run out of fuck to give.
But it's been my anecdotal experience that unless there's a real, justifiable reason for the code rewrite, then it's usually not worth it. Something like taking a client/server application and making it a web application requires a rewrite as a matter of course, for instance, but otherwise the costs and time tend to outweigh the benefits.
Sure, a code rewrite allows you to remove bad architectural assumptions and dump the cruft, but it also introduces the possibility of an exponentially higher number of bugs. So you've dumped a fair amount of developer work in the rewrite, and then it needs to be followed up with an even more massive amount of QA work. And, if you're actually making sweeping architectural changes, chances are very high that the new architecture will pose some unforeseen problems of it's own.
I'm not saying a code rewrite is never a good thing, but, IMHO, it's not something that should be done just because "this code is nasty" or "language X is cooler". If you can't think of a reason as to why a rewrite should be done, then it shouldn't.
From The Cathedral and the Bazaar, by Eric S. Raymond:
"... To put it another way, you often don't really understand the problem until after the first time you implement a solution. The second time, maybe you know enough to do it right. So if you want to get it right, be ready to start over at least once [JB]."
Portions of that quote are borrowed from The Mythical Man-Month. More is available online
"I'll say it again for the logic-impaired." -- Larry Wall.
Point one: rewriting from scratch can be a strategic disaster.
Point two: rectifying a "flawed" code base can be done within the framework of good software engineering practice, and is a very marketable skill besides.
Refactor, don't rewrite! First write unit & functional tests then start refactoring.
Sorry to say this, but we don't live in a perfect world. As long as management bases their timelines on when the next trade show is coming up, we're not going to see rewrites happen until one of these deadlines is very badly missed. The people in charge just plain don't care about flawed or badly maintained code bases. They're concerned with getting a product that meets the promises that Marketing made. Sadly, engineers are too good at working within their limitations, thus making rewrites not all that necessary. (As in "we can't launch unless we do that" necessary.)
I don't think finding articles that show off the wonderfulness of code rewrites will be useful at all. Instead, this topic needs to be made attractive to PHBs out there. Make "total rewrite" a buzzword (you know, like e-commerce, push technology, and multi-media) and you'll gain a lot more leverage than landing a report labeled FYI on your boss's desk. Make it fashionable.
You all may thing I'm being cynical or even sarcastic, but I've spent the last 6 years watching PHB's work. I've listened to some bizarre rationale. I understand enough about how they work that I could become a cartoonist. I'll be happy to carry the torch when Scott Adams retires.
"Derp de derp."
Let me quote a little from the linked article
Some managers will disingenuously try to use "Return on Investment" to retroactively justify past spending decisions, without regard to whether trying to salvage a poor decision is a cost-effective use of current and future investment.
/. will appreciate: Imagine you've spent 8 hours downloading a huge file over dialup, and it's 80% complete... and during that time, your friendly local Telco person has arrived and installed a DSL line that could download the same file in under an hour.
ROI should be used as a measure not of how far you've gone, but rather how close you are to your current goal.
As an example which anyone reading
Which, then, is the better use of your time: to wait two hours for your current download to complete, or to ditch it for the one hour download?
Of course, as a precaution you might let the slow download continue in case the faster line doesn't work right, just as a company could keep the existing model in operation as a prototype of the new system is developed...
Companies that can't see this, or that foster a corporate culture in which ROI is used as a retroactive cover-your-ass rather than as an estimate of future costs and benefits, will in the long run be easy pickins for their more nimble-minded competitors.
If you can make a business case for rewriting an existing piece of software, any sane project manager should go for it. The whole problem is that most programming still happens based on hunches & gut feelings. If there's too much money like before the .com crisis, you can afford to aimlessly optimize pieces of code like that and get away with it. However, in the current economic situation, people are not handing out money anymore. You have to convince people to do so.
Delivering a piece of code on time with the desired level of quality still boils down to black magic in many organizations. A large percentage of software projects either fails or delivers something different than was intended.
People have noticed this and telling them that you'll lock yourself up, hack away for a couple of weeks and maybe something good will come out does not do the trick anymore. They simply do not trust you to deliver what you claim you can deliver anymore.
Of course that doesn't mean you shouldn't rewrite code. You should however, make more of an effort to provide evidence to the relevant people that a) there is a problem with the code that should be fixed (e.g. bugfixes take an unusual amount of resources on that part of the code) b) you have a plan for resolving this problem (e.g. spending two weeks on refactoring) and c) this plan is cost-effective (this is the hard part).
I've seen a project where it was calculated that fixing the outstanding 100 or so bugs for a component (serious bugs) would cost a certain amount of money. This process was likely to introduce more bugs of the same serious nature (given experience with maintaining this component). An alternative was to redesign and reimplement the offending component, a plan was made to do so and it was approved because executing the plan was shown to be more cost effective than continuing to maintain the old component. The plan was executed and the new component has so far proven to be more reliable and maintainable.
Jilles
I am a sold on the concept of refactoring instead of completely rewriting.
About four years ago, I rewrote a 15 year old device handling sub-system. It was quite horrible. The primary justification for doing it was to generate billable hours to charge the government. The secondary justification was because the old code was horrible, nightmarish and impossible to maintain.
Unfortunately, I didn't get the opportunity to redesign the legacy file structures it used. I also had to maintain 100% backward compatability. On my personal agenda was adding useful features. In hindsight, if we had used the refactoring approach the end result would probably hve been better. Certainly the beta testing would have gone smoother if it had been done in smaller increments. (Having a non-technical manager didn't help either.)
With that experience behind me now, I recommend two things to justify a major piece of work (either rewriting or refactoring):
1) Identify the current design flaws
2) Identify the features missing
These days, the only time I'm presented with a rewite opportunity is when an application is being migrated to a new platform. i.e. Web enabling a legacy app. The only applications that remain out there are still there because they are too ugly, outdated and nightmarish to maintain. So once again, I would recommend refactoring.
Don't be mad at management for being gun-shy when they hear the word "rewrite." They should be leery of a major rewrite. The end-user training costs alone can be devastating. And unless we're talking about your own personal copyrighted intellectual property, they do have a lot vesting in the old "ugly" code that they (rightly) should be scared of losing.
"God is dead." - Frederik Nietzsche
I'm not going to touch the refactor/rewrite issues, because the fundamental problem is you are not effectively giving yourself enough time to do whatever the hell you want to do. Take what time you think it will take to implement something, double it, and round up to the next unit of measurement. Odds are the business does not give a rat's ass about the wrong or right way to implement something anyhow. Why confuse them with options?
How long will it take to fix this? (keep the snicker internal) Two weeks...
+++ UGUCAUCGUAUUUCU
I'd like to rewrite most of the projects I've worked on, especially one in particular which gets used and re-worked year after year. A ground-up reworking of the code would make future changes hugely easier, but instead we spend vastly more time trying to force more cruft on top of the massive pile of cruft and hoping it'll work. Even when we start six months before the usual time-frame, we always - ALWAYS - end up making modifications right up to the time it goes live. (I've actually had to recompile a program on a laptop, while in the car heading to the install site!)
I found the comments of the first few posters to be insightful, but there's at least one situation they overlooked - what if the code is crummy for perhaps the most common reason of all: insufficient time combined with constantly changing requirements? In that case, given time to really do things right could make a difference. Taking the pressure off lets you actually design BEFORE you code, instead of doing them in parallel, and you might even get to shift some of the user interface planning off onto someone qualified to do it... (I've never had the pleasure of working within a team on a project, BTW, so YMMV. Oh, and I've never had the pleasure of working on a project where there was anybody to say "no" to the customer's latest wacky, impractical, last-minute ideas...)
OK, where were we - oh yes, I'd like to hear the answer to the poster's question too.
Perfectly Normal Industries
Brooks' The Mythical Man Month reads "...plan to throw one away; you will, anyhow." However, I read this to mean a real working prototype rather than a production system that simply gets tossed. If you are wondering whether a genuine production system needs rewriting, it's probably too late. If you can tolerate the pain, factor where you can; if not, brave the job market for a better opportunity.
Healthcare article at Kuro5hin
I have a pretty good formula for getting the actual time it will take :
...' : 3 months minimum, 18 months average.
If your coder is just out of school take any time estimates and factor by x5 Estimates two days, it will take him two work weeks. (*)
Junior programmer with 1-2 years experience on this kind of thing - factor by x4.
Senior programmer with half a decade or more of experience = factor by x2 or x3 - it varies.
Any task with a description that starts 'Why don't you just
Manager says 'It can't be done.' - it will be done by Monday.
(*) - the reason the Junior guys do this is the estimate how many hours it will take them to complete (which is generally half the number it will actually take) and the math in their head assumes 24 hours of productive time in a day until they finish it. They -could- finish in two days, but they are not going to actually finish in two days. Senior guys recognise that they only get 5 productive hours in a day (on a good day) and divide the number of hours they think it will take by 5 to get days. Of course they also take twice as many hours to get it done, hence the 2x factor for them.
Glonoinha the MebiByte Slayer
This way, if you're rearchitecting big sections, you don't have to worry about keeping bidirectional compatiblity during the conversion. This approach also works well for porting between different languages.
However, you're not throwing away your old valuable work either. You rewrite most stuff by editing your old procedures. I usually find that I have huge chunks of code that simply get uncommented again because they work fine in the new incarnation.
If a codebase is sufficiently complex and mature
so that rewriting raises business issues, you're
almost always better off incrementally refactoring.
Take one bug at a time, and resolve it by improving
design, rather than by adding layers of cruft.
Formulate end-design goals, and work towards them
incrementally, preserving all current correct
behaviours.
Yes, it takes longer than rewriting, in order to
achieve 90% of the end goal, but the last 10% is
always the deal maker or breaker. If your
codebase is decrepit and complex, you almost
certainly don't have enough interface definition
to rewrite a fully functional replacement. The
process of incremental refactoring allows you to
simultaneously reverse engineer and document the
operative interfaces.
-I like my women like I like my tea: green-
author looked at the code I inherited, he would change his point of view.
.eof .eof then ...
We got a function thats 16 hundred lines long(not countinf white space and comments), it has 5 goto's.
The person who wrote the code was a self taught programmer. the next person was a self taught programmer with no experience in the language.
The use 3rd part controls that leak like a sieve.
there is redundant code all over the place. Comments are lacking.
there is stuff like:
If (X = 1) and (X = 1) then
and stuff like
while
if
loop
Nothing is generic, so swapping aout the major 3rd party control would entail change 95%
there a 75 forms, 25 of them identicle.
Use of the database is incorrect, we got dirty reads happening all over the place.
This code might be salvagable if there was a design spec in the begining, or any documantation for that matter.
When it was written there where no specs, and the people working on it had little experience with the industry.
The Kruger Dunning explains most post on
if it's worth to write, it's worth to rewrite
Actually I am talking about refactoring in this context, a targetted re-write of a core of the application to deliver scalability and stability that the existing 'design' simply doesn't deliver. We are talking about less than 10% re-write, whilst keeping everything that the customer is used to seeing in terms of logs, O&M management etc. However it seems there is little hard evidence to support this process, which to me seems so natural.
you might be interested in this book: Refactoring: Improving the Design of Existing Code
Any sufficiently patched-up code base is indistinguishable from magic.
My first project out of college was to "patch" some computer software. It was written by three different engineers in FORTRAN, working for about 2 years. When it started crashing all the time, the solution was to put a watchdog on the computer. Then it would loose the state of computation, so it started saving it's computation state regularly, and upon booting would restore it. Eventually it saved the state that led to a crash, at which point it started rebooting like a yoyo.
Looking over the software, it was a total mess. Every section of the code completely reordered all data on the stack, going several levels deep. I asked what the formula the code was trying to compute, and was shocked to see a simple iterative formula, with about 10 variables. I recommended a rewrite. I was heckled.
I was eventually given one month to waste, before they said I would be forced to patch the existing code. The first thing I did was remove the watchdog. Then I did a data flow diagram and found a straight line. No control over the format of the data going out, but total control of coming in. So presto, I structured the data coming in to match the required format of the output. It would all sit in place and the routines would work on it like a pipeline. It went from taking four or five minutes to do a single pass to nine seconds. It also didn't crash all the time. The size of the code shrank dramatically.
Two other times in my career I've ran into similar code. The next was about two hundred pages of FORTRAN, that I replaced with one page of lex.
The next time was a database engine written in FORTRAN by an engineer with no database theory (data/indices mixed, every index was of a different form, every violation of normalization you can think of, etc). I recommend a rewrite then, and just about got fired, because the VP wrote the code. It wasn't rewritten and their competitors have just about squeezed them out of the market, because they can't adapt.
Lesson: FORTRAN written by engineers is generally in bad need of a rewrite.
I used to wonder what was so holy about a silent night, now I have a child.
I'd say that the right thing to do (to start off with, anyway) is to refactor, not scrap it.
Take that 1600 line procedure and rewrite it more sensibly (modular, no gotos, break it into dozens of smaller procedures). Then do the same with the rest of the code. Remove duplication as you go through. You'll constantly have a system that works and is testable and you'll be able to reverse-engineer the spec as you go through.
If you tried to rewrite it from scratch, you'd have to analyse the whole thing to see what it did anyway - you might as well rewrite it as you go.
My Journal
If the code base is huge (and ugly) and the encapsulated knowledge is large (but the system runs correctly), then I would discourage a re-write:
Because when you blindly try to rewrite complex business logic you will make many mistakes. You will have to fix them, making your codebase patchy again and probably end up in the same mess as before because you do not have the time to clean things up.
Refactoring however in this case is the best you can do. Because the logic runs okay you can write test cases for the code you are going to refactor.
A rewrite is better than refactoring when business needs changed dramatically and the old code does not fit the new requirements at all.
Also code that is just *completely* flawed (such as written by a completely incompetent idiot), extremely buggy, unstable and not fulfilling requirements is good to rewrite as well.
The smaller the codebase the better the chances for a successful rewrite.
You will anyway
Fred Brooks.