Slashdot Mirror
When Making a Comprehensive Retrofit of your Code...
chizor asks: "My
programming team is considering making some sweeping changes to our
code base (150+ perl CGIs,
over a meg of code) in the interest of consistency and reducing
redundancy. We're going to have to make some hard decisions about
code style. What suggestions might readers have about tackling a
large-scale retrofit?" Once the
decision has been made for a sweeping rewrite of a project,
what can you do to make sure things go smoothly and you don't run
into any development snags...especially as things progress in the
development cycle?
we're doing something similar - and switching to java (JSP's + Tomcat, struts) to replace a lot of old perl cgi's. The java code is much, much cleaner. But object-oriented perl code can help if you don't want to take the plunge too far into a new language. And at least find a way to go mod_perl rather than CGI, for the things where performance matters at all.
Energy: time to change the picture.
refactor as a result of learning from your mistakes and redundancies;
and try to minimize the busy parts (where all developers have a hand) when things change (like lists of unique symbols, numbers, etc.)
You could've hired me.
I'm sure a bunch of code nazis will disagree with me (please note the clever way I attempt to pre-emptively undermine their arguments by labeling them as 'nazis') but sometimes massive engineering re-writes are not necessary.
Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.
Most companies that decide to massively re-engineer their code (do a big rewrite) usually end up regretting it because it forces them to re-fix the problems that caused the original strange looking code in the first place.
Does your CGI nest work? If so, maybe you should leave it alone. If you are fixing specific problems, then go ahead, but if this is a generalized attempt to fix the 'not invented here' syndrome that plagues engineers (who will almost universally agree that it is easier to write code then it is to read it), perhaps you should reconsider.
You should read up on Extreme Programming, in particular Code Refactoring. It's a method of cleaning up old code. A very well written book, as well as an excellent code-housekeeping method.
Do it doug.
Break the code into re-usable modules (or objects if you go another route besides perl, or even if you do, if you like that sort of pain), each programmer responsible for his own set (ie layout, calculation, database, etc)
Don't stop maintaining the old code code until the new code is on solid ground. No matter how sure you are that you can do it, the new code might never come through.
The Gardener
--
Basically, Joel's take on a similar problem is: don't do it.
Unless you have a _really_ good reason to do huge change to a big codebase, don't bother, and make something more productive instead.
cheers
It always seems to me that large rewrites (though I've never done one as large as the one discussed) tend to make bad code bases worse for a while. If there is any effort made to maintain functionality in a changing codebase, a piecemeal rewrite is more of a headache than a help. So, if you're planning on a big rewrite, please give your developers the full thumbs up to break everything and expect them to put it back together later--and expect to see nothing tangible in the short term. Or call it off.
That being said, my last company sat around and bickered about code style for nearly 4 months and produced no code that wasn't rewritten later. If you are going to concern yourself about style, settle that well in advance and make sure it's logical and consistent.
It's also been my experience that conformant code style is highly overrated. Once the Best Practices document extends beyond language constructs and caveats, into brace styles, spacing, tab size (yes, there was a 3 space tab stop standard at my last job--wretch), and even the naming of locals, parameters, members, constants, enumerations, etc... it got to be a thick ass bible of stuff that only a few people would digest or attempt to adhere to. The point I'm trying to make is, choose your battles. The hope is your developers will make sane choices independently, and use standards to help integrate different peoples' work together. Anything beyond that and it's pissing in the wind.
My $2e-2 or less.
Any connection between your reality and mine is purely coincidental.
My firm went through this sort of thing just two years ago. The PHB at the time decided, for some reason, that our 300,000 lines of semi-poorly written C code, and 50,000 associated lines of Java (Dont' ask).
Anyway, it took 7 of us over 2 months to get even halfway done. The pressure the boss was putting on us was awful, and he didn't really even understand what we were doing, even though he was the one demanding it. I think she read it in a trade mag somewhere. God I'd do a lot more work if she didn't read that shit.
Anyway, about halfway through the "Great Leap Forward" (as we [appropriatly] named it) the boss quit, and the next boss, who so far has been fairly clueful. He didn't think the whole deal was needed, but he was pressured by the former bosses husband (the CTO) to get it done. Seriously.
Hope yours goes better than ours. From what we did, heres some tips I can give you.
1. Be consistant through the whole thing.
2. Make sure everything is planned before you start. This was the one part we got right.
3. The team you have should have worked together before, because this sort of task requires previous knowledge of eachother.
Other than that, my condolences. Or maybe it will work better for you.
Good luck!
Slashdot is a site built around perl scripts, so one might expect there to be a considerable amount of knowledge and interest about a topic such as this.
Could you, dear writer, perhaps be the maintainer of a GIANT collection of perl scripts badly in need of a rewrite?
Are you the maintainer of Slash??
Will the real Bruce Perens Please Stand Up
Transitioning in new managers or having the current manager only look in on the project once in a while is as sure a path to madness and doom as no management at all.
Our due date was mid-August, we'll be lucky to get it through testing and into production by January 31st. All the while with the logjam we're having to put pieces of it into production and cross our fingers that the new changes don't break anything.
Love to talk more about it, but need another gallon of coffee.
A feeling of having made the same mistake before: Deja Foobar
...don't fix it!
;)
"In the interest of consistency and reducing redundancy." There's only 1 MB of code, and while that is a lot considering it's all text, I suspect that you will find yourselves recreating (and then refixing) most of the original problems that are more than likely fixed by your existing codebase.
Old adages got to be old adages because they represent sound wisdom. So if it ain't broke, don't fix it.
bc90021
"Python is executable pseudocode; Perl is executable line noise". - Someone wiser than I.
libertarianswag.com
You are going to rewrite the system from scratch. Design from scratch. Your new design might be able to use some old code, if the old code is useful.
A large scale retrofit is really an oxymoron.
IMHO, but with 15 years experience.
-pyrrho
This is an excellent suggestion. What I always try to do is break the codebase up into loose-coupled modules, and then take the nastiest one of them (well, I guess you could pick which one you want to do first based on anything, really) - and convert that to your new technology.
Doing this will help big time when something goes wrong, which it will. The area that caused the problem will be much more focused.
This also lets you start taking advantages of the benefits of your new technology much sooner. Roll the new in with the old - in production. Less scary as well. Small steps.
The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet. -- Michael Jackson
1) Know what you are doing. 2) Know what the code does.
Both are expedited by good documentation. This is so important, it deserves to be written thusly:
DOCUMENTATION!
Write everything down. If the code is not commented, figure out what it does and write it down. When you add a line or a module, write down what it is supposed to do. Declarations? Write those down too. Document everything so that you can figure everything out, both now and down the road when you decide to fix something else.
This is the voice of experience. I have had to reverse-engineer my own code 6 months after I wrote it because I failed to document anything. Learn from my mistake.
I have a strong belief in the Second Amendment.
In my experience, if you have something, and it is functional, then you are better off to migrate and refactor what you have to end up with improvements rather than doing the slash and burn and rewrite.
The advantage of the migration and refactor approach is you can do it in a way that leaves you with a functional product the entire time, giving you the ability to respond to new requests/requirements as they arise.
If you try to do a full rewrite, and some new urgent deliverable comes up in the middle of your rewrite, the only way you are going to be able to deliver in time is to hack it into your old functional code.
From what you say, you are planning on making these changes to clean up the code and make it prettier.
I would strongly urge you to reconsider this as the probability that you will end up breaking parts of the code while "cleaning" it up is quite high..especially since you seem to have a fairly large code base ~1MB
Ugly code containing redundant stuff is still better than beautiful code that doesn't work.
Make sure you have a suite of tests that produce known output for your old code, so that you can ensure that the new code works in exactly the same way. Don't add anything new until you are proven conformant with what you had before.
Why do people mod comments about alternatives as Flamebait (and presciently, this one)? Are people afraid to hear that you should't write large scale systems in Perl?
It really is valid (and in my opinion, correct) to say that if you _are_ going to do this you should look at other technologies and languages. Perl is for system administrators and system administrators-cum-developers, not real software development. Look at java. Look at PHP. Look at commercial and non-commerical web application systems, like Zope. Or don't rewrite it at all if it works. But for God's Sake, don't rewrite it in Perl - it's pointless.
There is no use in doing a rewrite unless you do it right from the beginning. If you can't spend a decent amount of time planning the architecture of the system, then stop now, and quit.
Also, since you have decided to pour resources into this thing, then my opinion would be to make as much of your code generic so that you don't have to make code changes later. It doesn't matter if there is an initial performance hit with the systems, becuase in the short term, you can convince your boss to get a new leet server, and in the future, hardware needed to run your apps will be trivially cheap anyway.
If you are going to cut a new release, try to avoid going back and taking snippets of code from your old system. It makes people slide back into the odl paradigm and can cause detrimental effects on the bottom line of your new system. This is new, so the least exposure to the old one, the better.
Bye!
Often I've had to modify systems which, once I understand how they work, which parts are actually still used, what's redundant/poorly written, I've usually pulled off changes in 1/2 to 1/5 the amount of code. Nice, neat, clean well documented stuff, too.
A feeling of having made the same mistake before: Deja Foobar
Well, i can't say it enough. Use a web framework, i don't know of any for Perl off the top of my mind, but i use Resin.
.xtp files (in the case of resin) and simple XSLT sheets parse the XML to render to the user.
Resin is a JSP, Servlet, XML, XSLT application server that support all the latest and greatest EJB components and Managed persistance on the database and makes a great framework to build from.
I have Postgresl for the Database, and use beans to run queries and output via xml and then i have XSLT draft xml data into the wonderfull HTML code.
The beauty is, my code is in beans, servlets and jsp's. My HTML is in
Just means i can produce output easily for WAP, Palm, CE, Normal Web Browsing, EMail, and what not without modifying the backend. Just create xslt using a session identifier to bring up the corresponding stylesheets for whatever device is acesssing the page.
Enough about java, but something similar, be it in house developed would be your best bet.
I also get away with using a Swing applicatoin to manage the database, users and run reports provide an easy to navigate gui which just interprests the same xml data that would be retrieved by the html client. Not a single change to the backend since i'm using the beauty of soap, jsp's, servlets and java.
Virtualize your interfaces, standardize on your backend and use re-useable components. Perl is similar in many ways that you can load libaries and abstract your code from the display which will save you tons of hours of hassle in any future upgrades compared to the bit of changes you would do now
A ton of people will tell you a ton of things, never having retrofited anything.
- Do not undervalue the investment you have learning your existing coding language. New challenges await you if you jump on a new language like java. Make the jump if you are excited about learning about the new language.
- If you use your existing coding language you will literally fly through the retrofit. Do it piece by piece. Make all those changes first, then test app, then make next set of changes then test. The simple fact is, most wasted time is spent on bugs not working on performance, and you've already knocked down a lot of bugs, don't let them pop back up by blowing everything up. There are books on this.
- Sometimes blowing everything up is worth it. Do it right this time. Realize it won't be as perfect as you might think it will be.
- Remember there are countless open source and shareware products that tried to create TNG with a total rewrite, got nowhere, and ended up improving their existing product. Remember the lesson, bite off what you can chew.
- Spend a week poking around researching possibilities. I do this all the time, bookmark things I think are important. Then for the next project you've got all the little things you might forget at your fingertips. Optimzations/Tools/Paradigms. Think you know it all? You'd be suprised at what is out there and what you missed. And what you spent a month in house re-inventing. This one's important.
- Use open source software. Nothing beats free. Nothing is more fun. Java's ugly standardization history makes me puke... the BS Sun has pulled with Java is staggering. That the Java Lobby swallows it and loves it even more so. This is irrelevant to your question, and not fair to the Lobby, but I like to give them a hard time.
- Colorary to Java. You need less abstract design then you think. Endless object hierarchies will weigh you and your app down... Their are books on this too.
- You need more documentation then you think. Ever found code someone ELSE wrote too EASY to follow. I don't think so. Especially if you are using perl and someone is enjoying the line noise capabalities perl allows. Perl has 20 ways to do EVERYTHING, you may not know the latest or twistiest. Document as you WRITE the code. Do not leave at the end of the day without catching up the docs. A week of documenting is the worst form of hell, avoided with a minutes worth of clarification each time you write a function/class.
- Hardware is cheap.
Anyways, have fun... and good luck. Be interested to read what others have to say.
He says the exact same thing as you, only better.
"If you look 'round the table and can't tell who the sucker is, it's you." -- Quiz Show
Keep It Simple Stupid. Keep code simple, why have funky lines of code that looks awe enspiring. Its pig ugly. I work on the principle of KISS. When I am working on a code area that is being cleaned up or fixed, I try to simplify that area. The simpler it is, the easier and quicker it is to maintain and bug hunt.
:) Exception handling for seperating error handling from actual code that performs the actual work).
Im a strong believer in managed code (whether its C# or Java, except managed extensions to c++ which are damn pig ugly
Make sure there is GOOD in code documentation (and out of code documentation) to explain the intention of that code. (Intentional programming is a research area btw, to program ones intentions).
You know when something is bad when you have to maintain that code later on, 6 months down the line or whenever. Thats when it bites you in the buttie.
----- Whats wrong with this picture? http://www.revoh.org:1234/whatswrong
Instead, very globally, identify the features and structures that you require in 3, 6 or 12 months time, from a marketing or similarly broad perspective. Identify the parts of your codebase that would need to be changed in order to achieve those goals (this is as good a definition of "out of shape" as any).
Then as you fix up your code: look at your fixes. And from these, formulate the rules that should guide your next development cycle.
Pushin' 'n dealin', shovin' 'n stealin'
To expand on the concept of Refactoring:
1. Write a test for a specific block of code.
2. Appliy the refactoring
You are going to want a good testing framework.
To expand on the modules post: Do a dependency analysis. If you are writing DB based code, look at what tables can be logically grouped together.
We did something like this at my company not too long ago. The basical level package we had was the security package, which identifies users and roles. Most other packages depended on this. All contact management stuff went into a package called Directory. All stuff for the people our system was managing went into Participant etc.
For each of the packages, split the code out into a set of interfaces, and a set of implementing code for business logic, and the UI required to drive that business logic. This is the standard breakdown for code. You may want to further pull out code into helper packages. Avoid the urge to call these helper or util, and instead try to name them based on what they contain: we have one called format for stuff like phone numbers and social security numbers.
Don't forget the make scripts. What ever build you use, it should be used to specify which modules you want to compile/deploy
I recommend a little UML modeling session for the end package structure.
Go in little pieces. After each refacotring, makes sure stuff still runs.
Good Luck
Open Source Identity Management: FreeIPA.org
Get a copy of Expect, or even better, DejaGnu, and start scripting your tests.
Do you have the original engineers for that code?
Most knowledge is in theyre head, not on paper unfortunately. Theyre experience in that area can never be written down.
----- Whats wrong with this picture? http://www.revoh.org:1234/whatswrong
I suggest writing a comprehensive test suite first and making sure that the existing code either works or that the bugs are documented. Then run the test suite against the re-write as often as is reasonable. Use a good source control system. Have seperate development and test systems (it's amazing the number of people who try to test on development systems).
Don't build it. Instead, evaluate the code you have now and plot a course towards the idealized system. Approach the actual work of the "retrofit" incrementally. Count on having multiple customer-facing revisions of the software tagged and QA'd before the system you're delivering looks anything like the planned rewrite.
Taking baby steps towards a new design is probably the only way you'll ever migrate your project to that design. With the knowledge you've accrued working on the old system, it probably seems straightforward to start from scratch. Even if this isn't wishful thinking, though, it's a waste of time. Part of the discipline of design is an understanding of where the "hot spots" are that can't tolerate inferior implementation, and how to tell those hots spots from the spongy mass of integration, reporting, tracing, and sanitizing that is neither performance sensitive nor mutable enough to justify engineering effort.
You can take early baby steps that make it easier to make holistic changes down the road. Refactor relentlessly. Migrate code recklessly out of subsystems and into common repositories and libraries. I've found it handy to distinguish between "proper" shared library and "dumps" of utility code that don't need scrupulously conceived interfaces.
Most importantly, design for testability. In this respect, the biggest asset you have is the steaming lump of old Perl code you're facing; use it to figure out the expected behavior of subsystems. Write replacements, in modules with clean interfaces, and unit test them. A unit test probes code (functions, statements, internal states) --- NOT entire programs. You'll work ten times faster when you can move forward as a team knowing what components you can trust and what components you need to worry about. You'll work ten times slower if you haven't clarified your outputs, side effects, and return values enough to know whether your replacement parts are valid!
We've seen articles on Slashdot before about this and I agree with the prevailing opinion: rewrites are often seductive traps and time sinks that don't offer value to customers. A better mentality that will eventually get you where you (think you) want to be is to adopt a strategy of constant measurement (testing, profiling, debugging) and improvement.
I have coworkers that throw around issues like performance, which can be a red herring if you are doing a web site with only a few users. (one guy insisted on doing all is string stuff as string buffer instead of as strings because it would increase performance)
You may want to look at your code in a similar fashion. There may be just a few modules that are causing problems with redundancy(or whatever is making you want to rewrite them). Why rewrite the ones that have worked faithfully without any problems?
love is just extroverted narcissism
It might seem like and obvious step, but don't throw away the old system until you're sure that the new one works! Keep somebody minding the existing, working system so that if/when your attempt to completely rework it fails you won't be stuck. Once you have rewritten it, try setting it up on a trial basis in parallel to the working system so you can find the crippling bugs before they take down your system.
While it's not a perfect example, Slashdot is actually a decent example with their switch to their new system. They kept the old, crufty version as the primary and set up a beta site with the new software. They knew that there would be problems and got some of their more loyal users to test the new system and only switched over it after they were pretty confident that they had gotten the worst problems out of the way.
You can afford to take a few more risks as long as you keep a known working system around as a fallback.
There's no point in questioning authority if you aren't going to listen to the answers.
Actually, one thing that can be very very helpful is to also write down your assumptions for each function. What format are you assuming the variables will be in when passed to a function? What if they're passed incorrectly? What if someone tries to "hack" the function? How would they go about doing it via a form submittal (for example)?
This is a good exercise and should lead to much more secure code.
Your tangled mass of spaghetti code paths are probably full of almost incomprehensible little design decisions and seemingly out of place declarations and functions, but most of those were probably added as specific fixes for bugs encountered under real-world use.
Yes, and if they're cryptic and uncommented, they are worthless. Eventually one of these incomprehensible, magical fixes will stop working. Perhaps the bug it works around is fixed. Perhaps how the function is being used changes to previously unexpected behavior. Some poor engineer will look at the little big of magic, scratch his head, and be forced make a blind decision about how to fix it. Perhaps he can change the code while leaving the bit of magic in working, but he can't be certain, since he doens't understand it. If the collection of cryptic tweaks becomes dense enough, any attempt to fix a bug or add a feature becomes highly risky.
On a related note, don't let this happen to you. If you add one of these strange little fixes, for the sake of the programmer that follows you, document them. Just a little "Need to toggle the Foo because Qux 1.4 is correctly fails to do so" will bring tears of joy to the eyes of future programmers.
Here is an interesting link that leads to others links
http://www.dreamsongs.com/WorseIsBetter.html
I don't know how well this translates into Perl, but if the project is in Java, the steps are pretty straightforward.
1. Determine a clean architecture for how you would have solved the problem, given the benefit of hindsight.
2. Create new interfaces based on your new architecture.
3. Code facades to make your old code implement the new design.
4. One by one, re-implement your new interfaces until the new implementation has solved all of the problems that led you to re-implement in the fist place.
Of course, during development and maintenance, you should be doing all of this stuff everyday, anyway. Reimplementing everything just means the team has agreed on a global change in design, as opposed to the local design changes you don't need any co-operation to achieve.
If you follow this approach, you always can always have working code at the end of the day.
I've written and maintain a similarly large web application for my company, that was all done in Active Server Pages and SQL Server.
All reusable code, and authentication verification in my web app is in a single 'functions.asp' file in the root of my project. Every single page, page in the project has an include for this file which takes care of the authentication, and provides that ASP code to the rest of the app.
Also in the root, I keep a javascript-functions.asp which works the same way but is included only in pages that have html forms. This file contains common client side functions to validate forms and what-not...
On the same note, I usually use a single CSS file that is also called universally throughout the app to simplify maintenance of the look of the app.
On the database side I'd recommend using stored procedures as absolutely much as possible. Obviously try to make all your transactions atomic and reusable in stored procedures to optimize performance and simplicity.
That's my 2 cents...
The best thing to do about code style is find some method or function code formatter (built into your ide) that most everyone can agree on. Basically, it should be very easy to format your code after writing it. Then code formatting is a non-issue. Everyone lives (and dies?) by that formatting.
managers...why god invented purgatory
Large overhauls are usually mistakes. Details in the previous code are lost. If the overhaul takes non-trivial time, people become frustrated that two weeks ago they had a working (if problematic) system and today most of the system doesn't work.
Instead, make small incremental changes. Pick something lots of code is replicating and attempt to unify it into a shared code base. Spend some time documenting key parts of the code. Pick a particularlly hairy class or function and untangle some of the worst bits. These sorts of changes can reveal minor bugs, build up to significant improvements, and leave you satisfied at the end of the day that you improved things.
If a signficant overhaul is necessary, try to overhaul portions while maintaining the existing bits.
More realistically:
For every time you modify a part of code, ten new bugs are introduced.
.sig wanted. Inquire within.
When making a comprehensive retrofit of your AI Mind for Robots Code, you have to sacrifice the early instantiations for the sake of smooth sailing on The Road Ahead to the Technological Singularity.
A major Open Source AI initiative has no choice but to r00t out and correct mistakes ASAP as soon as they are found. History will judge us be how kind we were to our robot and cyborg mind-brethren, not holding them back but equipping them to meet the world. Therefore Mind.Forth AI is for the robots themselves, while the JavaScript AI (q.v.) caters to human users who need tutorial instruction. Welcome aboard, all AI coders of good will towards bots.
1) Identify common functionality.
2) Encapsulate in libraries
3) Be sure to extract enough generality that you don't have special case functions
4) Don't extract so much generality that functional interfaces become unwieldy.
5) Write everything in the same language.
6) Find any complex pieces or algorithms. If they can be simplified or re-written, do it. If not, save it so you don't need to debug it again.
7) Throw everything else away.
Check out this site. Lots of great advice
0 02 47.html
http://www.joelonsoftware.com/navLinks/fog00000
Has anybody got links to real world stats on coding for things like bug introduction vs modification rates etc...
----- Whats wrong with this picture? http://www.revoh.org:1234/whatswrong
The first thing you definitly have to do is Setting up a test suit for regressoin testing.
For those not familair with the term 'regression test':
Program a set of so called "test drivers", programms calling your code: routines/scripts.
Define test data, either in a DB or in flat files, used by those driver programs.
The test programs and test data needs to work with the old code, of course.
As the new code should behave similar you only need to adjust PATH or script names to let the test programs work with the new code.
Plan your project by defining which test cases(test program plus test data) should work at a planned milestone with the changed code.
After making changes rerun all tests.
Well, there is a lot more you could do, but that above is minimum (basic software engineering, sorry no art involved here).
Regards,
angel'o'sphere
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
"Do such and such because I read in an interview that..."
"There's this cool buzzword called refactoring that my professor mentioned and so you should..."
"[blah blah blah] object orientation [blah blah blah]"
"I saw a book about Extreme Programming at the bookstore and..."
Don't stop development on the old tree and shift all work to the restructuring project.
Instead, leave most of the manpower working on the old tree as always. Take a small team of your best people and have them gather input from the others, while the others keep working on the old tree. Then have the small team outline the changes that need to be made.
Work on the changes while simultaneously working on the usual stuff. Say, 90% of your manpower should do what they do now, and 10% should work on restructuring things. One mouthful at a time.
When you have a mouthful that you think is ready, branch the old tree. Merge your diffs into the branch, TEST IT, and if things seem to work, land the change onto the old tree.
--
Mod up a post Rob doesn't like and you'll never mod again
Most of the computer industry now calls this Refactoring.
I would HIGHLY recommend the book "Refactoring" by Martin Fowler.
There is a number of things you should do here.
1. Document your plan and come up with an official PROPOSAL document. Allow others to comment on this document and incorporate fix all relevant issues.
I started using this under the Apache Jetspeed project and now a lot of other Apache projects are accepting this practice.
It really allows the community to become involved in your changes and encourages constructive feedback and involvement.
2. Break this into phases. You should NOT attempt to do this all at once. Each phase should be isolated and should consist of one unit of work.
Each phase should be branched off of CVS, worked on, stabilized, brought back into HEAD and tagged. You should then RUN this code in a semi-deployment role for a period of time to correct all issues which WILL arise with the updated code.
After this you can then start your next phase.
3. UNIT TESTS! If management (assuming you have management) has approved the time for this type of refactor then you need to take the time and write Unit Tests for each major component.
It is important that Unit Testing can sometimes be just as hard, if not harder, than the actual development itself.
In some situations you can avoid Unit Testing, some here are going to call me crazy for saying this but it is true. In a lot of high level applications, which are NOT used as libraries by other applications, you can bypass Unit Testing in order to increase development time. This is a dangerous practice but it is often outweighed by the extra functionality you will end up with in your product.
Anyway. Good luck!
Kevin
His comments about code reworking and rewriting have a lot of insight in them.
Here are some quotes from his article:
The point is this... The benefits of spending time rewriting code completely may be a waste of the companies resources, this is for you to determine.
His interview is here:
http://www.softwaremarketsolution.com/index_2.h
and his site has more information about the concepts here:
http://joel.editthispage.com/articles/fog000000
Code is unlikely to be that large. That would mean that you would have been typing 100,000,000 characters. I don't think that existing code libraries count, but original code does. Try to write a 1,000,000 character document and see how long it takes.
I second all that has been said about making sure that you really need to do this and that it is worth the time and risk. One sign that you may need to do so is an excessive reopened bug rate, where fixing one bug often creates another bug due to side effects and component interactions. If you decide that it is, then the three keys to success will be modularity, incremental rollout, and unit tests.
Modularity is probably what you're already thinking about. Go over the old code base, in a code review, and find where the same thing is done over and over either with copy-and-paste code -- the bane of crap engineers -- or with different code that serves the same ends. Look for repeated sequences in particular. Create a new library that encapsulates those pieces of code.
Incremental rollout is vital. Only replace small parts of your system at a time, doing complete retests frequently. Don't write a new encapsulated routine and then roll it out to each of the three dozen places in which it appears in the whole code base. Write the whole function library, with unit tests, and then start applying it to separable modules one by one, retesting as you go. Otherwise I guarantee the whole thing will fall apart and you won't be able to tell why. Ideally, you might set a threshold on the rate of replacement of old modules and work primarily on creating new modules with the abstracted logic.
Unit tests are crucial because, as noted, the messiness of your old code probably conceals a lot of necessary logic. We had this great phenomenon on Apple's Copland where people who had never used the old OS managers were rewriting them in C or C++ from the assembly source. When they saw something in the assembly they didn't understand, they just ignored it. Guess what -- the new managers didn't have any backwards compatibility. The only answer to this is to have a thorough unit test for any module that you replace, against which you can test the new version. This also confers other quality benefits, but during a rewrite it's critical.
Finally, once you have replaced a significant number of your modules, you will find that new levels of abstraction appear. The average size of each function or method will have shrunk considerably, and now it becomes possible to see new repeated code sequences that were not visible due to the old cruft. Move these into your new library modules and start using them in continuing replacement work. In addition, start going back -- slowly and incrementally -- through the already converted modules and replacing the repeated sequences with calls to the new abstractions.
Finally, figure out how you got into this mess in the first place. The worst programmer habit I know of is copy-and-paste coding instead of using subroutines. You can tell people not to do it, but some always will. Those people should be bid farewell -- you can't afford their overhead. Other common problems include lack of planning and review, a code first and think later mentality. Start moving your organization up the levels of the CMM and you may find that you wind up with fewer modules that need replacement.
Hope this helps.
Tim
in particular how do you write a large program? I work on a program that has over 3000 files that are over 30 to 300 pages each. The software does alot for and has been around for over 30 years and has just had code sloppily added to it. So how does one go about rewriting an application that noone knows all the functionality of?
Only 'flamers' flame!
Don't set out to refactor your codebase as one big project. Try and split up the code by functional areas and take them on one at a time. Now, this doesn't really work too well most of the time. You're going to run into way too many places where things are interdependant but try anyways.
If you can, resist changes to your database schema while you are refactoring code. Having both of these thing happen at the same time is pretty scary.
You say CGI so I asume you are using PERL, look into OO PERL, it's worth it. Even if you don't want to go OO all the way the 2 big things that can make your life easier are packages and layering.
Using packages, especially for things like DB access can save you tons of time and headaches. You have one place where you run a query and build a hash, all of your code calls it when it needs the data. HUGE adavantages here.
Layering your design is helpful ass well. I've found that you can do a lot of good if you have designed Data Access, Logic and Presentation layers seperately. All each one of these layers needs to do is take the hash ref passed by the other layer and do X with it. You can rebuild each layer at will as long as the data structures passed between them don't change.
chizor wrote:
My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code)... What suggestions might readers have about tackling a large-scale retrofit?
My advise to successfully accomplish the changes:
I led the development and migration of some very large mission-critical systems in my career. Too many programmers making decisions on-the-fly, totally centralized management, or a "leave the technical folks alone until they're done" attitude are sure recipies for disaster.
Good luck with the changes.
Merry Christmas, and God bless us everyone!
E
http://eugeneciurana.com | http://ciurana.eu
I currently maintain a code base of around 120,000 lines of php and html (written by myself in a long, hard year) and have had to "retrofit" it a few times.
I find that when it's time to do an "over-haul" it's generally best to:
1) Pretend I know nothing - redesign from scratch. Write out a spec with flow charts, DB table definitions, etc. - make it VERY DETAILED. Spend lots of time at it. More time spent here saves even more time later.
2) Ignore your spec. (See step three)
3) When a bug comes up, or new functionality needs to be added to the codebase, refer to the spec built in 1, and build to it, and then put in compatability wrappers to work with the existing codebase.
Make these compatability wrappers log their calling in some way, based on a global variable. This allows you to see when they're no longer needed simply by defining a variable in a config file and waiting a while.
4) You'll be slowly bringing the application up to the new spec - eventually you'll reach a point where it's easier just to bring the remaining pieces up to snuff than to build more abstraction wrappers. When you get to that point, you'll find most of the work is already done, just finish it to the spec and remove the compatability wrappers.
This can still be a painful process, but at least it isn't a "gun to your head"! This allows you to regression test your work as it's done, resulting in a more stable deliverable, and you can still meet clients' needs in the meantime without making them wait 6 months while you re-write all your stuff.
Hope this helps...
I have no problem with your religion until you decide it's reason to deprive others of the truth.
[-1 Flamebait]
First, read this essay: The Big Ball of Mud. It is an interesting look at why, when we all know that spaghetti, gnarly, twisted code is bad, that it happens anyway (hint: it may mirror your understanding of the problem).
Ignore the "don't touch it" naysayers. Even before it's done, it'll be much nicer code to deal with. You can make decisions with less nagging doubts. You'll code onward with gusto. You'll be able to accurately predict the names of methods without looking them up.
Test the current state of things at all points through the process. I'm hoping that you have lots of automated tests you can run everytime code is checked in; if not, make them FIRST, before the overhaul. You are majorly diverting the intent of the code at hundreds of points; you can run astray in so many places that the above naysayers would be correct. Constantly assure yourseleves that the code is working. Go out of your way to ensure that the code is buildable and runnable, even to the point of writing scaffolding you know will be soon thrown away.
Burning a little incense every day in obesience to the Gods can't hurt, and will make the room smell nice.
mahlen
Shantytowns are usually built from common, inexpensive materials and simple tools. Shantytowns can be built using relatively unskilled labor. Even though the labor force is "unskilled" in the customary sense, the construction and maintenance of this sort of housing can be quite labor intensive. There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth. Shantytowns emerge where there is a need for housing, a surplus of unskilled labor, and a dearth of capital investment. Shantytowns fulfill an immediate, local need for housing by bringing available resources to bear on the problem. Loftier architectural goals are a luxury that has to wait. -- from "The Big Ball of Mud"
Now, on coding standards and how to incorporate them into a legacy project. Your concern is NOT format. The format of your code, such as indentation, spacing, etc, should be the least of yoru concerns. Everyone has their own style, but there are wonderful tools that you can use to force everyone to a single style, ident and astyle just to name a couple. Use a wrapper script to the CVS (or similar system) on checkins. Force the code through an automated cleanup. check the code back out and make sure it compiles/runs as expected.
What you should worry about is how much your design team has embraced the "black box" design principal. Parameters go in, results come out with no "side-effects" that impact the remainder of the code. Make your code re-entry safe, i.e. stay away from globally scoped variables as much as possible.
Someone's going to give you the whole OO-Design sales pitch. Yes it's nice on paper, but don't sell out because something looks nice on paper. I learned this the hard way. I have a tendency to overdesign things. When OO, this gets to be really scary. I waste my time writing object classes for "everything" instead of simply designing the software to its functionality spec. Make things more "object oriented", "functional", or "blackboxed" when you find yourself repeating code elsewhere in the application.
Don't spend a lot of time with naming standards such as Hungarian, Modified Hungarian, etc. Find a style that you and your team is comfortable with for the Interface API level. Below the Interface API, be more lenient. It's likely that portion of the code will undergo many changes anyway.
And most importantly, document! This is the singlemost important issue of any coding project. Either force your developers to write docs as they go, use embedded documentation solutions, or hire a techwriter to follow you and your team around for a few months. Documented API is the quickest way to start someone off in the project, and a great way to keep track of the flow of the program.
assert(expired(knowledge));
When you look ahead to a large scale rewrite, there are few things you must keep in mind before even attempting such a task. If you expect to jump in and hope to tread water, think again. Unless you sit down before you start dividing the coding tasks up amongst the engineers, you are simply begging to go down the same road twice.
;)
:)
Before you begin, make sure you weight the cost of a rewrite against the cost of maintenance. Many engineers (myself included) feel so ashamed for poor code that they want to dive right in and fix it. Whereas this could potentially solve the problem as far as you can see it, it won't make the financial people happy after watching your team spend 4 times the money than they would just dealing with maintenance. In other words, make sure you are adding a hell of a lot of value with such a huge task. This is a good time to analyze the risk of a rewrite as well. How critical is the system? How many users? How will this affect any users that have built systems around your interfaces? Will they need to recode portions of their systems as well? There are many more questions to ask yourself, but these are generally the ones that will come from people outside of your immediate group (and usually right away.)
If you do decide to rewrite, there are a few things you should do before analyzing the current code base. Review the original requirements and compare them with any new requirements you may have now (or later, if you have any insight into the future, such as rumors of change in business practice, etc.) Once you have worked out any inconsistencies or pitfalls, it's time to compare the original design against the original requirements. Look for inconsistencies between the two. Usually you will come across a few requirements that didn't quite make it into the design as expected (if at all.) If the two are so far apart that there is no feasible reconciliation, your hopes of a rewrite are pretty much dashed - it's time to start looking at a fresh start. Hopefully, however, the design met the requirements adequately, with no "creative designing" put in place to get around poorly thought out requirements. Note that you should do all of this before you start going through the code. Leave the code out of it for now.
Once you have established that you can safely move forward (i.e., the design followed the requirements), hold code review sessions with your entire team, targeted at the original code. Ensure that the design was followed properly and that any bug fixes/workarounds/etc are clearly documented and easy to spot. Usually this is not the case, else you wouldn't be contemplating a full rewrite. This is by far the most painful (and time consuming) task, and is usually the point of failure in rewrites. If not enough attention is paid to the caveats of what you have now (and why they are there and not resolved), you will paint yourself into the same corner with the next go 'round.
Without spending too much time on that topic (which is easy to do, believe me, I've done my fair share of code reviews with large teams), I'll assume that you are planning on continuing with the rewrite.
Once you've made it past the reviews, it's time for a new design. You won't necessarily need to completely redesign the system, but you will have to (at the very least) document a plan that will provide a solid path to the new system. Even if your functionality isn't expected to change, it will, so a mild to moderate design document will be your life-jacket. This will be difficult, as it's no easy thing to feel like you're simply writing the same words over again, but stick with it: you will find new methods in this design that will help to avoid the traps you were caught in the first time around. Have the utmost diligence with this step. Review the design often, and brutally. Bring in members from other teams for sanity-checks - a fresh set of eyes can do wonders, even if they are unfamiliar with your product.
Now, onto the coding!
Whatever you do, do NOT use the original code for anything more than a guideline. Create new projects, start with blank sources, whatever it takes to start anew. When you rewrite a system, you are actually writing a new system that simply mimics the funtionality of the old. Your performance is going to change, your stability is going to change. Don't think for a moment that you will end up with exactly what you had before, only prettier. This is a pitfall that fools many an engineer, especially the engineer who trumpeted the idea for the change.
Finally, more than just about anything else, remember this: when rewriting a system, be sure to adopt a coding style that will prevent spaghetti code. Be true to a modular design and cleanly separate the functionality into logical groups. This is always easier in an object oriented language, but very achievable in the procedural world as well. Make sure that this time you document your code inline. In other words, instead of simply commenting your code to say, "This function does this, then that", say something like, "This function performs the following business rules, as dictated by requirement #x, conforming to the component design section 5.1." Tedious at best, but far more valuable than most people realize. Make sure you also document the bug fixes that were migrated into the new code base, also. Eventually, someone will want to review the code to make sure the same bugs that were once in the old system aren't going to rear their ugly little heads in the new system. Finally, unit test the living hell out of your code as you write it. This is the last ditch effort (before QA and integration testing) for ensuring stability (which is the main justification in the end for the rewrite.)
Anyway, sorry for posting a novel, I hope something I had to say here can be of some use. Best of luck should you tackle the rewrite!
-------
[McP]KAAOS
It goes from God, to Jerry, to me.
Put as much in possible into tables. That makes it easier to find, view, add new pages, menus, product categories, etc.
Of course there will be exceptionss that need code instead of tables. Have special "override" points that allow you to override stuff as needed. Make the "overrides" their own function. (Example: sub foo(row); if row.clientID=7, then color3 = blue;.....). The override points are a lot like "on_x" IED-GUI or database triggers/events.
If you store SQL in tables for a report generator for instance, there may be situations that tablized SQL cannot handle very well (or not worth adding new columns if only one or two instances differ from the majority). So make a routine to override each SQL section (SELECT, WHERE, ORDER BY, etc.), and another to override the whole SQL for the really complex stuff like correlated subqueries.
I have successfully tablized HTML forms also. It is a lot of up-front work, but is better than coding each from scratch if you have hundreds.
Table-ized A.I.
Noticed you have a lot of perl scripts, which is similar to the situation I've been in before. Come up with some agreeable style (i.e. indenting) but stay away from the semantic issues, because a perl script speaks identity loudly. Case in point, how do you reference an item in a list: @list[0] or $list[0], Both are accepted, and in way, both are intuitive, it just depends on how you're learned the ropes. End Transmission.
-jc
Ever since I saw that horrible, nightmarish Steve Ballmer video.... "DEVELOPERS DEVELOPERS DEVELOPERS DEVELOPERS DEVELOPERS DEVELOPERS... DEVELOPERS DEVELOPERS DEVELOPERS!!!!!!!!!!!!" Oh god lets call it anything but Developers! The image of a big, fat, bald, sweaty man keeps entering my head when I hear that word.
Treat your existing product as one big, well developed prototype. Not many people have that advantage in a large-scale development effort. If you can somehow put a function-freeze on the existing product, that will enable to you to develop the new product to match the functionality of the old product without trying to hit a moving target. When possible, scope out and implement your improvements and efficiency improvements at the same time. In many ways, this is the best way to do a large-scale project.
Unless you want to go and reformat all of your old code, leave it as it is. Apply coding standards to all new code. With old code, use whatever style is already present in the file. If you think that somebody can automate a way of reformating all of the old code in a half a day, do it... but don't bother wasting anymore time on it. Only considering reformating old code if somebody does a lot of work (e.g. bug fixing or functionality changes) in the future on the file anyway.
Yes, I know, it is nice to have a clean slate. But nice is all it is... it's not really necessary, especially for parts of the old code that doesn't get touched very often anyway.
Don't do it Man!
action:Leaps behind the nearest dune and holds head low in the sand:action
Just let it go! You've laid down your mine field (developed) and activated the detinators (deployed). Don't go walking back through and "sweeping" anything!
Kaaaabooooooommmmmmm! I expect to come back to your page to see camel parts everywhere and your little Nomad guy hanging from the cool purple font!
You are checking your backups, aren't you?
First, read all previous comments.
Second, realize that a "sweeping rewrite" entails just as much work as the first write. The only thing you will (should) get from current code is a functionalty demonstration. (I.e., the current version is a "throwaway" prototype.)
Then, take everything you know about software development, learn a lot more, write the requirements specification, get it approved, then start thinking about technology, languages, and design. Coding should be about a third of the effort - the middle third. (Final third is test/debug/test cycle. If you didn't know that already, then don't even attempt this.)
Actually that was kind of funny, but you would have to be familiar with said phrase.
No, Thursday's out. How about never - is never good for you?
Go to joel.editthispage.com. He has years of hands on experiance and when it comes to software management his advice is right on.
Chris
Chris
M-x auto-bs-mode
If it works don't break it. Pointless rewrites do nothing but feed the programmers ego and knock the company out of business. Its happened over and over and over and over again. Progress, not regress.
If you're sticking with Perl make sure that all the code runs flawlessly with warnings.
#!/usr/bin/perl -wT
and
use strict;
..both projects were >250,000 lines of code. Assuming you don't have the resources to redesign from the ground up, my advice would be to retrofit in a piece-meal way. Try and factor out key pieces of code and rewrite if necessary. Whenever you make a change, run the old and changed system side-by-side to verify you have not introduced bugs. As you progress, it makes sense to write scripts which can automatically perform this side-by-side QA. As you rewrite sections of code, use the Facade Pattern to maintain compatibility with old code you don't have time to rewrite. If you do things right, the Facade can be stripped out later if you do decide to change internal API's.
I've used this exact method (XP) for cleaning up a project as well as with new projects. It takes some work, but the rewards are far more than you have any right to expect.
If it's really just a case of redundant functionality, then it wasn't designed well in the first place, or later add-ons/bug-fixes broke the original design. That should be fixed. If you need to add more functionality, then design in important in avoiding breaking the current design.
It seemed like a pain in the ass when I first started, but a good design job save so much development time, time that could be used later for documentation!
You can never put too much water in a nuclear reactor.
First of all, I think its important to realize that you have a medium-sized website and not a big software project. Therefore, some of the above comments recommending refactoring, UML, and eXtreme programming may be a bit overkill.
Web programming != software development! Its usually done at a much faster pace. Even if an object-oriented approach is taken, you are still probably talking about simple function libraries rather than complex C++ or Java classes. Again, overkill.
150 files is still a small enough project to be managed by one or two decent coders. Actually, I just looked at the amount of stuff I've written over the years for my online bookstore and its more like 500 files and over 4 megs of code. I don't feel like its too much of a job to manage this codebase by myself.
So, here are my recommendations.
You probably have gotten better at programming since the time you started your project. Take a few of the most recent CGIs you have written and compare them to the first ones you wrote. You just might notice a glaring difference in the quality. Also, the first pages you wrote are likely to be among the most important in your project, yet they are also likely the worst quality-wise.
Regardless of what language you program in, I think its important that you can tell whats going on in the program by reading the comments. If a manager can understand what a program does by reading the English bit, there's a good chance other programmers will be able to jump in and help as well. One specific rule I also follow: if you do regexes, say IN ENGLISH what those regexes do. I say this because regexes are one of the hardest things to read.
Look for any code that can be "factored out" of your scripts and put those into function libraries. Then include those in your program. The only problem with this occurs when you have huge function libraries that slow down your scripts when you include them. In that case you would logically separate your functions into different files. I have included very common functions in different include files, so I can make the actual code compiled or interpreted as small as possible.
Consider using a flowcharting tool as an aid to programming and/or documenting your code.
Standardize how you name variables and functions, write comments, identation, and spacing.
Be sure and include the date you write your scripts in the comments, in case the filesystem wipes this out.
I'm sure theres other things I've left out, but following the above guidelines have helped me do exactly what you are trying to do: manage a growing codebase. But don't forget, this is web programming, not rocket science, and some of the above suggestions may be more trouble than they are worth. Keep it simple.
No, Thursday's out. How about never - is never good for you?
One of the success factors they found was documenting the interfaces for each and every call between modules. The documentation turned out to be excruciatingly precise - but this led to zero ambiguity (and thus 100% interoperability). It also required meetings (sometimes arguments) between programmers to hash out what was actually going to happen. Another factor was that they decided to allow zero 'overloading' of functions by different modules. A programmer was not allowed to duplicate someone else's work, nor create a second, incompatible version of a function provided in a different module. If the function was provided by someone else's module, the programmer had to call it (properly). The result was that they reaped the benefit of object oriented programming - reuse and refinement of modular libraries.
It would be better if you could get the real scoop from the real programmers - but this might give you something to think about.
"The most sensible request of government we make is not, "Do something!" But "Quit it!"
I would set up a Slashdot-like Weblog in order to collect people's learning during code review, and be able to browse and search the results later.
This advice makes sense if you are embedded in a culture like Microsoft, where you have a huge quantity of legacy code, and a really shitty software process. Microsoft has no record of what design decisions were responsible for making their code the way it is; they just have the code itself. They can't change it because they don't know what they might break by doing this.
If you enforce a good software process for every line of code that is written, then you have more flexibility. For example, suppose that you use Extreme Programming. Then you will have a unit tests for every bug that was fixed. Each time you re-run the unit tests, you find out if any of your previous bug fixes have become undone. This gives you the freedom to refactor your code whenever it needs to be refactored.
Doug Moen.
I have written a truly remarkable program which this sig is too small to contain.
Baloney.
Contrary to what lots of people around here seem to think, especially those who like to make wide-sweeping declarations about what things are and what they are not, Perl is a tool. Nothing less, nothing more. Like all tools, it can be used to create well-engineered systems, and it can be used to create crap.
The community that grew up around Perl is all-welcoming and generally free of elitism. That's why a lot of newbie programmers and "system administrators-cum-developers" use Perl -- because they can without getting crapped on by others who think they know better. As a result, there is a lot of ametaurish-looking Perl code out there, but that's not a result of the language, that's a result of the all-inclusive set of people who use Perl.
Let's be clear: If you write code in any language and the code sucks, it's your fault, not the language's -- the language is just a tool.
Don't blame the problems of programmers on their tools.
Easy, automatic testing for Perl.
Some here are warning you that major changes always require a total rewrite; yet in real life, total rewrites result in inability to compete (look how long the Netscape rewrite paralysed Netscape, unable to meet Microsoft's challenge!). There's some good discussion of the danger of rewriting at a former MS software engineer's site, and some limited advice about how to get away without doing it.
But you've decided to rework rather than rewrite, you say, so I have no doubt you'll ignore the naysayers here. So what CAN you do? After all, as you recognise, reworking is dangerous!
The following rules have worked for me; I've refined my own experience with advice from Fowler's Refactoring, a book as useful as Design Patterns, and with study of Extreme Programming, a design methodology forged in the traditions of Smalltalk, and in the knowledge that maintainance, the most important and expensive part of software engineering, is also the least studied.
First, do the simplest thing that could possibly work. Don't EVER take your program out of commission for more than a day; make sure it runs at the end of each day. If you're doing something and at the end of the day your code base is broken, STRONGLY consider throwing away your changes and going back to the design stages.
Second, rely on unit tests extensively. Start every change by writing as extensive of a unit test as possible. Unit test every function you touch, BEFORE you touch it, and after. Unit test every change you make, and run the unit test BEFORE you make the change to ensure that it fails (i.e. it detects the change). Write your unit tests BEFORE you write code, whenever possible; you'll objectively know your code is done when your unit tests pass.
Third, don't design too far ahead; you don't know what tricks the old code is going to throw at you. Implement one feature at a time, bringing the code into compliance. Once everything has a unit test (thanks to your following the above principles), THEN you can safely embark on larger design changes -- and in the meantime, you have working code with new features, a win even if your customer/boss/manager decides not to continue.
Fourth, don't be afraid to redesign your own code. The stuff you wrote has more tests, so it's safer to change, but it's more likely than the old code to lack some critical understanding only age can give.
Fifth, use the principles of refactoring. Whenever possible split each code change into two parts: first, a part which changes the structure of the code without changing its function (and which therefore allows you to run the same unit tests); and second, a part which uses the new structure to perform a new function (thereby requiring new unit tests).
Good luck. If you want more advice, read up on Extreme Programming.
-Billy
Me too!
:).
Kidding aside, I wish I could mod the parent up to 6. Tim's right on. Probably the biggest mistake you can make in this kind of task is to bite off more than you can chew. And the more crud there is in the codebase, the easier it is to do! So take it slow. Refactoring is like orbital mechanics; you slow down to speed up
I recommend Martin Fowler's "Refactoring" book. It's geared towards object-oriented Java development, but the basic principles will apply across languages.
My company made the mistake of trying to rewrite our product from scratch. The "Tiger Team" effort got about halfway completed. Nobody knows if it would have actually resulted in a better product.
-Thomas
What a monstrous Christmas troll you are. What qualifies you to make this judgement? Perl, like any other mature language, has people who write kludges with it and people who write clean, elegant code with it. Your lousy Perl code is not indicative of a language problem.
That said, you're probably stuck with it, and AFAIK, you may be forging new paths in programming for reusability by applying the above concepts to Perl.
And this shows how much you know, since the Perl community is full of activity around design patterns, refactoring tools, unit-testing, and other practices which are in favor among experienced people trying to write solid, maintainable code.
My suggestion for those who are looking for actual useful advice rather than this kind of "throw away all your work and learn Java" crap, would be to head straight for http://perlmonks.org/ and read up. There's tons of advice there for serious Perl coders. You would also do well to start reading the mod_perl mailing list, which often has informative discussions about these issues.
Depend on tests, not documents. Documents lie. Tests don't.
Use as little documentation as possible, BUT NO LESS. (In other words, for heaven's sake don't ever try to get away without any documentation.) Documentation should state fundamental premises -- things like "The customer wants X." and "This code checks that I'm fulfilling the customer's requirement of X." Documentation should not state intrinsic properties -- the statement "this code does X" should be made as a test, not a document.
-Billy
If you need to make big changes to the code, then you are already hosed. If you have no record of what bugs you fixed, and no way of testing if those bug fixes are still working after you make code changes, then keeping all of your spagetti in place is no guarantee that you won't re-introduce bugs later on.
The Code Nazi approach is to write a unit test for each bug you fix. And you have an easy way to re-run all of your unit tests. After you make a big change, you can test if all of your bug fixes are still working. And now you have the flexibility to refactor your code whenever you want. Which means you can keep your code base clean and elegant, and you'll never reach the crisis point that this group has reached. This is the approach advocated by Extreme Programming, as well as by other software disciplines.
Doug Moen
I have written a truly remarkable program which this sig is too small to contain.
If you're going to refactor your Perl code, I would suggest pushing to have the code well documented with a tool like PerlDoc. One of the key problems that new people have with code is trying to understand the decisions that previous people made in the design of the code. Pushing more documentation into the code will help that in the long run. Plus, making documentation a standard part of your code will help make the code better as the developers will tend to throw out and/or rewrite code that they can't explain in the documentation.
I agree with your point and I'd like to expand on it.
Yeah, adding a gig of ram or increasing the CPU from 800Mhz to 1.8Ghz isn't expensive. But, if you go beyond what a singl-processor machine can handle, you run into another host of problems.
Adding a second CPU means *MUCH* higher chances of race conditions and other threading bugs. If you know you're coding for a single processor, you can often use a single-threaded model which makes life so much easier.
Adding clustering brings a whole hoist of data synchronization problems. It's *ALWAYS* easier to code for a single-machine than to code for a cluster. There are tools you can use to make shared memory easier, but those often flood the network.
Citizens Against Plate Tectonics
As someone who's been through what you're describing more often than I would have liked, I'll strongly advise you to bring in a consultant to oversee and direct the refactoring process.
;)
The reasons for this are many.
You bring in a fresh viewpoint, someone who isn't married to any particular design decision or strategy. This is one of the biggest stumbling blocks in re-implementing existing code.
You bring in rationalized realism. A consultant to oversee such a process is able to make rational and reasoned projections about timeframes and milestones, and can be held to them. Without a voice of authority it will often end up being a battle of wills between you and your manager(s), each side having it's own ideas about what is, and isn't, possible. You might tell your manager something will take 2 months, and you may or may not be right, and he may or may not believe you. But if a percieved "expert" says it will take 2 months, the manager will have no choice but to agree, and he's likely right to boot.
You'll learn something. You really will. My first interactions with consultants tended to be dismissive, after all, who knew my code better than I? But what they have to teach isn't technical so much as it is organizational and structural. I learned a great deal about how to get things done, skills which served me very well as an employee and now as a consultant myself.
Inviting in an outside professional is a difficult thing to do, it requires admitting that you need help, and freeing up additional funds to pay for it. But in a situation such as yours it will not only pay for itself, but will make your job much easier and stress-free. The only real problem is the difficulty of finding a truly qualified consultant with experience and insight into your particular situation. But that's a topic for a different reply.
Don't spend a big chunk of time refactoring it either. Waste of time too.
Instead, make slight refactorings as you go. But make sure you are doing what you are really being paid for: implementing business value.
And you'll find that you'll have much more courage to refactor if you have a full set of automated tests, so maybe you should work on tests first.
In this particular case, it was necessary because the site was right at the max. If the traffic increased, it would kill the site. Since it was an E-Commerce site, rewriting it was fairly straight forward. The old code kept running, until we were able to finish the new system and make sure it was stable and ready.
As a consultant, one of the most important aspects is detailed documentation that explains both the high and low level details. Often I will include very specific details about why a design was chosen and what limitations it has. When applicable, I will also describe how to extend, or modify the code to support additional features. This means you spend a lot of time doing documentation, but it forces you to think about a design more thoroughly and will expose weaknesses. Always keep an open mind and never fall in love with your design. There is no right way to build something, only right for the situation you are given.
You're probably planning on working solidly for a couple of weeks or months and coming out with the same product, but cleaner.
Well, consider all the (big or little) features you might like to add, and when you do your rewrite, consider HOW those features would work. That will improve your refactoring and also keep some fun in the project.
I think you should concentrate on improving your processes so that you never have to do this again. So, basically, catch up on all the unit tests and reviews you've never done!
Others talk about tests. That is a valuable practice, but keep it limited to things that are easy to test. If your tests are ten times flakier than the code because its really really hard to write a good test, well, fuckit!
I'd recommend a checkin-review process. Every should be reviewed for:
1. comments-missing-from-unclear-code,
2. violations-of-coding-standard,
3. corner-cases-not-covered.
Those things are pretty easy to discuss with developers, they wont be taken as criticism.
Checkin review REALLY DOES improve your code and it DOESN'T have to be onerous!
Obviously you didn't read this slashdot article about Joel on Software.
What is the #1 thing he says causes software companies to fail? Rewriting from scratch!!!
I won't deny that there are times it has to be done. Joel points out some of those time (and yours isn't one of them even from the little you've written). Ours happened to fit everything he said, and we did rewrite, and not only was it the right thing to do, it was the only option for us, but the company just barely survived the process.
Don't take that lightly. I speak with experience and Joel is right. You don't rewrite from scratch unless there is no conceivable alternative. Joel describes it well, so I won't "rewrite" it. You can just click the link and read it yourself.
There is NO decent evidence that OOP factors better, or does *anything* else better than procedural/realational programming (or other non-OO paradigms).
IOW, don't switch paradigms just because something is in fashion right now.
Look carefully before leaping into something that may just turn out to be a fad or only shines in specific domains.
Table-ized A.I.
is to move every other year. When you find yourself in the position where you need to rewrite a whole system, get a new job.
a good project management application is important for any development team. usually, these are hard to come by unless you plunk down $10,000 or more, although these come with a gazillion features that you probably won't end up using.
i discovered a new tool on sourceforge which is an open project written in php.
i'm impressed with it. the code is also well documented.
the homepage can be found here.
i recommend checking out the screenshots as well.
When you first design and implement some module, a
lot of time is involved in cycling between "ok, I
know what to do" and "huh, maybe not". I've found
this crucial, esp. in team work, in order to gain
a good conception of the scope of the task. Also,
many external issues, e.g. how the module interacts
with the system, efficiency, etc. that aren't pure
functional issues, are first grappled with here.
Refactoring is different from this, in that you're
probably very comfortable with the "state of mind"
of the code. Instead of creating, you'll be
clarifying. So, most of the refactoring is in
your head (99%). All the external issues have been
addressed before (or else this probably isn't really
refactoring), so just work at a white board with
your team until writing the code will basically
be transcription (1%).
I've found this to yield the best code.
Per my professional experience with an obsolete, out-of-date PASCAL on DOS system that keeps beeting the daylights out of would-be replacement systems, leave the convoluted, undocumented pile of spaghetti as-is. If your people have too much time and money, they could productively spend it figuring out what everything does and documenting it properly, but the mess that works is priceless compared to the alternative!
Certainly one of the keys to Microsoft products "working" is when they make well defined interfaces and well documented interfaces. In my experience, if you define your interfaces well and document exactly what they are supposed to do, no matter how terrible your code is, you can make it work, and make it flexible to later changes.
This is where a language like perl is not extremely conducive to well designed systems, however you can of course design a system well in perl.
- Adam
If a penguin dies in the woods, and nobody is around to hear it, what sound does it make?
Joel Spolsky has an article up on his blog site that speaks to this point.
He uses Netscape's decision to rewrite Netscape 6 from scratch as an example, and expands upon many of the points mentioned above.
Do it better! Do it faster! Do it cheaper!
/* Jeebus H. Christ, I quit! */ on top of an odd looking hack. Understand why things were done they way they were done, because, no matter how well your design, you too, will have to make clever hacks and changes upon encountering real world impasses.
:)
Just don't forget to convert from inches to centimeters, and vice versa. (:
Nasa jokes aside, reconsider doing it in Perl. Perl, while a jewelled spectacle upon the crown of Language, is glue. It sticks things together. It fills in cracks and niches. But, like glue, if you just sit there with a handful of it, it's going to be sticky.
Before you touch *any* code, make sure you read the existing code. Go over it. For weeks, or even months. Understand why programmer X added a comment saying,
Designs always look great on paper. It's when they hit the real world that they go to hell.
You should never do something that will require a long time/resource investment unless you have a good reason. Is this just some clever way for management to justify firing the developers because they won't have accomplished anything at the end of this rewrite?
If you don't have a reason, you can try to find one. You might identify things that are truly broken about the system, and try to fix them. A typical problem with a lot of code is that its not as general as it could be (not portable, or you can only run one instance at a time because of some common resource or something like that.) By finding a concrete reason you will be able to direct your efforts and have a way of assessing your results when you are done.
Well of course you are quite right: it is a catastrophe to hide tens or hundreds of cases in a nest of code. The logic of a program has to be visible to the maintainer and this means using a language that is appropriate to the problem at hand. A tabular language might be the right one for a particular problem.
It should be observed that there are unpleasant performance implications to doing things this way. For performance reasons it would be nice to have that stuff hard coded! But how do you resolve these two wishes?
You use a code generator that builds code (which is later compiled) using the tables. Now everyone is happy. Speed plus maintainability. What approach should be used for code generation?
You could build your own code generator by hand, spitting out program text by generating strings. But that is not very nice. There are many ways to screw that up both in terms of correctness and in terms of the power you provide to the programmer who uses the code generator.
And so we arrive at the point of this little screed: a macro system is the correct approach for creating a little language inside of a large program. This is one reason why programmers should care about the macro facilities of a language. Perl and Java have none built in. C has the unpleasant cpp. Lisps have lovely macros, making it absurdly easy to do tabular programming, complete with breaking out special cases easily.
I have just carried out a similar project, and one thing I learned to beware of is don't suggest any timelines until you know *all* the functionality. You may think you've explored the site but there may be a lot of hidden functionality you don't know about, eg xml exports to affiliates.
I disagree with the "code doesn't get rusty" comment. Often there is a lot of pressure to get a project out the door and the initial launch version contains many compromises the designers didn't want to make. I don't think there is any project I could look back on and think "hey, the next time I would do it *this* way". People learn from other people's (or their own) mistakes, it's how we move forwards.
In response to the person that said don't rewrite as you will have to rediscover all the little quirks that the old site fixed: by doing a rewrite and documenting the little quirks as you find them means you will have a record of them rather than staring blankly at a screen of code wondering why on *earth* the previous coder did *that*. By rewriting from scratch you won't miss any quirks, which then come back and bite you inevitably at the least convenient time.
As for the Perl vs Java vs PHP debate, I'd say use the one you all know best. No language is any better or worse than any other. Some have libraries to make life more convenient, others have cleaner OO support, but in the end if you all stick to the accepted coding standards then half decent programmers can write maintainable code in any of the languages. Just bear in mind that your replacement will need to to know the same language so don't pick something obscure or dying where there will be no pool of available talent.
Phillip.
Property for sale in Nice, France
The term "re-write" scared me when reading your query. In large scale systems, especially of the size you're talking about it's best to re-implement certain portions / areas that can be broken apart and handled independantly.
If you wish to convert into a differant environment then treat it as a seperate project from the re-write. If the conversion to a differant environment is not straight-forward ( I don't have a lot of experiance with CGI / Perl.. ) then write implementation wrappers for the missing / alternate functionality so it's merely a simplified change in the function naming rather than a re-implementation.
If you need to do both - handle each independantly. You'll thank yourself when an important requirement that marketing promised pops up when you're in the middle of it.
Seperate, Seperate, Seperate - There are no huge projects - only people who are too all-encompassing to break them down into achievable sub-goals.
Keep breaking them down until each is achievable in a months, 2 months time. None should go over 3 months even for the worst case. Ideal time-spans would be 1 month periods.
There's a gorilla from Manilla whose a fella that stinks of vanilla and has salmonella.
I saw a quote once that said something to the effect of you can always tell a computer scientist's code because he won't have any comments: his variables are his comments (something like that, not sure of the exact words). While descriptive variable naming is important, excessive variable naming is annoying. For one thing, it's a lot easier to type no_acc_click() than it is MakeNotAccidentallyClickable(). Comments are necessary. When code does anything of even a slightly complex nature, there's no such thing as intuitive code. There will always be a need for comments in anything beyond the simplest programs. The best code is properly indented, case is consistent, variables have descriptive but not lenghty names (i is sufficient for an iteration variable, malloc is sufficient for a memory allocation function, etc.), a description at the beginning of the function of all expected uses and known bugs, and last but not least, comments of each change in the code, ChangeLog, and CVS logs.
A solution to the problem with music today
Word to your mother. Code reuse via copy-n-paste is one of many basic software engineering blunders that is committed constantly in, I would guess, most software development organizations.
Ultimately it's a culture problem. The more junior software engineers in a lot of places just don't care about quality, so they don't pay attention and learn from each other and from more senior people. They don't criticize the flaws that they are no doubt observing in each other's work. A few senior staff can't do all the teaching and critiquing; it has to be an effort from all the developers.
At some point you have to blame management for creating perverse incentives that ignore or discourage high quality results from developers.
I want to start by saying, before you do ANYTHING make it a priority for your team to start writing test cases and implementing them. If you are going to change your system you want your tests to find the problems not your users. Once you have a test suite that effectively tests your system (you will be surpised how quickly these tests will lead to enhancements in your software BTW) then you need to decide if you really want to maintain two seperate teams.
One team will be doing maintainence on your old system that is in use. You can't not keep this team in place, you have to keep customers happy while you do the new thang. You need the second team developing the new sytem (their job will be difficult with good tests, but impossible with out them.) Without tests they will have something that looks kinda like the old thing, but behave differently in a million small ways, some without a doubt will be better, many will not.
The other thing, and more sensible IMHO, is to tell pairs of programmars, to spend some hours each week working on the system together. Pairs becuase this is an excellent opportunity for people with deep knowledge of their systems to share it with other memebers of the team as they ferret out ways to refactor the code so more of it can be shared.
One day a week you can have a team working on some part, documenting spaghetti code, or pulling out examples of where code was cut and pasted and create a function which is called from both places. The stated goal should be fore each team to remove some number of lines of overall code from the project each week.
Remember the metric is the more lines of code you have the more errors you have, make it a mandate for your team to be removing code as well as adding, actually on our team we don't brag about how many lines we added in a week, we brag about how many we removed.
Anyway, the idea is by getting teams to do pair programming looking a specfic areas with good tests you can whip your code into shape pretty fast, and spread the knowledge of the system around as well. AND instead of a jarring change when you unveil the new system you can have gradual changes as fixes are tested and added.
Obviously if your overall architecture is broken and can't move forward this won't work. But it seems to me you are not fully convinced of that fact yet. At the very least by having teams look at code they may not be overaly famililar with you might find that team members can cross pollinate and share good ideas.
Jer,
There's a concept in the world of stuff you can touch called 'depreciation'. Why the software world hasn't caught onto this is beyond me, but from my experience it seems to apply reasonably well.
The value of your code goes down in value over time. Now, don't confuse that with the value of your design - your design could be grand, but the code is part of your equipment, like your hardware. Depreciate it over time to reflect the increasing cost of maintenance and integration costs in a migrating business.
Reworking your code allows you to make adjustments to your design to reflect a new environment, or to move away from languages/APIs/toolkits that might be hard to maintain.
I depreciate my code over about 3 years. It's all modular, and I replace code about as frequently as I add. A number of years ago, I had tons of bandwidth but not much CPU power, so I tended to push data rather than compute. The reverse is now true, so I made some design changes as part of a standard rewrite - no need to wait until it broke. Overall, most of my code hasn't radically changed in it's design, but it has been rewritten several times. I've had code cut back to 10% of it's original size by adopting a new toolkit, etc. I've made it more robust, faster, cleaner, better documented. I can't think of a case where it's gotten worse, and I can't think of a case where the rewrite took much longer to write than the original code - and more often than not took much less time. It's worked well enough that I've added a considerable amount of functionality, but spend no more time reworking code because it becomes increasingly efficient and is never too far removed from future additions.
Many people suggest that code rewrites are a waste of time, but it's a maintenance function. People that only budget time to write new code often find those extra work hours devoted to maintenance. Budget it in - and the best way is through rewrites.
Why?
What is pushing you towards such a drastic move, and what do you hope to gain from the effort?
The answers to these questions might make answering Chizor's inquiry easier.
Chaeron Corporation
I entirely agree with Joel's claim and I think the poster was correct to quote him. But notice that Joel doesn't say it's always a bad thing to rewrite your system - it's only bad to do so if your motivation arises from the specific reasons he mentions. IMHO, the original poster has more fundamental reasons for wanting to get rid of his current code base.
If the orginal developers of the code (be it yourself with less knowledge than you have
now or someone else) wrote the programme in such a way that its absurbly bloated, poor designed and
all the rest of it then theres nothing wrong
with the 'start from scratch' mentaility. If the
people at microsoft decided to redo dos in the early days and get rid of that 640k limit, if they
redid a lot of there less successful product lines
(there is a lot of them, like ms publisher, etc)
then these products wouldn't be useless.
The entire prototype theory comes from starting
from scatch. Its important cycle in software development
I have been through this before - with a different language, but the same principles apply. Here is some adivce, and some ideas:
1. Don't do a 'big bang' rewrite. A Big Bang rewrite is one where you totally rewrite all the code, but have no functional - or very little functional difference.
2. Do refactor or rework small portions of the code. Work on the code as you implement new functionality that the system needs. In other words don't waste time on rewriting working code unless it needs additional features or is broken.
3. Do Introduce Unit Tests and Acceptance Tests. This means you will know quickly when changes you make break code. Don't do a massive "unit test" building exercise however. Write a Unit Test for every peice of new code you do, and every peice of code that needs fixing.
4. Identify "Problem Code". Problem code is the bits which seem to take up most of the maintenance time. Identify that code, and try to work out ways to fix it.
Note : Some people think they are 'clever' by writing complex code which is hard to understand, but may perform better etc. This is the kind of code which usually ends up being a problem. As a professional programmer I try to be clear to understand - not only for the benefit of others, but for my benefit when trying to maintain it.
5. Read about how to manage projects. Extreme Programming is popular now, and I can say that it has some very good ideas. However don't become religious about it, and take a look at some of the 'classics'. I can recommend:
- The Mythical Man Month (tad dated, but a classic).
- Code Complete (also a little dated, but still good advice)
- Extreme Programming Explained
To take a look at my Open Source Resource, which has many articles around the subject of project management, take a look at:
Devcentre
if it works... don't fix it.
If you feel you have to fix it, then prioritise the most problematic parts and fix them according to a set plan/policy. Use a naming and calling convention. Break functions that do more than one thing up into component functions that can be tested, verified and reused by other parts of your program. Fix it incrementally not all at once. Try using an interface contract when you make objects; that way, your new functions can call new methods and the old code can depend on old methods to be there. Deprecate the old methods when there is no code that depends on it. Don't forget to comment - comment the code then come back the next day and read your own comments. Make changes to the comments so they make sense today.
Blah... blah... blah... Etc... etc... etc...
Codifex Maximus ~ In search of... a shorter sig.
Yes, so if you want to write code like Microsoft, you follow their principles. However, unless you have the legal team and monopoly position to go with it, that strategy will probably not work as well for you as it does for Microsoft.
Deciding to completely rewrite your product from scratch, on the theory that all your code is messy and bug prone and is bloated and needs to be completely rethought and rebuild from ground zero.
Well, he is right to the extent that if you use the same tools and the same skills, you are going to end up with the same mess. Given that Microsoft has been using the same primitive tools for the last decade, it's not surprising that rewriting doesn't work for them.
It's not like code rusts if it's not used. The idea that new code is better than old is patently absurd.
Again, that's Microsoft-think. Sure, for Microsoft, things don't change unless they change them. For the rest of us, code does "rust", because the environment changes. And for the rest of us, there are new and useful tools and programming languages that we can use.
There's a few things that I'll say. One, not a single person posting here really knows what your situation is (well, unless your coworkers read /. and are posting advice for you...). I don't want to be mean, but most of what's been posted here is likely not applicable to your situation. For starters, ignore all the threads arguing over languages: they'll never agree, and in the meantime you have work to get done. Next, I would suggest ignore all of the posts from people who aren't talking about web sites. There's a lot of methodology recommendations there, and it takes a long time to really understand *any* methodology. So if you don't already know it, it won't help for *this* project. If you're interested in that kind of stuff, read up on it; maybe it'll be useful on a future project.
:)
Next, no matter what you decide, from your current position, there's substantial risk involved. If you don't have a good way of estimating the costs and benefits of your alternatives, then you will wind up shooting in the dark no matter what you choose. This isn't really unusual, but it can be stressful! Somebody in management has to decide if they want to play high stakes poker with this project. This will establish how many nice risk mitigating activities you can budget (like unit tests, code reviews, documentation, etc). One thing a lot of technology people have difficulty grasping (this isn't directed at anybody in particular), is that if your management decides on a high risk course of action, it's not WRONG because it's technically inferior. Now, if management doesn't understand that it's a high risk proposition, then there's a communication failure somewhere that will screw you no matter what gets chosen.
Ok, now that you've decided where you're going, I've got 3 pieces of useless advice (all advice is useless advice, btw...)
1) Know your priorities. This is the most important thing to getting ANY solution.
2) Understand your requirements(I don't care how you do this). This is the most important thing to getting (close to) a CORRECT solution.
3) Remember it's only a job. This can be very important for retaining sanity
when you inherit 70000 lines of code that do not work and the lead developer was logging only one system update per month but actually changing production code every few hours...
... switches and vastly simplify logic of price line assembly
... and ensure no crossout when Just Reduced third price
... is blank (no sale beyond discount)
... modification pull number being reverted by offer or
... exporting option (via check box)
... in stripping some strings from psuedo-price-line, so go
... back to promotion record for Just Reduced flag
... Discount price does not get proper style sheet, remove
... placement of unnecessary tabs, and kill $space_char
... for other types, but mistakenly put two tabs after
... full price and none after discount, oops)
A va ilable)>0)
_ Override_Price_Mod))
o ti on]Available)>0)
P ri ce#"")|(vp_Priceline_Third_Price#"")
P ri ce#"")&(vp_Priceline_Third_Price#"")
o ut _Mode}
u t_mode="All Selections")
e duced_selection)
t _or_sale_exists
r os sout_Format}="Under Pricing")
those pearls of wisdom will really hit you.
practical matters, bullet points and examples:
1) write code to be read
and if means going against the culture, fuck the culture of perl or c or whatever... believe in karma--got that urge to be clever, you'll get yours someday, trust in the dark side.
2) comment your readable code
but strip out old, boring comments (after they are petrified, before they are eroded by time)
3) put interface documentation in the interface
4) minimize documentation
face it, documentation is great for training new users and system roll-out, but it falls out of step fast and will be of minimal help to coders when the shit hits the fan.
if you cannot afford to have developers who think they should be managers sitting around keeping the documents ship-shape, or (miracle-of-miracles) have a writer on staff, your docs will be a joke.
LAMENESS FILTER AREA
** big fuck you to everyone who thinks shit like this is too cool... (declarations omitted)
$done:=False
Repeat
$position:=Position($3;$4->)
If ($position#0)
$2->:=Substring($4->;1;$position-1)
$4->:=Substring($4->;($position+1))
$done:=True
$0:=1
Else
RECEIVE PACKET($1;$receive;(32000-Length($4->)))
If (ok=0)
If (Length($4->)=0)
$2->:=""
$0:=0
Else
$2->:=$4->
$0:=1
End if
$done:=True
$4->:=""
Else
$4->:=$4->+$receive
End if
End if
Until ($done)
** refactoring can be fun, let everyone in on the party. very few people will know this syntax, but nobody should be mystified...
` spb20011109 standardize on vl_JRDC_position (kill vl_JRDC_Fposition)
` spb20011109 old cycles (2001) gone, remove $simplified_crossouts
`
` spb20011112 add Crossout Mode and Format interface menus
` spb20011119 fix bug where $second_price not initialized
`
`
` spb20011127 add redtag format menu
` sbp20011127 update redtag logic to consider case of price
`
`
` spb20011128 simplification of sale price method resulted
`
`
` spb20011212 fix tagging issue with Just Reduced where
`
`
` spb20011218 fix tab placement for Just Reduced (good
`
`
C_BOOLEAN($discount_or_sale_exists)
C_BOOLEAN($discount_and_sale_exist)
C_BOOLEAN($crossout_required)
C_BOOLEAN($just_reduced_selection)
C_BOOLEAN($redtag_selection)
C_STRING(16;$crossout_mode)
C_BOOLEAN($crossout_under)
C_BOOLEAN($redtag_graphic_required)
$redtag_selection:=(Position("RTAG";[Promotion]
$redtag_selection:=$redtag_selection&(Not(Offer
$just_reduced_selection:=(Position("JRDC";[Prom
$discount_or_sale_exists:=(vp_Priceline_Second_
$discount_and_sale_exist:=(vp_Priceline_Second_
$crossout_mode:=vDigs_Crossout_Mode{vDigs_Cross
$crossout_required:=(($crossout_mode="Redtags Only")&$redtag_selection)
$crossout_required:=$crossout_required|($crosso
$crossout_required:=$crossout_required|($just_r
$crossout_required:=$crossout_required&$discoun
$crossout_under:=(vDigs_Crossout_Format{vDigs_C
Hmmm... reminded me of an article about code rewriting that slashdot posted a while back.
Slashdot Story: How To Make Software Projects Fail
Relevant interview link
Social scientists are inspired by theories; scientists are humbled by facts.
For example, most enterprise applications do something important -- trade shares, manage accounts, track patient records, etc. How it's done is governed by business logic -- the company rules, policies, regulations, procedures, and so on. Now, you can spread this logic across your web site (as one might do using PHP, which is tied to the web site). Or you can bundle it up into an independent application, keeping all of the business logic in sensible, cohesive compartments that run on an application server (e.g., using one of the existing Perl app servers or one you've rolled yourself via, say, POE). This not only makes the business logic easier to understand and manage but also makes the logic independent of and accesssible from any number of front ends that you might need. Simultaneous web, client-server, and even command-line interfaces become possible, and for enterprise projects, multiple simultaneous interfaces is often a requirement for backward compatibility with older interfaces.
In summary, for shallow web-only stuff, PHP is a reasonable tool. For stuff beyond web work, PHP is out of its design envelope. However, Perl works here just as well as it does for web work.
Easy, automatic testing for Perl.
Let your next feature show you where.
The project I'm on now is doing exactly the same thing you are contemplating.
We are letting new features drive the cleanup. When it is time to add something else, change the existing system to make it easy to add the new piece. Maybe this means touching a substantial number of modules to tweak them to use a new helper function. Fine, you're not changing routines for the fun of it - you have a definate goal in mind. Maybe everyone needs to use a new html formatter. Maybe everyone needs to use a common databse interface. Whatever. When consolidate something, really consolidate - hunt down every thing that is like it and adapt the existing code to use your consolidated routine.
I would strongly encourage you to do it this way, because if you embark to the Great Cleanup it might be a long time (many many months) before you can put new business value on the table.
If you do it this way, it can become part of your team's culture - you should always stop to clean up code every now and then. It is an ongoing process, and it gets easier the more you do it (kind of like laundry or dishes... why wait until life is unbearable to clean up a little?).
Letting features drive your clean up does the following for you:
helps keep you focused (reduces the chance you'll get lost in the woods and wonder around forever).
cleans up areas of the system that are likely to change again soon
finds the most important code
makes it easier to change next time
make it easy for you management to have faith in you.
This last one is really important. If they get regular improvements to the system, they stop worrying/fearing what you are doing and start fighting to shield you so you can keep doing it. At least this matters in a for-profit environment. Make it easy for you manager to believe in you and you will make your jobs easier.
And scope it small - small bites, easy steps, and release often.
And buy the Fowler Refactoring book, and the DesignPatterns book wouldn't hurt... but that helps C++ and Java people cope with brittle type systems... this isn't such a problem for you in Perl. You would benefit more from the principles in the Refactoring book, even though it is in Java.
1. review all of your existing code and list the things that each script does
2. decide how to consolidate and divide out these tasks, and make sure that any functions or data that needs to be shared, or may in the future need to be shared, is separated out into modules
3. document all of the programs to be written, their interfaces and their effects
4. then, and only then, begin coding
-jeff
-- Two men say they're Jesus. One of them must be wrong. - Dire Straits
The most important factor is having a ruthless project director. You don't accept slippage, you don't allow mistakes.
However, a good project director understands the project and the time it will take, and doesn't curry favour with management by suggesting impossible time-lines in the first place.
Beware the project director who comes in in mid-stream.
Duncan (not the MacBeth one)
wouldn't suggest using it for a large project.
And I know some serious NT system administrators who ftp from arbitrary terminals to their production servers, using their Administrator accounts.
Sounds to me like someone did a bad job of designing your system. Although I obviously don't know the details of your project, you may want to consider starting over. If you need to do a retrofit, then some bad mistakes were made at the start of this project. Doing any kind of retrofit can only make things worse. If you have time to do a retrofit, you'd be better off taking what you've learned from this failed project and doing it right this time. I'd start with losing perl. I wouldn't touch it with a thousand foot pole for a large project. Then go back to the design phase, and do it right!
One example is one client wants a column in a report added up. fine. no problem. It was desigened to do that. Problem is that column is user supplied and not numeric. So the rule is something like an orange counts one and an apple counts 2 if its one type and three if its another.
Your saying you can plan for this kind of crap and design it in? This is a one off and perl makes that easy. In fact the orginal code spawns a 'report' and if checks in ~luser/bin/final_report and runs that if its needed.
You can never underestimate (or plan for) the lame request that some paying customers will want.
But de important part isn't the indentation style. The important is that it describes the prefered constructions to solve certaint types of problems. It describes the rules for a good, uniform implementation.
And DO NOT make it a book of rules. For every rule there has to be argumentation why it has to be this way, including examples and they main advantages of the rule (improves robustness, maintainability, clarity, portability, uniformity).
The coding standard will help new and unexperienced to build good solid code. The professionals will like it because after reading the coding standard they know what is kind of code is expected from them.
And, maybe most important, will keep the hackers under control. Hackers won't adhere to rules just because they are the rules. But if they know why the rules are as they are they will use them.
I was a hacker who was coding only his way too, I didn't like rules because they made no sense to me. Right until I read Patrick Savalle's (Synalyse (sorry guys, dutch only)) C coding standard, that one made sense.
I am still using it, both for private and professional work (on most jobs there isn't a coding standard, the one I bring is very welcome and if there is an coding standard it is mostley only about identation so those are replaced just as easy).
Just my EUR 0.02, Hans
...and name the function so that the comment is unneeded
The only function whose name makes the comment unneeded is "exit()".
"my first software released under GPL (November 1988) -- now the Slashdot kiddies have to shut up; I was writing GPLed software before they were born, bwa ha ha!"
Posted Thursday, December 13, 2001 on his site..
He has alot of good thing to say about software management and programming.. read and learn instead of getting caught in Microsoft bashing..
I'm currently doing something similar- the program I maintain is Mastering Tools, which has long been an over-featured, densely packed program operating on a text input basis with a certain amount of visual feedback on the text input.
However, I've long known that there are two things my real users (ideal users? the serious mastering engineers) want: familiar interface and realtime processing. I can't deliver realtime processing without literally doing the whole thing over again in a language I don't know (it's done in REALbasic, which is little better than a scripting language for speed, though it's got really, really nice prototyping abilities and GUI support.
However, the time's come to completely overhaul the interface, partly because I have some ideas for mid/side processing and don't have any room in the current layout to fit them! The ideas have to do with rectifying the side channel and using it to either enhance or remove signals that aren't in both channels equally- where regular mono is a 'node' that totally eliminates out-of phase content, it's also possible to completely eliminate R and L-only content and keep only R+L and also the out-of-phase content.
That part's the easy part- it's simply signal processing (and will be duly released under the GPL as soon as it's done, as always). However, the interface is asking for a total, complete overhaul in several ways, and that's what's taking all my effort currently. Here's the situation and how I'm handling it...
Layering. It's no longer possible to fit the whole app in a 640x480 area, even with small print. There are several possible answers to this: one would be having separate windows. This would be more adaptable to larger screens, but it's untidy and there are issues with closing windows and still referring to controls they contain- so that's out. What's looking more reasonable is tabbed panels- RB implements a nice little drag and droppable tabbed panel control that appears quite easy (though I had some trouble attempting to do nested tabbed panels- after one experience of having all the nested panels (at identical coordinates) switch to the top panel of their parent panel, I quickly gave up on that concept. Instead I'm using more panel real estate and trying to divide the controls into logical categories. That is, of course, a real headache- doing interface properly is hard! (says 'interface is hard Barbie') It's only somewhat easier with the additional space. Complicating matters further, is the expectation of the intended audience here. It has to both be organized and look and feel like a mixing board, amplifier or rackmount box of some sort.
Solution: implementing controls like knobs and meters. It's actually quite fun to code a knob appearance out of graphics primitives- and surprisingly hard to get mouse gestures to work on the damn thing- using a two-arg arctangent routine in RB that I don't fully understand. I also have a single meter control already implemented for azimuth display- think that it might be best to tear that apart and re-implement it in a more general sense.
That's because the Knob class turned out to be the right thing to do- it's implemented almost totally separate from the main body of the program, as if it were a RB control or something. Knowing I was going to be using it in different sizes, possibly different colors etc, I wrote the knob code as completely scalable- from maybe 20 pixels to over 100. It reads its size from the control width and height, runs itself as far as handling mouse input and storing a control value, and does not inherit comparable interfaces to the controls it will be replacing- so when it's time to plug 'em in I can run the program and see what routines crash and burn (and need to be rewritten for the new control interface). I'm thinking meters need to also be handled in the same way- somehow- not clear on the form yet.
So: dunno what else to tell you, but it seems like the things I'm doing that are helpful are: compartmentalize new interfaces, make them adaptable and get them working independently of the existing code, while leaving the existing code in working condition. Then when the new stuff is brought online do it in such a way that you could do it one control at a time or in small batches.
Dunno if that's relevant to where you're coming from- but it's what I've found necessary when facing a major re-implementation.
First, decide WHY you want to re-write all of it.
Second, document, in detail, what it is you want to accomplish (what should each script do, etc).
Thirdly, write up a project plan, IN DETAIL, so you have something to work from
Lastly, actually translate that into code. Do not start writing code until you have the modules fully documented.
There are lots of good reasons to refactor, but "consistency and reducing redundancy" doesn't seem very compelling. Anyway.
A lot of folks have talked about process and tools, and those are important and I assume you know what you're doing. The real issue is morale. Let's face it, at this point you're requirements are probably stable and your team is familiar with the system and everyone's probably been thinking about how they might do differently in the back of their head. And code is easy to write.
If you have people on your team see what they've done and know they can do better and want to give it a try, things will go smoothly. If you have people on your team who regard this as a needless rewite, on the other hand, morale will suffer and production will be surprisingly low.
If you have some of both, take the people who want to rewrite it and do it from scratch in another branch. If you don't have buy-in from all of the developers and try to fold changes into the original product and it doesn't go smoothly, company politics will turn against you and you'll probably be looking for a new job.
Yes, it'll probably be much easier if you just hash out a set of rules and style guidelines and then rewrite everything from scratch, to those guidelines.
You're going to end up rewriting everything along the way anyway, if you're doing a coding-style-level rewrite rather than just a bugfix, and it'll be a lot easier if you start from scratch.
Josh
bwaaaahahahaha!!!
Even if its probably possible, its a bit hard to define a coding style for a language whose motto is "There's more than one way to do it". Probably the amount of things you should define to have code that looks similar its BIG (where should you use fors instead of maps? should i use $_ or assign it to something else? etc., etc., etc.)...
Its easy to define *your* coding style, but imposse it to your fellow coders might be harder (always speaking about perl). Anyway, you might like to read the "Common practices" chapter in Programming in Perl where Larry gives some good advices about coding style in perl....
My suggestion, if you think the code might start to get bigger... start slowly to switch to another language, 30K lines of perl code just gives me the creeps...
Santiago
[There's only so much a methodology can bring to the table -- after that it's about finding and keeping a competant staff who understand the value of what they're doing.]
You are so utterly right. The key idea is "people over process". I've rather have 5 great people with any old process, than 25 mediocre people with the perfect process.
[look at the console gaming industry for tips]
I have always been impressed with the gaming industry for this reason. Sometime people erroneously think of game programming as being somehow less "heavy duty" than business apps, etc. But games are usually under incredible schedule pressure (for vital marketing reasons) yet also need to ship very few bugs, because buyers are prone to return a game that doesn't work, not become a source of upgrade revenues.
First of all; make an analyzis of the code; Is it modular? What dependencies are there within the code? Was it thought to be modular from the start, and in that case, has the modules started to deteriate? Are there functions/variables that are just positioned at the wrong places? Functions/variables only used by one or two other functions? Functions/variables used by way too many functions?
Based on those questions; start moveing functions and variables between modules. Consider that a module may be more than a single file, but probably not more than 10 files. Try to keep the number of functions per file to about 10 or less. Also, try to keep the number of files per module to 10 or less. Since you have more than 100 files, try grouping them in at least two blocks. Why not try to use the model-view-control or (interface-control-entity if you're into UML) model?
All this can be done without endangering the code, since you do not rename anything; and thus, you do not risk to suddenly start using some other variable/function than you intended. But, remember not to change more than one thing at at time without testing!
Also, if the language can't help you protecting (file-, module- or block-) private functions/variables, use comments to mark them as non-accesible from functions not in that file, module or block.
Now that the code is logically arranged, it is easy to see if there are functions that perfom a similar thing, since they (probably) would have ended up in the same module - or even the same file.
At this stage, you might want to actually start re-writing one or several functions in order to get rid of those functions that does almost the same thing. Also, you might want to re-name some functions to better represent their actual use.
However, if you do that, first think if you want to enforce any coding-standard starting with new functions? If so, create a coding-standard that most closely matches the code written so far, thus only a smaller portion of the code will fail to comply with the standard.
What about personnel? That strange looking code that embodies solutions to real-world problems had to be figured out by SOMEBODY. Such people, as much as working code, are a valuable resource that should not be abandoned lightly.
Additionally, there are difficulties adding new people to a team. Brooks' book "The Mythical Man Month" describes some of the pitfalls.
Absence makes the heart grow fonder:
Isle of Beauty, fare thee well!
Isle of Beauty. , Thomas Haynes Bayly. (1797-1839)
Also the first line in an anonymous poem in Davidson' Poetical Rhapsody (1602) but probably popularized by Bayly.
See also:
Absinthe makes the heart grow fonder.
Abstinence makes the heart grow fonder
Maintenance makes the heart grow fonder
Obsolesence makes the heart grow fonder
Absynth Makes the Heart Grow Fonder
etc.
I think one of the biggest problems is that people believe that because they named something clearly, it must automatically be clear and logical to others. More generally than just naming steps of an algorithm, this is a problem with naming commands, functions, variables, etc. I think this paper makes a strong case http://citeseer.nj.nec.com/furnas87vocabulary.html for the naming problem (yes, this is even a problem with experts in a particular field). Names are great, they don't always stand on their own. I'd highly suggest that people read the short section on "armchair" naming. I haven't seen a programmer who wasn't tempted to use this at one point or another.
/ \
\ / ASCII ribbon campaign for peace
x
/ \
The main reason for redevelopment is when the underlying data structures change and the legacy code won't work with the new data base product or schema.
Typical reasons for changing the data base are:
the platform is obsolete, e.g, the vendor abandoned the product or went out of business; it doesn't run on a new operating system.
some critical feature is only available on a new platform
the original data schema is inadquate or poorly designed
In general, if the underlying data design remains the same, then you don't need to do a massive redevelopment.
Complete redevelopment to make maintenance easier, that is, for consistency and reducing redundancy, is rarely cost-effective. It's better to begin adding new features in a more robust or easier-to-maintain platform and convert legacy systems as time permits according to some set of priorities.
If you do redevelop, make note of why and make sure you don't make the same mistake again. Often a poorly designed system is the result of some business requirement or limited resources. If those pressures remain, the new system will have the same problems as the old one.
K&R style was developed for presenting C code in a compact format for publication. It has popularity going for it, but there are better styles available for everyday code, some at least as common. K&R style becomes hard to read far more often in C++, where you have things like try-catch blocks around, in-line function definitions in class declarations, and so on. As always, of course, it's more important to choose a consistent style and stick to it anyway.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Design documentation is highly overrated (after all, it's a duplication of the code itself, which is the most detailed possible design document). Requirement documentation is essential. I suspect that this is what you're asking for, anyhow, since you're talking about showing it to the person who asked for it.
I think that requirement documentation is the only universal documentation requirement. Everything else can be ignored in some (special) situations.
-Billy
That's a very good point about console games.
I hadn't thought of that, I was thinking of games that run on a PC. PC games get the "best of both worlds", so to speak.
"My programming team is considering making some sweeping changes to our code base (150+ perl CGIs, over a meg of code) in the interest of consistency and reducing redundancy."
Do you mean that you have consistently repeated the act of re-writing your code in an effort to reduce redundancy? Or perhaps you are seeking to reduce redundancy by consistently repeating yourself? 8^}
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
The first answer is that when more than one line is covered by a comment, those lines form a functional group and MUST be a function of their own.
When you scream "must," you seem to claim that "there are no exceptions." What if this new function needs to use a lot of a function's local variables without passing them as parameters (expensive for stack memory bandwidth, especially in the inner loop)?
There is some info that can't be conveyed by code; it's called "requirements". That info can't be conveyed by comments, though, either.
However, you can refer to and briefly quote the use cases in the comments, to give future maintainers a bit of context.
Will I retire or break 10K?
programmers should care about the macro facilities of a language. Perl and Java have none built in.
Nonsense. Perl's macro system can even translate Latin into Perl at runtime.
Will I retire or break 10K?
Take a look at their HTML/CSS and see it is HUGLY bloated
It's bloated because the webmaster tries to support Netscape 4.x, and doing CSS inheritance (which would unbloat many sites' CSS) will crash Netscape 4.x.
and not gzipped
Is there a Free module for Apache HTTP Server to do this? (In other words, is mod_gzip Free?)
Will I retire or break 10K?
The worst programmer habit I know of is copy-and-paste coding instead of using subroutines.
Sometimes, you don't want a lot of if-then's or subroutine calls in an inner loop, especially if software is supposed to run in real time (less than 50ms latency from key input to video and audio output) on fixed hardware (such as a game console) with all required features.
You can tell people not to do it, but some always will.
And some never will. I understand that, as a prominent CS scholar has stated, "premature optimization is the root of all evil," but, once we have a working sub-real-time implementation, some corner cases in software that processes digital signals really need code duplication if we expect them to run efficiently on architectures that lack predication. (Predication refers to machine instructions that say "execute this instruction if and only if the given bit of flags is set.")
Then there's the problem of not being able to quickly pass a function's local variables to another function because C, C++, and the Java language do not allow functions to "own" other functions, a construction that was quite handy in Pascal.
Will I retire or break 10K?
Function calls are cheap, performance wise.
Not in an inner loop, on fixed hardware, in real time signal processing. And yes, I have profiled my code; for some functions that have four separate cases, inlining each case separately improves performance. However, using compiler features to inline a piece of code tends to destroy portability (say if you have one version that runs on PC and another that runs on GBA) if you use anything but GCC-across-the-board.
Will I retire or break 10K?
I think an important thing to clarify is that documentation is good and comments usually mean your code is bad. Comments are not documentation.
I attempt to keep a documentation/history somewhere for each function, class, or any other major component. In this documentation I typically explain what the code is for, it's inputs and outputs, and a history of why certain things were done in the way they were. This is often just one or two lines but despite good naming practices and good coding there are times where it just isn't obvious that on platform Foo you have to do it this way or you see Bar bug. I don't however go through code line by line commenting what it's doing and why. That should be perfectly obvious if your code is any good.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Not if it's done properly, it's not. Attempting to duplicate your code is actually damaging, because your docs will become out of date and misleading. However, maintaining a 10 page summary of the basic foundations of your system is far more helpful when bringing a new team member up to speed than sending them away to look at your "most detailed possible" 500,000 lines of existing code.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
I agree wholeheartedly. I think most of the problem is McManagers failing to understand what "hardware is cheap" means.
Hardware is cheap, but this is a relative statement. Shipping your kit on a platform with a 1.5GHz processor instead of 1GHz is often a better choice than having your whole development team spend six weeks profiling and optimising the code to get a 50% speed improvement. This is very expensive in terms of development time (= time to market hit and extra wages). Worse, it tends to lead to obscure optimisations in otherwise clean code that rarely get documented properly and thus hinder maintainability. So, in this respect, hardware is cheap.
However, this is not an excuse not to write good code in the first place. Don't prematurely optimize, sure, but for goodness' sake get the basics right.
- Use suitably efficient data structures and algorithms.
- Design your code in a modular way, so that if you really do need to apply optimizations, you can do so in an isolated, controlled and easily testable way.
The cheapness of hardware does not justify getting sloppy and ignoring the above. These are the major causes of bloatware and the reason why I could start typing faster with Word 5 for DOS on an 8086 more than a decade ago than I can with Word 2002 on a 2GHz Athlon running WinXP.If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
(* keeping all of the business logic in sensible, cohesive compartments that run on an application server.......In summary, for shallow web-only stuff, PHP is a reasonable tool. For stuff beyond web work, PHP is out of its design envelope. However, Perl works here just as well as it does for web work. *)
I never suggested that PHP do *everything*. The trick IMO is to use RDBMS as the "transport and messaging mechanism". All kinds of different langs and apps can communicate thru it. Even IBM mainframes partitioned stuff into interface-driven stuff (TSO) and batch jobs.
You are describing 30-year-old technology.
Also, some people just don't like Perl. Don't take it personally. Everydody's neurons just happen to be wired different due to Darwinian divsersity (or a flayful Allah).
Table-ized A.I.
>Not too long ago a link was posted to an interview with Joel Spolsky who used to work at Microsoft.
WOW, you are easily impressed. Just because somebody used to work for Microsoft does not make them an expert software engineer. In fact judging from poor quality of most Microsoft software we can safely assume the absolute opposite.
This is a bit late, but...
You don't have to refactor your code in one swoop. You can do it gradually, like when you open up a piece of code to add functionality. Then, you can make changes that make the code cleaner, and make it better suited to support the new features. As long as you write tests for the existing and new functionality first, there should be few surprizes.
I recommend Martin Fowler's book on "Refactoring". The classification and analysis of typical code transformations makes it much easier to think about ways to improve code without braking it.
-- Juanco
And having him look at the requirements, and then work on the unit tests, and finally pair with a more experienced programmer, is better still -- and this way there's no extra paperwork to maintain.
-Billy
Do you have any evidence to support that, or is it just a little gratuitous XP evangelism? ;-)
But seriously, on a large project, you can't always afford a sufficiently experienced programmer to work as a pair with all the newbies. For example, on the project I'm working on at the moment, there are only two people with anything like complete knowledge of the project. This is a serious liability, and we're trying to spread the knowledge out to the rest of the team. However, there are several more than two junior guys. It's far more efficient for those who know to document the basic knowledge as a starting point, than it is to try to convey it to each junior team member directly during day-to-day development.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Do you have any evidence to support that, or is it just a little gratuitous XP evangelism? ;-)
;-)
The second.
I seriously do recognise some need for "get up to speed" documentation; it's just that it's almost impossible to write enough of it, and not write far too much. Of course, if you're doing pair programming regularly you won't need any (and you will have ALL team members knowledgable about the code in general), but if you can't do that (and politically it's almost impossible) you'll HAVE to write that getting-started documentation.
A pity that almost nobody will read it; the only exception will be the people so concientious that they hardly need it. Of course, you could be the exceptional project in which the people who know the code are also talented writers. But I doubt it.
-Billy
Sad, but true. :-)
Probably not exceptional at all, actually. I'm just noticing that around 18 months ago, as I came onto this project, I started a steep uphill learning curve to get up to speed. So far, about four people have made it to anywhere near the top in seven years of the project's history, not counting the guys who designed it all in the first place. These are the only people who can do serious maintenance or development on a project critical to the company. Everyone else on the project does their bit, but there is a certain point when they can't do any more to help.
Now, a year and a half later, I'm responsible for training up the next guy to join the team. He's enthusiastic, and has a great attitude, and you can work with someone like that. But he has the same problem I did back then, and I have the same one my own team leader had back then: there's no basic docs to teach him the ropes, so I have to spend large amounts of my valuable time -- like my team leader before me -- training him up instead. He may or may not ever grok the really key stuff at the heart of the system, but I hope that he will. However, if someone would authorise the time for just one person who knew the stuff to document the basics, it would literally save weeks of effort in future on the part of senior team members, particularly in training up those who are only ever going to use the basics.
And then I woke up. :-)
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.