Slashdot Mirror
Ask Slashdot: What Tools To Clean Up a Large C/C++ Project?
An anonymous reader writes I find myself in the uncomfortable position of having to clean up a relatively large C/C++ project. We are talking ~200 files, 11MB of source code, 220K lines of code. A superficial glance shows that there are a lot of functions that seem to be doing the same things, a lot of 'unused' stuff, and a lot of inconsistency between what is declared in .h files and what is implemented in the corresponding .cpp files. Are there any tools that will help me catalog this mess and make it easier for me to locate/erase unused things, clean up .h files, and find functions with similar names?
Who about "rm"?
Or laser from orbit!
If you're company is willing to pay for it, you can get something like Coverity. On the free(as in beer) side there is CppCheck and clang.
Because these two parts are crucial. Just like there are no "suffiiciently smart compilers", there are no "sufficiently smart optimizers/cleaners" that can do the job for you. You'll have to roll your sleeves up for this one.
https://www.jetbrains.com/clion/
Seriously, that's mid-sized at best.
scan-build and scan-view from clang++ will show you what is being used and what isn't as far as static code analysis goes.
cd Large_Cplusplus_project
sudo rm -r *
sudo apt-get install java
So, figure out the layers or logical components between each module and then you will be able to chew smaller chunks.
Then, doxygen the whole lot, making sure to use dot to create the graphs for callers and callees. This will let you see the interaction points so you can see what impact a change in one method will have (ie which callers you have to check).
Some people will say "write unit tests" but frankly, it never works with a legacy code base, to effectively unit test you have to write your code differently to how you'd normally do it. You don't have that luxury here. So a good integration test suite should be developed to test the functionality of the whole thing, then you can repeat it to make sure your changes still work. Its not as instant as unit testing (but more effective) so you'll have to invest in a build system that regularly builds and runs the (automated) integration test and tells you the results - and commit changes reasonably regularly so you can isolate changes that end up breaking the system.
The rest of the task is simply hard work running through how it works and understanding it. There's no short-cuts to working hard, sorry.
Any decent IDE has the capability of pointing at least towards unused blocks of code and will generate a tree of function calls. I've worked with Eclipse and Xcode both of which have these capabilities. Even GCC (or another C compiler) can warn you about chunks of unused code or missing/bad header files. You can also rename functions across the entire codebase if necessary.
If your code has warnings or errors, continue fixing until the warnings are gone. As far as functions that do similar things but are named differently, that is a bit harder because 'looks like they are doing the same thing' doesn't always mean they ARE doing the same thing (if they have the exact same code, you could perhaps solve with statistical analysis or simply a text finder).
Make sure that if you replace a function that it has the same behavior in all cases. Even mediocre developers have learned that reuse existing code is a "good thing" and often different functions that do "the same thing" have edge cases (often undocumented) where it does behave differently (especially in C/C++ eg. difference in signedness, memory mapping method, characters etc)
Custom electronics and digital signage for your business: www.evcircuits.com
This strikes me as a very risky undertaking. If there are a lot of functions/modules doing similar things, any attempt to combine many similar functions into one runs a huge risk of introducing bugs if you can't wrap your head around the entire program (which is unlikely imo). There is a huge time and budget risk in this endeavor.
Seriously, you never know when some previous programmed made a "duplicate" function to do something bizarre, like force a particular initialization order of static-class-member variables between translation units. Sometimes deleting pointless code can do... terrible things. Just be careful, test your changes, etc.
"Sorrow is better than laughter, for by sadness of face the heart is made glad." [Ecclesiastes 7:3]
While I dislike writing unit tests, I have to admit they are useful in protecting your butt when something breaks, since the test should catch it first. Of course you need to decide whether in a particular scenario they add value or just make you manager happy.
In a case like yours, you can make code modifications and hope nothing breaks or build unit tests and ensure that you don't break any of them when refactoring. Initially rather than just ripping out the seemingly duplicate methods, rip out/tweak their implementation and have them point to what they seems like a the right method to provide the common functionality. If your unit tests show breakage, then you know that you missed something.
If you do things wholesale, then you are likely to break something in an unmanageable way. Oh and make sure things are version controlled ;)
Jumpstart the tartan drive.
graphviz can visualize the inter-functional and inter-file dependencies.
It's free and built into the functionality of doxygen.
I'd recommend recommenting all the functions using doxygen - because to clean up a large project you need to know it.
There's no one-click "solution" that's going to rewrite your code for you. I'm assuming the program design isn't well-documented other than maybe an occasional inline comment here and there, so you basically can't do much until you know exactly what the program is supposed to do and how it's currently doing it. Then you can start to identify unused functions. Until then, it's best not to make any changes just because it *looks* like something is unused.
It's also good to be able to compare a massive cleanup project with rebuilding from scratch. If the codebase is truly as fucked up as you think it is, rebuilding from scratch could be a viable alternative. Especially if the cleanup looks like it would take 6 months to a year, plus extensive testing to make sure stuff didn't break.
Modularize the software. There are a lot of tools which can help you to analyze static dependencies in the code which can help you to identify components. You could also use a run-time analysis tool for example Kieker which is initially for Java, but there is an extension for C/C++.
Use your brain while exploring the code.
When you find code that you think is irrelevant, remove it.
Try compiling.
If the compilation fails, see what's actually using the code you thought was unused. Remove it, too, if you think it is unused.
Try compiling.
Repeat as much as is necessary.
When everything is compiling, make sure the software still works. Make sure you didn't remove any code that is dynamically loaded, too.
Commit your changes to your source control system.
Repeat as often as is necessary.
Hi,
200 K lines is not a very large codebase, you can fix it up with emacs or just some grepping (or check the GNU idutils, ctags, etags, etc but not really needed for 200K lines), and some good regex. Auto-open all files matching the grep pattern with your editor, then apply the appropriate regular expression to all open files/buffers, or just check them one by one (it's better, and faster than you'd think, it's still a size you can manage by hand).
Good luck!
Doxygen was my first thought as well.
Python :-)
Table-ized A.I.
You admit you don't know what it's doing.
But you want to "fix" it?
HELLOOOO!!! Disaster awaits if you mess with code you don't understand.
If it doesn't work, toss it.
Either way, you're back to DO NOT FUCK WITH IT. At least not until you understand it. ALL of it.
Or butterflies...
Find a bug and call your team irresponsible // fork it to libre[your product name] // Upload the source to OpenBSD repo.
I used it a log time ago and it was excellent
You didn't mention a version control system, so assuming you aren't using one:
Turn it into a git repository so you can easily back out of changes.
Then run doxygen and start reading through the documentation.
I've successfully used this pattern:
When I run into some badly designed code where areas of responsibility were blurred or utterly gone, just do this:
- Flatten the whole thing into a single function (or as few as possible).
- Restructure the result, removing redundancy wherever possbil.
- Factor out into smaller, more logical units afterwards.
Make sure that the whole thing works at every step along the way. (In other words, use functionally invariant modifications of the source.)
That's exactly the kind of crappy code to get rid of. It's a hidden risk that should be exposed and eliminated. More often than not that kind of stupidity is due to some hotshot dickbag "brogrammer" trying to show off and strut his stuff. It is otherwise completely unnecessary. Programmers like that do stupid stuff that requires "heroic" workarounds to be used, and then portray themselves as "heroes" when they implement these unnecessary hacks that they forced in the first place. To hell with them and their awful code.
220K is still not that large. Trouble comes if it is that large you cannot create projects w/ cross references in the IDE of your choice ...
To be quite frank, what you need are man hours. There are many tools out there that can help you finding corners or edges to start working on, but you can do the same with a coin toss, no tool will significantly reduce the amount of man hours that will have to be spent fixing, re-factoring and re-organizing. Take a good loooooong look, devise a simple strategy and then jump in somewhere. From personal experience, add lots of assertions as you go.
There's over 1000 lines of code (on average) in each source file? I'm sorry. That sounds like a mess.
1. Modern IDE with good gcc parser: Eclipse, Netbeans, 3rd party paid ones. Not Visual Studio. You want it to build call hierarchy tree for you, so that you can find methods that are unused. It will require some manual steps
1a. if you have $, Understand for C/C++ is proprietary tool that will map a hierarchy of your code.
2. perform structural coverage analysis of code in live action, will help map the dead code. gcov is free if you can use it.
And crank up the warning level to help you find inconsistencies between headers and declarations. In fact, you might need to start by cleaning up header files.
Doxygen can help you find truly dead code.
Cloned code is a pain to deal with - I don't know how you fix that. I guess it depend on how much of it there is..
I'm available to help.
I'm sure I'll be down voted into oblivion for saying this on slashdot, but visual studio is actually a fantastic IDE. It will give you clear visibility on where a function or variable is referenced anywhere in the code which makes it very easy to remove duplicate functions and other legacy nasty-ness like thousands of lines of functions that are never called.
It doesn't do anything that you can't achieve with a combination of open source tools but it's a hell of a lot easier to use IMO.
Along with coverity as one of the commenters suggested, you can compile the code with stricter compilation options (like -Werror in gcc, which will error out if variables/functions are not used etc), you would then need to go through each of these files manually and resolve all the issues. Tools like bcpp can help you make sure your complete code base follows a common coding standard. Apart from that, if the name of the function is not indicative of what the function actually does, there are no tools smart enough to help you with that. You'd need to do a lot of cleanup manually by hand.
they are going to release this new feature later this year.
Thus, you are looking at my code!!!
You need to write a test suite to confirm what works and what does not work.
Once you have tests, you can start running coverage tools (like gcov or Coverity).
If your tests are not covering parts, you need more tests or need to consider removing that part of the code.
When tests are complete, then you can think about how to clean it up (refactor, rewrite, organize or whatever word the cool programmers are using now days). You can use your compiler warnings as a lint. And start to work through the spammy build logs to eliminate all the warnings. A good goal is to have zero warnings and after that build with -Werror which will cause builds to fail if any new warnings are introduced. (if you have 3rd parties or customers that build these libraries, you might not want to do that)
Another option that becomes available after writing proper tests, is that you can make the decision to discard the entire project and start over from scratch. This is good if the requirements have changed dramatically over the years and a lot of messy hacks exist to support obsolete requirements. I must warn you though, usually rewriting is a waste of time. Time that is better spent understanding and fixing the existing code, after all source code is just a text file, you know how to edit a text file right?
“Common sense is not so common.” — Voltaire
You have text !
Whacking text into the shape is what Perl was built for !
Seriously, you need the opinion of an experienced application designer.
A hands on, I looked at the code & documentation, asked questions, (rinse & repeat this cycle, until clarity)
This is a forensic job.
And should be PAID for by the managers that had their customers & people build this !
"Here ya go, kid...some real-world experience."
Try using indent, works great!
Who said anything about doing the job? They're asking for suggestions for automated code analysis that can hilight potential "problem" areas/code duplication/etc. Seems like a common enough situation that someone may have made a tool for it. Automated *repair* would be a far more challenging task, but just hilighting potential inconsistencies and redundancy "hot spots" is something that could be done with fairly high false-positive/negative rates and still be extremely useful when faced with cleaning up an atrocious codebase.
--- Most topics have many sides worth arguing, allow me to take one opposite you.
See: Working Effectively with Legacy Code book review (2008) for a book of that title by Michael Feathers (PDF article) on that very topic.
There is even a summary of key points at Programmers @ StackExchange. Hundreds if not thousands of programmer's blogs address this very topic.
You're welcome. Now get back to work.
Your glance IS superficial. It takes at least 2-3-6 months toget a (basic) grasp of any project and figuring out what needs to be removed in that period is a waste of time. You will find you need to re-implement most of the "crap" you removed in the first place. So patience is your friend. Look, learn, study and then after you know pretty much all code paths decide what can and cannot be refactored, if anything. Good luck!
http://sourcenav.sourceforge.net/
to look around and get a feeling.
Editor is not that good (but OK), but an external editor can be used.
Call graph etc. all is doable. FOR FREE (beer and freedom).
Wow, what an easy pitch. :-) At Mozilla, we've put together a tool called DXR ( https://github.com/mozilla/dxr... ). It indexes your code and lets you do text and regex searches. But if you can get your project to build under clang, you can really have some fun, with queries that find...
* Calls of a function (great for dead code removal)
* Uses a type
* Overrides of a method
* Uses and definitions of macros
* etc., etc., etc. There are something like 24 different structural queries you can do.
Because all of this is informed by the internal data structures of the clang compiler, it's nigh on 100% accurate (aside from more dynamic behaviors like sticking function pointers in a table and passing them around). You can also explore a hyperlinked version of the source, bouncing from #include to #include and drilling into methods.
Here's how to set it up: https://dxr.readthedocs.org/en...
Here's our production instance you can play with: https://dxr.mozilla.org/mozill...
If you run into trouble, pop into #static on irc.mozilla.org, and we'll be happy to help you.
A non-free but worth it tool for making code make sense from https://scitools.com./ I don't work for the company. There is a 15-day free trial. It costs $1K-$2K but if the code is important and you are going to live with it a long time it is worth it.
This can help you produce a dependency graph and visualize your rats nest. 200 files isn't that big. My project has 1500. Coverity can identify unnecessary include files, leaks, bugs so forth.. Read Lakos to learn a lot about dependency control and physical design. There is a big difference between cleaning up the code to placate Coverity and producing a good testable design.
A lot of the coders who contribute to my project are monkeys who don't care about making a polished final project for the end user. I do. What I've done is to first compile everything with g++ (or mingw if you don't have a Linux machine available), which enforces the standard, as vc++ has "language extensions" that allow a lot of crappy code to compile.
Secondly, run doxygen on your code base with graphviz installed. This will "kind of" document the source code, and it will generate call graphs for classes. You'll be able to look at a class and see if anyone actually calls it. Remove any that don't get called by anyone, then some of the things they call will be uncalled, and repeat.
There are a few code beautifiers that can help make it look like one person wrote it instead of dozens. Stack Overflow has lots of suggestions on that front. I couldn't find one that truly served my needs, and since I had to add copious doxygen-style comments to generate usable documentation for the end user (not necessary just to get basic code documentation and call graphs) anyway, I just did it by hand...still doing it actually.
First off, 220k lines of source isn't that big.
You're not going to solve this with a big bang so get that idea out of your head. You're going to solve it gradually, and for a code base of that size it's going to take maybe a year of relatively slow improvement. Everyone on the team has to be on board, and every code review must include "What has been improved?" and "Did anything get worse? If so, that's not okay."
1) Pick your battles. The code you're not changing is code that doesn't need to be looked at. Address your pain points as they come up.
2) When you find a pain point while making a change, MAKE IT TESTABLE. Since you're in here making a usually simple fix, a single nominal test verifying that fix is fine. Testing anything else is a waste of time. Testable code will improve over time.
3) If you can't make code testable because of an intractable dependency graph, welcome to the hell of "Design Dead". The only way out of this scenario is a rewrite of that area.
4) Find your comfort level with regard to time boxing refactoring work. On my engagements, they just happen automatically, without explanation outside the team, nor apology to anyone. When estimating a piece of work, pad it with some extra time for cleanup. Only actually create work items for design dead areas. Your definition of done must include testable, tested and improved code.
5) Duplicate code in itself isn't evil, and inconsistencies are simply inevitable. If you find duplicate code, pick one and deprecate the rest. However, code that is tightly coupled to the deprecated code will need to be refactored and if the coupling traverses an extended dependency graph, you'll simply have to live with the duplication and just stop adding to it.
there are no "sufficiently smart optimizers/cleaners" that can do the job for you.
There are also no hammers that can build a house for you. But a hammer is still useful if you are building a house.
Boy it sucks to be on the job market again for the first time in years and see a slashdot article like this....
I do these types of things for fun in my spare time. I love optimizing and cleaning up nasty codebases.
Seems like you likely want someone like me who does this at an above average level due to passion.
I'd doubt you are local but I could telecommute and would be happy to begin immediately. I have extensive C++11 experience and optimization is my specialty.
Link to a job posting and I'll send a resume....
-1-
Install "OpenGrok" ( https://github.com/OpenGrok/OpenGrok ) and index your code.
OpenGrok is the best source-code browsing option out there.
Use OpenGrok to extensively read and understand your code based.
Examples:
Which files in the linux kernel call 'printk':
http://lingrok.org/search?q=printk&defs=&refs=&path=fs%2F&hist=&project=linux-next
Where is 'printk' defined?
http://lingrok.org/search?q=&defs=printk&refs=&path=&hist=&project=linux-next
-2-
Use Clang's static code analyzer, 'scan-build' : http://clang-analyzer.llvm.org/scan-build.html .
Depending on how good/bad the code is, there could be many false positives.
but it will give you a sense of what's going on, and what to focus on.
-3-
Enable all possible compilation warnings (either in GCC or CLANG).
The more the better. Use "-Werror" to ensure you don't ignore them.
Do it iteratively if needed by enabling more warnings, fixing what breaks, and repeat.
A good list is here:
http://git.savannah.gnu.org/cgit/gnulib.git/tree/m4/manywarnings.m4#n103
Especailly eliminate unused code and variables.
-4-
Analyzer the McCabe Complexity ( http://en.wikipedia.org/wiki/Cyclomatic_complexity ) of your code, using pmccabe ( https://people.debian.org/~bame/pmccabe/pmccabe.1 ).
Focus on functions with too-high score, and re-factor them.
-5-
Add automated tests to your program, and combine it with code coverage (lcov/gcov).
In addition to the general good advice of 'try to increase coverage', focus specifically on code sections
which are critical but not covereged at all - write tests specifically for them.
Having some tests is better than having no tests at all.
-6-
Decide on code style (e.g. linux kernel style, GNU style, any other style) and build shell commands to tests them (i.e. a combination of grep/awk etc.).
New commited code should adhere to the style. Use git hooks to enfore it.
Existing code should be (slowly) refactored to the new style.
Which style is a matter of personal preference, but having a consisted style across all code really helps.
Ideally, it should be something as easy as 'make syntex-check' in GNU Coreutils.
-7-
With all of the above, integrate the tests into an automated system (e.g. autotools or cmake or just makefiles) that will allow you to run and re-run and re-run these checks easily.
If it takes 10 shell commands to do static analysis - you'll be too lazy/busy/whatever to do it more than once.
It should be as easy as 'make static-scan' or 'make coverage'.
Investing in writing a good makefile is worth the effort.
Good luck.
- gordon
I've done this on occasion, turn a "C/C++" project that really was just a lot of C with some nonsensical use of C++ features, in something that needed a lot less in terms of macros, compiled to a smaller binary, and actually used the features C++ brings to the table to good effect. All I needed was basically my usual development envionment of editor and compiler (nvi in one session, shell prompt in another, screen to tie them both together), though a SCM (even CVS did spiffily) helped a bunch to the point of being indispensable. Another thing that helped quite a bit was something to cross-reference the source (I used ctags, works together with nvi). The rest is elbow grease.
Personally I like consistency, so I expect ".h" files to actually work as includes in ".c" files (so for ".cpp" use ".hpp", easy does it), and this is a reasonable time to do a formatting pass, which also is a good excuse to take a look at the code without touching its content, just the formatting (to 80 cols, uniform indentation and brace styles, etc.). Afterward you can do passes doing simple changes, like deduping copy/paste code, abstracting out functions, reshuffling code so same-purpose code sits in the same files, and so on. Then cook up vehicles to do more sweeping replacements with. At all times, keep the code working; if you make a boo-boo back the fsck out until it works again, then try again.
....Is NOT anything like a large project.
It's almost small.
You should be running at Warning Level 4 when coding. Its good practice to prevent the issue you have now.
It will give you a crap load of warnings (which are all worth fixing if you have the time), but, it will highlight any unused variables and/or functions.
in Visual Studio 2008-2013:
- Project > Properties
- Configuration Properties > C/C++ > General
- Change "Warning Level(W3)" to W4
Anecdote from the mists of time:
There was this C program which had been around a while which had undergone some evolution and maintenance. The decision was made to 'clean it up' There was a data structure, an array I think, which was unused in a subroutine, lets call it subroutine A. So it was removed. The next test runs of the application and suddenly the program started core dumping. After some agonizing debugging it was discovered to come from another subroutine, lets call it subroutine B.
There had been an array in subroutine B which a loop had run over the end of. But subroutine A had loaded just prior to B and allocated memory for the unused data structure. This had provided enough space to handle the array out of bounds error in subroutine B but when removed subroutine B began overwriting subroutine A causing the crashes.
It was good that the crashes were easily reproducible or could have been one of those intermittent things that drive people insane. An automated tool may not catch things like that since it may not show up until run time. It is C/C++ we are talking about now isn't it?
putting the 'B' in LGBTQ+
You're going to need it.
That's exactly the kind of crappy code to get rid of. It's a hidden risk that should be exposed and eliminated. More often than not that kind of stupidity is due to some hotshot dickbag "brogrammer" trying to show off and strut his stuff. It is otherwise completely unnecessary. Programmers like that do stupid stuff that requires "heroic" workarounds to be used, and then portray themselves as "heroes" when they implement these unnecessary hacks that they forced in the first place. To hell with them and their awful code.
Wrong.
Changing someone else's code that isn't broken because it doesn't meet your stylistic tastes is what "some hotshot dickbag "brogrammer" trying to show off and strut his stuff" does.
It's the only way to be sure....
Seriously though. C++ is one of the most powerful, complete commercial languages.... with a code interface and syntax designed by Satan. You couldn't have *designed* a coding system that would better encourage missteps, fuck-ups, obfuscation and a plethora of errors.
It's a product of 90s math nerds whose machismo came from knowing more and better than regular folks. It was never designed to get work done efficiently; it was designed to feed the egos of C++ programmers.
Better to take a relatively sane language like C# and make it scalable to the point where it can do everything C++ can do with a more restricted syntax and structure that ensures consistency and readability.
Please do not read this sig. Thank you.
Hey, MC Hammer built my house for me.
Unfortunately, I'm not allowed to touch it.
If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
Comment removed based on user account deletion
It's not a tool trick, but I found valuable in some project to rename functions and variables to make them telling really what there do. It's not rare that the name was a poor choice or that his semantic changed in the evolution of the project. From my point of view, it's a kind of documentation.
No easy answers, but I've had to work with a project that was likely very similar to this. I don't what the code looks like in your case, but in mine it was utterly appalling. Not only did functions seem to repeat the same things over and over, but they all had minor variations on the same name, e.g. DoIt_1, DoIt_2..., DoIt_n. Worse, they all used the same (very terse) variable names. Some ran on for 600-700 lines, often with multiple logical lines of code in the same line in the editor. The few, spartan comments in the code had obviously been copy/pasted freely because most said the same thing (and clearly had nothing to do with code). The code 'logic' (of it could be called that) ran all over the place, to the point where you begin to doubt your sanity (DoIt_1 calls DoIt_50 calls DoIt_2 calls DoIt_4, etc etc).
Things that helped get some semblance of sanity:
1) Doxygen - it can give you an overview of call trees, classes etc. This may reveal whole files that are completely unused (it did in may case), in which get rid of them.
2) As you learn more, begin to add long, meaningful names to functions and variables - it can make a world of difference
3) Document everything you learn about the code in comments. Trust me, you will forget what you learned otherwise with so many parts that look alike. It also helps you keep track of code you've reviewed
4) As your understanding of the code improves, start refactoring. Try to rationalise the number of functions and exactly what they do. In my case, that sometimes meant breaking huge functions into smaller ones with a clear purpose.
5) Begin to modularise the functions (if possible) to give better structure to the project. This might help you see where the real duplication lies.
Clearly this is a long, hard slog. Good luck!
That's one person's project for a year to write that volume of code.
I do not fail; I succeed at finding out what does not work.
SciTools has a Understand series that is designed to do what you are asking. I personally found it to be very useful in a very large project I was maintaining.
Small, for most people, is something with tens of kLOC or less, medium projects have hundreds of kLOC and large projects have millions of LOC.
A large project would be something like the Linux kernel which has around 16 million LOC.
I would advise using doxygen to have a global view of the codebase, some kind of lint like g++ -Wall, and a good editor preferably with refactoring support as tools. Plus static code analyzers and valgrind.
First thing you should do is backups. Save the old codebase source repository somewhere safe. If the code is stored in an old repository like CVS, SVN, or worse no repository at all, you should migrate it to something better like Git. Then you start working by removing dead code, indenting, do static and run-time code analysis to find bugs, then merge duplicate code. Start with the more mechanical parts first. Once you get that working and bug free you basically have the new version 1.0.
Then you can start analyzing the codebase in order to understand it and refactor the code for real. That will be version 2.0. You will be proud of it because it has your touch on it but will probably be crap and worse than 1.0 was.
Then you work on version 3.0 which downscopes the feature creep and bad problem analysis mistakes you made in 2.0 and is finally better than 1.0.
What makes you so certain that its not the intern assigned to the task who is posting the question?
Please, Slashdot, don't hurt him.
...said the hotshot dickbag brogrammer trying to show off and strut his stuff.
Easy!
The dangers of excessive individualism are nothing compared to the oppressiveness of excessive collectivism
a lot of inconsistency between what is declared in .h files and what is implemented in the corresponding .cpp files
That's impossible unless you're talking about comments in the header files, or the implementation (.cpp) files don't include their own headers. Generally speaking, every .cpp file must include its header in the first non-comment line of the file.
Good:
#include "foo.h"
#include <cmath>
Bad:
#include <cmath>
#include "foo.h"
A successful API design takes a mixture of software design and pedagogy.
Wrong.
Changing someone else's code that isn't broken because it doesn't meet your stylistic tastes is what "some hotshot dickbag "brogrammer" trying to show off and strut his stuff" does.
Agreed. It's a slippery slope that ends with systemd incorporating a web server at one end and a boot loader at the other.
Okay so I had the chance to cleanup a 400kloc java project some years back. And a 1kloc C++ project more recently. The process is roughly the same.
1) Read the documentation and/or find the previous coder and politely ask them what were they thinking.
2) source control. You really need to use it so you can feel free to break things and role things back when you can't quite get to the bottom of why or you find you can't quite make all the necessary changes in a reasonable way
You have an average of 1000 lines per file. That is fairly high in my experience. Not always but over a large project... It is a warning sign.
3) Find the files that are statistically large. Set break point on the functions/methods in them and profile for them.
4) .h files are good & bad. Good they give you an idea what to expect in the .cpp and if it isn't in the .h there had better be a reason it isn't. Bad because people misuse .h's and stick macros and other oddities especially of premature optimization in them.
5) Documentation comments. Whether it is doxygen or javadoc the mind numbing dull task of putting in documentation will a) help you gain familiarity with the code b) let your brain start to internalize the code c) make it obvious when code is simple data encapsulation and when it is spaghetti...
6) Change compiler if you can. If the project requires javac 1.3 go to javac 1.6. GCC switch to clang or vice versa. Compilers keep getting not so much smarter as better at reporting the dumb things you (or the previous coder) are trying to do. So just trying to compile with the wrong compiler will give you a hit list of files. Files that compile cleanly with a different compiler are probably not the problem.
A buddy of mine works for Lattix building code parsers and the Lattix tool suite seems to be a good bet for understanding and mapping code complexity and relationships as well as modularity, quality, etc. http://www.lattix.com
find / -iname "*.cpp" -delete
but learn to do them right. If it's painful to write the test then you're probably writing the wrong kind of test. Not everything needs testing. Also be aware some code may rely on errors in other code.
http://www.gimpel.com/html/ind...
Go through the source, find all the functions that do the same thing, comment them out. Write a replacement function you're happy with then repeatedly compile the project, replacing each broken reference with your new function until it compiles successfully.
Now do this for all other functions.
It's not the quickest method but after this project there's always the possibility you'll not find work and starve to death, so enjoy it while it lasts.
http://www.joelonsoftware.com/articles/fog0000000069.html
Debugging code that prints or logs may act to synchronize access to some data structure. Sometimes that can prevent a deadlock or illegal pointer access as a side effect:
http://stackoverflow.com/quest...
http://en.wikipedia.org/wiki/D...
So yes, complex programs can act in strange ways from seemingly minor changes.
I spent a couple years helping maintain a large complex multi-threaded app (which included message passing between the apps, for another layer of fun) which supported 24X7 operations where a minute's downtime could cost millions of dollars in some situations, and it was not easy. The code base was easily 10X to 100X of what the poster of the story is tasked with maintaining. Versions of the code had been in production for over fifteen years. Much of the code had been ported from C++ & Tcl to Java (although C++/Tcl systems remained), but the threading model was somewhat different between the two, and the port had not taken account of all the differences. It would have been nice to be able to rewrite some key parts of the system to make them more maintainable, but there was never enough time for that in a big way -- and realistically, bigger rewrites likely introduce new issues. Still, eventually we got most of the worst deadlocks and memory leaks and similar such things fixed and the system got to the point where people stopped even remembering off-hand the last time a core part of the system needed to be rebooted (previously a fairly frequent event). But each deadlock could involve days, weeks, or even months of study and discussion, adding log statements, writing tests, lab tests, analyzing quite a few multi-gigabyte log files (and writing tools to help with that including visualizing internal message flow), and so on. And, same as you mention, hardware and OS issues could interact with it all, making some things hard to duplicate under virtual machines for developers. One thing is that to the end user, a system that is more stable may not look that different than one that is less so -- there are no new features, so it is not obvious what is being paid for.
Although obviously if the program you support core dumps from a bad address or stack overflow, rather than just freezes up, it is probably something else. Still, even then, a bad pointer address can sometimes come from one thread freeing a data structure when another thread is still using it. The original C++ in the above mentioned project generally was highly reliable, but it still had some odd issues too. In one rare case, memory was freed in an unexpected way under certain conditions by other code running in the same thread but in code nested way deep with essentially recursive calls processing complex messages. I finally also traced part of that too what looked like maybe a bug in a supporting third-party library (a RogueWave data structure). Because that C++ code had been in production for years, and we were loathe to change it at the risk of introducing new issues, we mostly "fixed" that issue by making changes elsewhere in the system to prevent that component from getting the pattern of data that it had trouble handling. But we would not have known exactly what to change elsewhere without a lot of analysis.
Sadly, just as we got it mostly working well, the new shiny thing of a mostly COTS system that did something similar came along to replace much of it (at a much bigger expense than maintaining the old, but granted with some nice new features).
As I saw someone else comment recently about a "stable" OS, the end user generally cares more about how much work a system lets them get done, not how "stable" it is. A reboot can be acceptable, depending on the situation and the alternatives, even if not desirable. Erlang code is probably the master at that approach of rebooting code when it fails. :-) Here
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
Use unifdef to remove code that isn't used anymore.
Well, no. That *is* code you want to either get rid of or *THOROUGHLY* document. But until you understand it you'd better not touch it in any non-reversible manner, and test each change so that reversing remains trivial.
I think we've pushed this "anyone can grow up to be president" thing too far.
No, I like fixing projects like this too. Unfortunately, I never seem to get the opportunity these days. Companies either prefer I cope with really broken tools in perpetuity or are full steam ahead "developing" a new clusterfuck (ERPs).
Refactor, like what I did with 200,000 lines of horrible PHP code. And rewrite some parts, too, if they are architecturally unsound.
there isn't much difference between mid size and large size projects once you figure out how to move forward. It's going to take you a long while anyway.
On one hand, document what you expect this application does for you.
On the other hand, locate the redudant codes. There are ways to achieve this goal. You may have to apply a handful of these though. Yeah right, you can't build a house with just a hammer. Static code analyzer is one, if you can afford the money. The other dumb but straight forward way is to have a log message written at the start of each function. You don't need to do for all of them. Try to start with the closest to the main first. I bet you will be able to eliminate a large chunk of useless code. You will have to observe quite a while, depending of the nature of the application. E.g., for a Finance application, it's like to take you at least a quarter of a year. Then, divide and conquer. Repeat the same to a deeper level of code.
You will have to analyze and refactor the *useful* code at the end.
Hire me, I love cleaning up code.
you need to automate it a bit, mass editing, the way I do it is old school, emacs, grep, perl, sed, awk, etc. There are a lot of little tricks you can use, but it's important to start with something that works first, then you keep making sure you don't break it, let the compiler and linker do a lot of work. Identify something you are going to target, change the name of it, compile and start editing to eliminate errors. As you edit, you'll develop a sense of what's rote/repetivie, shell out and mass change that, compile again and catch the cases you missed. It goes a lot faster than you imagine it will, and you learn a lot about the code.
do not go too long without a working version, if you are buried in hair and can't build and you have to go home, don't be afraid to roll back edits and start fresh.
That is about the only thing that will really help.
I am very small, utmostly microscopic.
we even named the process DevOps for legitimacy!
There are probably other tools, maybe even better tools but it is what I know. I'd say try adding the whole thing to a C++ Visual Studio project. You can then set things on to give you build errors for all unreferenced junk. Find all references etc. Other IDEs probably can do it too but at least entry level VS is free and I know it will do it so ... Only issue you might have is if it is a *nix app or whatever perhaps you'd get a lot of false errors because it won't conform to VC++. But I'm guessing their close enough to get the bulk of the work done.
There are two ways. One is to purchase a tool like we did for Y2K code analysis of 10M lines of code - cost about $150,000. The other is a smart and experienced C++ programmer with a good foundation in object oriented analysis. They would get a tool like Sparx Enterprise Architect ($200USD), reverse engineer the code into full UML diagrams so they understand all the relationships between the various classes, its behavior at a high level, and then start looking at the code where things seem "wonky". Such a person would cost as an employee about $100-120K per year. As a consultant, you are looking at at least 6 months effort at about $150-200 per hour. You can pay less, but you will get what you pay for. FWIW, my consulting rates for this sort of work is $200USD / hour, but then right now I am full-time employed at about $150K + benefits.
Cleanup for the sake of cleanup projects never work. Current code performs some function and nobody can keep enthusiasm reading bad code for months just to have it perform same function in the end.
Instead, you can gradually raise code quality by setting a high bar for new changes. For example, have each change reviewed by a couple of developers other than the author who are known for good style. If a new utility method is added, ensure that the code was searched for existing similar facilities. When legacy mess has to be used, it should be wrapped into a clean interface. And so on.
I'm assuming you're here because this code is critical to your business, it works well enough today, and it can't be easily replaced. You need to keep it working as you go, but you desperately need to modernize it. There's a lot you can do to set yourself up for success, and it's not just tools.
First, get it building in the most current environment available. Is it Visual Studio? Port it to VS2013. Is it Eclipse? Get it into 4.4. Is it not even in an IDE? Get it into one - they're a great timesaver. Pick a refactoring tool, too, something that will help automate common refactoring activities like "extract method." You're going to do that a lot.
Next, get it checked into your source control system, and building on your team's build server. This would also be a good time to revisit the packaging of the deliverables. If you don't already have a task and bug management system like Jira, Mylyn, TFS, Bugzilla, or whatever, get one that integrates into your workflow and your IDE. You have a lot of work to do, and you don't want to waste it filling out Excel spreadsheets. You really need your tools to be as unobtrusive as possible.
There is no sense starting with sub-optimal tools, or fighting a crappy build or development environment. Your time is best spent on coding, and is wasted on everything else.
Now that you're almost ready to get working, build a small suite of automated integration tests before moving on to addressing the architecture. They'll be ugly tests, but you need to know the code is still working as you begin making changes. Make sure the build machine can launch your tests and tell you when they fail.
Now you can dig into the code base. Identify the underlying architecture. Is it event based? Does it closely model MVC? MVVM? Once you clearly define the architecture, break the solution into individually compilable libraries that represent the layers (controller, business logic, data accessors, etc.) Move the existing modules to the most appropriate library project. (Some won't fit cleanly, so you'll end up splitting those into parts later.) For now, make sure it builds and the tests run successfully.
Pick one of the layers to work on first, perhaps the UI, perhaps the data access layer. Get it compiling clean, with no warnings, and turn on the compiler switch to enforce "treat warnings as errors." Run a static code analysis tool (Coverity, Klocwork, Fortify, /Analyze, lint, or anything, really) and fix whatever warnings it gives you.
Tolerate no bugs. As you go through the code, when you find a bug, fix it then and there. Your QA staff will no doubt be finding plenty of bugs on their own, but you need to keep the project as clean as you can.
Next, start refactoring the chosen layer into appropriate subdivisions, such as a controller, business layer interface, etc. You'll want to do a bunch of other housekeeping work here: get rid of globals and singletons, push stray business logic down into the business layer, pull stray UI interactions from the business layer up to the UI layer, etc. This would be a good time to introduce some automated unit tests to the logic you extract and move around. Unit tests force you to make the code testable; things like dependencies on databases, services, files, etc., cause problems with tests, so you start treating them with dependency injection. The primary outcome is that by making your code testable, you make it modular and readable. Plus, you get a few more tests under your belt.
Run a complexity metric across the layer, and look for the highest complexity modules. Start chipping them down. Again, look to adding some unit tests to prove that the code you're isolating does what you claim, and that you're making your logic stateless.
Decide on an exception handling strategy, and make your exception handling consistent. Pick the one appropriate to your app and technology: SEH, try/catch, C-style return codes, whatever, just apply it consistently as you go. Sim
John
Did your contractor go bankrupt for the insurance claims by the workers hurting themselves and crying, after which it was the hammer time?
Break it down.
Use paper and pencil, and regularly assess your progress so you can state in meetings that you've analyzed another 13.4% of the source code. It's practiced job security.
Get a good C IDE, CLion from Jetbrains comes in mind. And start document the code. Dont change anything just document it. Then pick one part and brake it out. Run any tests you have to make sure it works. Then start all over.. get an other IDE.. na.. document, change, test run... until you done :-)
Valgrind is a useful tool to get a profile run, and build up a call tree. As such, you can find the functions that are never called and can be removed as such. Moreover, you can patch a few memory leaks in the bargain.
I am in the same position. Before touching any code, do a backup, put everything under source control with a tool like SVN, install an IDE such as Eclipse, find existing documentation, get advice from previous programmers if possible, divide your work and stay focused.
I would suggest that you try SonarQube (http://www.sonarqube.org/). It is free and does a pretty decent job of finding the duplicate / unused / ... code in your project.
Because fire is cleansing.
Incidentally, to all you guys saying "220K lines is not so much" I reply "a good programmer does not need 220K lines to accomplish any task".
NASA sent men to the moon with fewer LOC....
I'm sorry, but anyone suggesting he should start by writing unit tests for everything is being naive. To begin with, many legacy applications, especially those written in non-object-oriented languages like C/C++, aren't structured to support legitimate unit tests. The act of setting up the environment to actually be able to run a single unit test becomes a significant chore that can consume many days or weeks of time. After that you get to tackle actually writing the unit test for a method which has no clearly defined contract, and you are further hampered because you have no way of tracking down or even testing all of the non-obvious side-effects of poorly encapsulated code. You could easily spend 6 months trying to write unit tests for a small subsystem and end up with a test suite that guarantees absolutely nothing about the accuracy or completeness of the code.
I absolutely agree with the suggestions from others that you want to work out class diagrams, sequence diagrams, and develop end-to-end functional tests for regression testing. It's critical to understand the complete ramifications of refactoring a particular class or some particular methods and functions. As you write new code to replace what is there, you can write unit tests where it makes sense. You should understand the complete contract for every method you write, so you can write the unit tests to guarantee the contract.
Most importantly, take it slowly. Start by looking at the entire system, identify what you feel are the biggest offenders, and work to understand those areas better. And as another poster mentioned, absolutely come up with a definition for what it means to "clean up" the code. Why does the code need to be "cleaned up" in the first place? Understanding your end goal will help you prioritize which areas you spend the most time on, because I guarantee you won't be able to clean it all up unless you re-write the whole thing.
Ocaml: As fast or faster than C++ with much less LOC.
or if performance isn't an issue go Ruby and be under 15,000 LOC.
C++ is brain-damage
Having cleaned up a lot of projects, I don't think a magic tool is the answer. You have to have management approval to fiddle and deal with the testing and new bug outcome. You should work hard to remove all warnings in the code, that often does wonders and exposes a lot of flaws. You need to understand the code and slowly, a, step at a time, evolve the code until it's pretty and we'll structured. Don't rewrite from scratch. Evolve a, step at a time. Fix one architectural issue, make sure it still works, and then keep going, keep refactoring.
While this is in general great practical advice (and no doubt hard won), I can quibble about your point #3 on complex dependency graphs requiring rewrites as the "only way out". Certainly this is more of an issue in C++ than something like Java where code can be more easily replaced at runtime. However, at least in Java, the idea of "mocking" can sometimes be useful to test code even with complex dependencies without (significant) initial rewriting.
I used mocking with JMockit successfully in the large Java project previously mentioned. I tried other frameworks, but preferred that one. JMockit supported creating unit tests for code which was not originally designed to be testable and had complex interdependencies in how objects were constructed. However, JMockit did have a substantial learning curve, even aside from hours spent trying to come up with tests for domain-specific specific code. Eventually I created some supporting code to make the mocking easier for our project, and then another developer improved even further on my work, making mocking our specific application much easier. So, at least in our situation, with a huge complex Java codebase in production, limited developer time, and limited tests initially, mocking was a big win IMHO that let us start to get a handle on everything without having to rewrite a lot of code at first.
That said, in general, code is easier to maintain and understand when it does not have complex dependencies. "Dependency Injection" is a good idea in a lot of cases -- although it can have its own downsides in making object construction code harder to follow:
http://en.wikipedia.org/wiki/D...
So, while I'm quibbling about "only way forward" because of the possibility of mocking, I'm not saying rewriting in such situations in necessarily a bad idea or even quicker than mocking sometimes -- especially as mocking can introduce its own issues.
With JMockit, one such unexpected issue was that mocking an object created mocks up the entire class hierarchy (causing issues when you wanted to mock one class but test a sibling class). This was a subtle issue that took a while to understand, and I did not see documented explicitly anywhere (at least in introductory material) although I think there was a bug/feature request about it somewhere.
Another JMockit issue was that mocks were instantiated and removed in relation to threading somehow and there could be issues with mocks remaining in place when previous unit tests had not completely finished running all their threads. This could sometimes lead to unit tests failing occasionally due to thread timing issues and the mocking, when a class that was mocked in one test or with certain "expectations" was then accessed by another unit test which mocked different objects or had different "expectations". Sometimes this (unfortunately) happened embarrassingly on other developer's machines with different OS or hardware or on our Hudson/Jenkins build server just by the force of numbers of times the tests were run. Usually I could get around these cases either by adding delays at the end of the unit test to let all the threads complete or, better, by having improved mocks or other code that ensured the threads were finished before the test ended.
That said, even with both of these issues, both frustrating to understand and then work around, mocking was still a big win for the project IMHO.
I have not used any C++ mocking frameworks so I don't know how well they work or what their limits are. However, for suggestions about some such frameworks see this StackOverflow discussion:
http://stackoverflow.com/quest...
The top rated answer there is about "Google Mock" but there are other choices.
https://code.google.com/p/goog...
I do not see the word "mock" used so far in this Slashdot d
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
-1, eh mods?
The world is full of self-confident rock-star developers who turn their noses up at professional software engineering practices.
As for the other thing, here are the good old citations: Amazon 1, Amazon 2, Amazon 3.
Late to the party, but I have to chime in: Emacs ETAGS & CTAGS is just awesome for exploring code. find-tag-other-window.