Slashdot Mirror
Ask Slashdot: What Tools To Clean Up a Large C/C++ Project?
An anonymous reader writes I find myself in the uncomfortable position of having to clean up a relatively large C/C++ project. We are talking ~200 files, 11MB of source code, 220K lines of code. A superficial glance shows that there are a lot of functions that seem to be doing the same things, a lot of 'unused' stuff, and a lot of inconsistency between what is declared in .h files and what is implemented in the corresponding .cpp files. Are there any tools that will help me catalog this mess and make it easier for me to locate/erase unused things, clean up .h files, and find functions with similar names?
https://www.jetbrains.com/clion/
Seriously, you never know when some previous programmed made a "duplicate" function to do something bizarre, like force a particular initialization order of static-class-member variables between translation units. Sometimes deleting pointless code can do... terrible things. Just be careful, test your changes, etc.
"Sorrow is better than laughter, for by sadness of face the heart is made glad." [Ecclesiastes 7:3]
While I dislike writing unit tests, I have to admit they are useful in protecting your butt when something breaks, since the test should catch it first. Of course you need to decide whether in a particular scenario they add value or just make you manager happy.
In a case like yours, you can make code modifications and hope nothing breaks or build unit tests and ensure that you don't break any of them when refactoring. Initially rather than just ripping out the seemingly duplicate methods, rip out/tweak their implementation and have them point to what they seems like a the right method to provide the common functionality. If your unit tests show breakage, then you know that you missed something.
If you do things wholesale, then you are likely to break something in an unmanageable way. Oh and make sure things are version controlled ;)
Jumpstart the tartan drive.
Modularize the software. There are a lot of tools which can help you to analyze static dependencies in the code which can help you to identify components. You could also use a run-time analysis tool for example Kieker which is initially for Java, but there is an extension for C/C++.
You need to write a test suite to confirm what works and what does not work.
Once you have tests, you can start running coverage tools (like gcov or Coverity).
If your tests are not covering parts, you need more tests or need to consider removing that part of the code.
When tests are complete, then you can think about how to clean it up (refactor, rewrite, organize or whatever word the cool programmers are using now days). You can use your compiler warnings as a lint. And start to work through the spammy build logs to eliminate all the warnings. A good goal is to have zero warnings and after that build with -Werror which will cause builds to fail if any new warnings are introduced. (if you have 3rd parties or customers that build these libraries, you might not want to do that)
Another option that becomes available after writing proper tests, is that you can make the decision to discard the entire project and start over from scratch. This is good if the requirements have changed dramatically over the years and a lot of messy hacks exist to support obsolete requirements. I must warn you though, usually rewriting is a waste of time. Time that is better spent understanding and fixing the existing code, after all source code is just a text file, you know how to edit a text file right?
“Common sense is not so common.” — Voltaire
See: Working Effectively with Legacy Code book review (2008) for a book of that title by Michael Feathers (PDF article) on that very topic.
There is even a summary of key points at Programmers @ StackExchange. Hundreds if not thousands of programmer's blogs address this very topic.
You're welcome. Now get back to work.
Wow, what an easy pitch. :-) At Mozilla, we've put together a tool called DXR ( https://github.com/mozilla/dxr... ). It indexes your code and lets you do text and regex searches. But if you can get your project to build under clang, you can really have some fun, with queries that find...
* Calls of a function (great for dead code removal)
* Uses a type
* Overrides of a method
* Uses and definitions of macros
* etc., etc., etc. There are something like 24 different structural queries you can do.
Because all of this is informed by the internal data structures of the clang compiler, it's nigh on 100% accurate (aside from more dynamic behaviors like sticking function pointers in a table and passing them around). You can also explore a hyperlinked version of the source, bouncing from #include to #include and drilling into methods.
Here's how to set it up: https://dxr.readthedocs.org/en...
Here's our production instance you can play with: https://dxr.mozilla.org/mozill...
If you run into trouble, pop into #static on irc.mozilla.org, and we'll be happy to help you.
Anecdote from the mists of time:
There was this C program which had been around a while which had undergone some evolution and maintenance. The decision was made to 'clean it up' There was a data structure, an array I think, which was unused in a subroutine, lets call it subroutine A. So it was removed. The next test runs of the application and suddenly the program started core dumping. After some agonizing debugging it was discovered to come from another subroutine, lets call it subroutine B.
There had been an array in subroutine B which a loop had run over the end of. But subroutine A had loaded just prior to B and allocated memory for the unused data structure. This had provided enough space to handle the array out of bounds error in subroutine B but when removed subroutine B began overwriting subroutine A causing the crashes.
It was good that the crashes were easily reproducible or could have been one of those intermittent things that drive people insane. An automated tool may not catch things like that since it may not show up until run time. It is C/C++ we are talking about now isn't it?
putting the 'B' in LGBTQ+
Comment removed based on user account deletion
Debugging code that prints or logs may act to synchronize access to some data structure. Sometimes that can prevent a deadlock or illegal pointer access as a side effect:
http://stackoverflow.com/quest...
http://en.wikipedia.org/wiki/D...
So yes, complex programs can act in strange ways from seemingly minor changes.
I spent a couple years helping maintain a large complex multi-threaded app (which included message passing between the apps, for another layer of fun) which supported 24X7 operations where a minute's downtime could cost millions of dollars in some situations, and it was not easy. The code base was easily 10X to 100X of what the poster of the story is tasked with maintaining. Versions of the code had been in production for over fifteen years. Much of the code had been ported from C++ & Tcl to Java (although C++/Tcl systems remained), but the threading model was somewhat different between the two, and the port had not taken account of all the differences. It would have been nice to be able to rewrite some key parts of the system to make them more maintainable, but there was never enough time for that in a big way -- and realistically, bigger rewrites likely introduce new issues. Still, eventually we got most of the worst deadlocks and memory leaks and similar such things fixed and the system got to the point where people stopped even remembering off-hand the last time a core part of the system needed to be rebooted (previously a fairly frequent event). But each deadlock could involve days, weeks, or even months of study and discussion, adding log statements, writing tests, lab tests, analyzing quite a few multi-gigabyte log files (and writing tools to help with that including visualizing internal message flow), and so on. And, same as you mention, hardware and OS issues could interact with it all, making some things hard to duplicate under virtual machines for developers. One thing is that to the end user, a system that is more stable may not look that different than one that is less so -- there are no new features, so it is not obvious what is being paid for.
Although obviously if the program you support core dumps from a bad address or stack overflow, rather than just freezes up, it is probably something else. Still, even then, a bad pointer address can sometimes come from one thread freeing a data structure when another thread is still using it. The original C++ in the above mentioned project generally was highly reliable, but it still had some odd issues too. In one rare case, memory was freed in an unexpected way under certain conditions by other code running in the same thread but in code nested way deep with essentially recursive calls processing complex messages. I finally also traced part of that too what looked like maybe a bug in a supporting third-party library (a RogueWave data structure). Because that C++ code had been in production for years, and we were loathe to change it at the risk of introducing new issues, we mostly "fixed" that issue by making changes elsewhere in the system to prevent that component from getting the pattern of data that it had trouble handling. But we would not have known exactly what to change elsewhere without a lot of analysis.
Sadly, just as we got it mostly working well, the new shiny thing of a mostly COTS system that did something similar came along to replace much of it (at a much bigger expense than maintaining the old, but granted with some nice new features).
As I saw someone else comment recently about a "stable" OS, the end user generally cares more about how much work a system lets them get done, not how "stable" it is. A reboot can be acceptable, depending on the situation and the alternatives, even if not desirable. Erlang code is probably the master at that approach of rebooting code when it fails. :-) Here
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
No.
It would also be stamped on by management and any competent product owner, unless it was absolutely dripping in tests before he embarked on anything of the sort. If the code is producing the desired numbers but is simply a total and utter mess, no-one is going to thank him for declaring he's going to rebuild it from scratch, and the only way it would be sanctioned at all is if he could absolutely guarantee the same numbers before and after (to within rounding and ordering error). Given the state of the codebase he's talking about, those tests would have to be end-to-end tests since as others have noted writing unit tests for legacy code is in general a thankless and time-consuming task. (Then again, so is attempting to build end-to-end tests that satisfy every useful codepath.)
I genuinely have no idea how large the codebase at my company is; at a guess I'd wager we're in the many millions of lines of code (certainly enough to render Intellisense an utterly useless, chugging, unusable piece of shit) and quite possibly more. Some of it is really quite good code with thorough unit test coverage -- that tends to be the more recent stuff. The rest is covered, in principle if not in reality, by a large number of end-to-end tests that at the very least exercise some extremely fragile pieces of code quite effectively. Even with this, rampant refactoring is discouraged, let alone rampant rewriting. It soaks up developer time we can't afford to spend, and the danger of hitting a bug that isn't covered by our end-to-end tests (or, even more infuriatingly, fixing a bug that clients have grown to trust the results of) is pretty high. Unless there's a very good reason to out and out rewrite, it's to be very much discouraged. Careful refactoring, once every self-contained block of work rerunning all the unit tests and as many of the end-to-end tests as is practical, is about the only way to proceed.