Tools For Understanding Code?

ewhac writes "Having just recently taken a new job, I find myself confronted with an enormous pile of existing, unfamiliar code written for a (somewhat) unfamiliar platform — and an implicit expectation that I'll grok it all Real Soon Now. Simply firing up an editor and reading through it has proven unequal to the task. I'm familiar with cscope, but it doesn't really seem to analyze program structure; it's just a very fancy 'grep' package with a rudimentary understanding of C syntax. A new-ish tool called ncc looks promising, as it appears to be based on an actual C/C++ parser, but the UI is clunky, and there doesn't appear to be any facility for integrating/communicating with an editor. What sorts of tools do you use for effectively analyzing and understanding a large code base?"

Comments

Wait for cenqua's solution

[ccguy]

I hear that the commentator guys are finishing a new product that instead of commenting your code is able to comment other's.

Stepping Through

[blaster151]

I've always found that stepping through the debugger at runtime is a decent way to start making sense of a large code base. Easier, anyway, than trying to read static code printouts. Just set a breakpoint at a point of interest, fire up the application, and use it as a starting point. You get a sense for program flow and it's a great way to generate questions--lots of them. (What does class SuchAndSuch do? It looks like the application is handling remoting in such-and-such a fashion; is that right?) You can also choose one aspect of the architecture and selectively ignore or step over other aspects, building up your understanding one aspect at a time. In my case, with Visual Studio as a development environment, I can hover the mouse cursor over variable names to see their current values. In the case of variables of a certain type, like datasets or XML structures, I can use realtime visualizers to browse the contents and get a much better feel for what's going on.

If there's no one at your company that can help answer your questions and bring you up to speed, I feel for you - your employers ought to know enough to give you some extra margin. It can be very hard to take over a large code base without some human-to-human handover time.

Also, is it an object-oriented system? I assume that it's not, based on your post, but you don't say either way. If it is, the important aspects of program flow often live in the interactions between classes and objects and the business logic is decentralized. OO is great, but it can be harder to reverse-engineer business logic because it's distributed among various classes. A debugger that lets you step through running code is almost essential in this case.

Re:Stepping Through

[daVinci1980]

This post is dead on.

Place a breakpoint somewhere you think will get hit (e.g. main), and then start stepping over and into functions. I usually attack this problem as follows:

Place breakpoint. Use step-in functionality to drop down a ways into the program, looking at things as I go. What are they doing, how do they work, etc.

Once I feel like I understand how a section of code works, I step over that code on subsequent visits. If I feel like this isn't taking me fast enough, I let the program run for a bit, then randomly break the program and see where I am.

Lather, rinse, repeat.

Also, this should go without saying, but you should ask someone who works with you for a high-level overview of what the code is doing. The two of these combined should get you up to speed as quickly as possible.

Re:Stepping Through

[The_reformant]

Absolutely since joining the real world I have found the visual studio debugger my most prized tool. Somehow I managed all through my degree to never come into contact with one (probably because all the free ones are rubbish and most schools won't shell out for visual studio). I now extol the virtues of debugging to all and sundry!

Re:Stepping Through

[dupup]

The parent post is correct, IMO. In fact, I have found that, for me, the easiest way to start understanding a new code base is to jump with a bug or two to fix. It's a little painful at first, but a specific goal combined with judicious use of the debugger will help you understand how the system works more quickly.

Re:Stepping Through

[JesterXXV]

I don't think there's any replacement for talking to the real-live developers who wrote it. Failing that, any design documentation they left behind. Failing that, just get a task to do, and try to get it to work. Nothing like learning by doing.

Re:Stepping Through

[smittyoneeach]

I think unit tests are actually better, for code that is suited to being driven externally.
Pick a tool to wrap something, start writing little bits to excercise the code.
You can comment and version unit tests, giving a sense of history.
Debuggers, on the other hand, mostly exist in the present tense.
Sure, you learn something now, but how about some breadcrumbs for later?

Re:Stepping Through

[Assmasher]

I certainly think that stepping through is by far the most valuable method; however, it can be difficult when dealing with asynchronicity and/or parallelism. In those cases, commenting is the only solution that seems to help me... LOL.

Re:Stepping Through

[orclevegam]

Much as I would love to agree with you, unfortunately the world isn't always so accommodating. Sometimes you have to suck it up and stay with a job till you can find something better, and most employers won't let you toss anything out, let alone a major chunk of their code base. Doesn't matter if it's utter crap, they paid for it, and as far as their concerned turd polishing is better then starting from scratch even if starting from scratch would be a hell of a lot cheaper. Can't expect MBAs to understand the difference between good code and bad code, to them it's all just code, and as far as their concerned, the more the better. It's the old idiotic idea that more lines of code means a better product, therefor anything that reduces lines of code must be a bad thing.

Re:Stepping Through

[superwiz]

The guy asked about a large code base. I am assuming that means on the order of at least half a million lines. Stepping through the code won't even get you into most modules of something that big. Never mind that it will do nothing to help you understand that a certain chunk of the code is a module that gets used only under certain extraordinary conditions. To be sure, what you suggest is what you do on day 1. The post was essentially asking what do you do three weeks into it after you've understood what the loop in main does and yet you still don't know what's tied to what and how.

Re:Stepping Through

[Nethemas the Great]

Clearly you don't write (or at least read source for) applications of any substance as that would be mildly described as tedious if not impossible.

One of the best ways to understand code is to do so visually with the software equivalent of blueprints. UML is generally considered a very capable way of modeling/communicating both static structures and dynamic behavior of software. There exist any number of tools that are capable of reverse-engineering existing source into UML. Two tools that I consider to be more capable than others are IBM's Rational Rose, and No Magic's MagicDraw. If commercial products aren't a possibility there are likely a number of open-source/free tools--though likely of lesser ability--available. A Google search on "reverse engineering UML" should point you at some.

Re:Stepping Through

[jgarra23]

Clearly you don't write (or at least read source for) applications of any substance as that would be mildly described as tedious if not impossible.


I have no idea how you formulated this from parent based from 2 or 3 sentences.


One of the best ways to understand code is to do so visually with the software equivalent of blueprints. UML is generally considered a very capable way of modeling/communicating both static structures and dynamic behavior of software.

A lot of times a programmer is stuck without those tools for any number of reasons. A lot of times people are stuck with spaghetti code which there is no documentation or design pattern to work with. I think your answer is assuming that the planets are aligned and we live in a utopia. Do you have any suggestions for people who have to deal with reality?

Re:Stepping Through

[orclevegam]

Yes, I know the difference between their, they're and there. I noticed the mistake after I posted it. It's one of the few mistakes I seem to be prone to in writing.

As for using "lines of code", I don't, they do. It seems the biggest issue they have with rewriting code (or refactoring if you prefer) has something to do with the way it's budgeted and accounted for. Apparently adding new code/features to a project comes out of a different budget, than replacing or repairing already existing code does. Don't ask me why that is, I just know whenever we've tried to push to replace some horrendous piece of code they would tell us it wasn't in the budget, and as long as the code ran we weren't allowed to change it. We had to work our way around the bean counters eventually by carefully picking features to implement that touched on code we wanted to replace, then as part of implementing the feature we would rip out and rewrite the code we wanted to.

Re:Stepping Through

[plover]

(Warning: you asked!)

Well, the learning curve is certainly important in the real world, although I expect a professional to know his or her tools before they arrive on the job. But there are a metric crapload of things I like better about Visual Studio that make it a much more effective debugger than gdb, in my opinion. (Note that I am not a big gdb user, so I may be cutting it a bit short in the feature set here. My apologies in advance if I do so.)

Things I've found I prefer include many tool windows simultaneously showing the states of registers, memory, the call stack, an object or seven (expanded to show a few properties), and automatic resolution of virtually every symbol and name, including the operating system (although you have to download the symbol files for your OS version from Microsoft.) And you still have full navigation through the source.

Simply hovering the mouse over a symbol will bring up a tool-tip to display the contents. If you highlight an entire expression such as pFoo->pBar->Blah.count+7 and hover, the tooltip will display the calculated result.

You can set a temporary breakpoint by setting the cursor on a line of code and clicking "run to cursor." You can run, single step, run to the current cursor, or run till function return. That last one is great for re-entering a function multiple times to test different conditions.

The variables window contains the current call stack as a dropdown list -- changing the stack lets you see the newly-local variables. Watch windows can display data as hex or decimal, just right click and select. Watch entries can even be used as calculators (enter a literal value, such as 0xf0 + 12, and it will display the results.)

In the watch windows, you can also call arbitrary functions (good for testing without driving your code to that point) or other functions in your memory space, such as the C runtime memory checkers. If you're trying to track an errant pointer, create a debug build, start running and break, type _CrtCheckMemory() into a watch window, and every time the watch window is refreshed, it will check all your fenceposts. You might get lucky and spot your corruption as it happens. The /GZ compiler option will perform a similar task at the function level, but this would let you do it at a line level.

There are also dozens of possible formats it can display your watch variables in -- suffix a pointer with ,s and it'll display the contents as an ASCII string. Only see one byte because of Unicode? Suffix the pointer with ,su and you'll see the unicode string. A ,wm suffix displays window messages by name. ,hr suffix displays HRESULTs by name.

The memory windows will highlight in another color any data that's changed since the last time it was refreshed, whether it be a single step or a previous breakpoint. You can have memory displayed as bytes, shorts, or longs. And with the newer visual studios, you can have multiple memory windows, so you can keep track of two, three or four arrays simultaneously. You simply drag and drop them wherever they're convenient, then step through the code and watch for colored variables indicating change.

Again, all these windows are automatically updated every time the debugger drops from the program to your control. I've got two 17" monitors, and I can fill them both. The problem with debugging is that sometimes you are really starting blind, and the faster you can get more information, the less time you waste debugging.

There's a cute "magic trick" I like to show people with the memory window and the disassembly window. Let's say you've had a crash, and attached the debugger to the running program. You're looking at a corrupt stack in the call stack window -- just one line of garbage data. What to do? Where did it break? Enter @ESP in the memory window. Change the view to 'long' and it displays the memory as 8-digit numbers. If y

Re:Stepping Through

[plover]

Thanks for the compliment!

I used to teach a course in debugging with Visual Studio, and I basically trawled through my syllabus looking for the cool tricks. Using the stack-crash demo to drop into the source code of the crashing module is a real attention-grabber.

I found debugging in gdb to be a lot like debugging in WinDBG. You have to learn a lot of esoteric commands that you don't use very often, so it takes a lot of practice to learn them. And if you aren't constantly searching for the side effects of each step, you can miss a valuable clue. Seeing the color change on watched values that have changed is a great way to pick up on otherwise subtle corruptions. Seeing an entire object's value hierarchy go red because you munged its pointer really stands out, at least to my eye.

Here is the bibliography and recommended references from my syllabus. It's pretty out of date these days (I especially miss the C/C++ Journal,) but the references are still good if you can find them.

Bibliography

Debugging Applications by John Robbins (Microsoft Press, 2000, ISBN 0-7356-0886-5)

Visual C++ Development Stunts by Mike Blaszczak (Lecture at Tech Ed 98)

How to Debug Quickly and Effectively with Visual Studio 97 by Martyn Lovell (Lecture at Tech Ed 98)

MSDN Library Visual Studio 6.0 (Microsoft, 1998), see also
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvc60/html/memleaks.asp

Customising Autoexp_dat.doc, by EMCC Software, see http://www.emccsoft.com/devzone/tools/devstudio.html
The Bugslayer by John Robbins (Column in Microsoft Systems Journal / MSDN Magazine)
MFC mailing list, 1998-1999 (see http://www.microsoft.com/workshop/essentials/mail.asp)
2600 Magazine, Finding and Exploiting Bugs, Spring 2000, (2600 Enterprises, Inc.)

Resources

MSDN Library Visual Studio 6.0 (Microsoft, 1998), Visual C++ Documentation, Using Visual C++, Visual C++ Programmer's Guide, Debugging. Actually, the whole of the MSDN Visual Studio Library is my number one resource for Windows development. Lean on the F1 key first. It's available online at: http://msdn.microsoft.com/library/default.asp

Microsoft Knowledge Base: http://search.support.microsoft.com/kb/c.asp

MSDN Magazine (the combined former Microsoft Systems Journal and MIND Magazine) is the official Microsoft developer's publication. You'll find all the current and upcoming Microsoft acronyms detailed here. The writing is usually top notch, but the content is usually based on the over-hyped acronym of the month, and is frequently too specialized to be of real value. Since about 2000, it's been the official mouthpiece of .NET. But, it's pretty much required reading to stay on top of what Microsoft is rolling out the door next month.

C/C++ Users Journal is a cutting edge independent magazine that offers the latest developments in C++ techniques, STL work, exception handling research, and C++ language development. It has articles written to many levels, from beginners to experts.

Dr. Dobb's Journal is another independent magazine that more broadly approaches development with a wider variety of tools including C, C++, Java, perl, and Python. It is very strong in any subject it touches, but it is not Windows specific (it has a definite UNIX slant and a frequent anti-Microsoft bias) and some of it will be of less value to a Windows-only programmer.

Doxygen

[Raedwald]

For C++ code, Doxygen can be useful, as it shows the class inheritance. As requested, it uses a (rudimentary) parser. It works with several other languages too, although I can't vouch for its utility for them.

Re:Doxygen

[zeekec]

Doxygen can produce UML diagrams for undocumented code. (UML_LOOK and EXTRACT_ALL)

Re:Doxygen

[Bill_the_Engineer]

Doxygen I thought did java-doc like parsing for C++? I was thinking he should look for something able to build a UML diagram based on the code... I hate UML, but if there isn't any documentation telling you the structures of the code it might be a place to look.

Doxygen is more than a javadoc replacement.

I like Doxygen + Graphviz. Just set Doxygen to document all (instead of just the code with tags) and set it to generate class diagrams, call trees, and dependency graphs and allow it to generate a cross reference document that you can read using your web browser. Set the html generator to frame based, and your browsing of code will be easier. I would also set Doxygen to inline the code within the documentation.

I've use Doxygen to reverse engineer very large programs and had good luck with it. I will say Doxygen is not going to do all your work for you, but it will make your job easier. Especially if you add comments to the code as you figure each section out.

Now if you like to see the logical flow of each method then try JGrasp (jgrasp.org). It has a neat feature called CSD that allow you to follow the logic of the code a little better. It's a java based IDE so that may be a turn off for you. I do whole heartedly recommend the Doxygen (w/ Graphviz).

Good luck.

Re:Doxygen

[mhall119]

Only problem is, it is a pain to configure. Also, windows versions don't look very stable. Windows version has been very stable for me, I've not had any problems with either Doxygen or Graphviz. It also includes a configuration wizard that is both easy to understand and powerful. There is also an Eclipse plugin that lets you configure and run Doxygen.

When I was your age...

[russotto]

I use the Mark I eyeball, grep, emacs, and of course, the little gray cells.

(and GET OFF MY LAWN).

Re:When I was your age...

[Mr. Underbridge]

When I was your age...I use the Mark I eyeball, grep, emacs, and of course, the little gray cells.

(and GET OFF MY LAWN).

They have lawns at the old folks' homes these days?

Paper

[raddan]

You should really be sitting down and attempting to understand the code, ASAP. Asking Slashdot for fancy tools isn't really going to help you. The real barrier here is your own brain.

Re:Paper

[bunratty]

I don't think I've ever been able to understand a large body of code by simply looking at it. I've always found that attempting to make modifications (fixing bugs, adding features) to the code gets me to understand it fairly quickly. Often, I'll find myself adding comments or cleaning the code up as I go. There have been times when I've just thrown all the code away and reimplemented the same functionality form scratch. That may not be an option here, but perhaps writing an implementation of part of the code from scratch will help to gain an understanding how that particular feature is implemented.

Re:Paper

[cjonslashdot]

I agree. I have found that it is fairly easy to uncover program structure. But UNDERSTANDING the intention of each line or function is another matter. This is where one wishes that there were documentation of design decisions. This is why whenever I build something I simultaneously maintain a design document in which I record each decision that I make and each pattern that I devise and use. As I revisit decisions, I do it in the design, and only when I have worked out the design do I try to code it. This is not the traditional "big up front design" - it is an agile approach to design, attacking it incrementally and in a just-what-is-needed manner.

Re:Paper

[sdpuppy]

Well when all else fails, look at the variable/function/structure names.

Obviously a program with labels such as "Frodo" Sam" "Gondor" must be doing something Lordly with rings

and if you have labels such as "string1" "string2", then the program must be solving some particle physics problem involving string theory.

... and when that fails, you go back to your old college, find the smartest CS geek and slip him/her a few dollars to figure it out.

Need I add :-) :-) :-) ?

doxygen

[greywar]

If its in a language that doxygen can understand, thats the tool I would HIGHLY recommend.

Ctags

[pahoran]

google exuberant ctags and learn how to use the resulting tags file(s) with vim or your editor of choice

Old School

[geekoid]

Printouts and colored markers.

Re:How / why did you get the job...

[wampus]

Sometimes its hard to follow execution, especially in a large codebase. Its made even more difficult when a smug jackass wrote it to be as terse as possible.

Understand C++

[SparkleMotion88]

Sorry I don't have an open source tool for you, but I've used Understand for C++ in the past and it was pretty helpful. To me, the most useful piece of information for understanding a large codebase is a browseable call graph. I'm sure there are simpler tools out there that generate a call graph, but this is the only one I've used with C++.

RR & EA

[Anonymous Coward]

Sometimes tools like Rational Rose or Enterprise Architect are successful at reading in the code an building a UML model that you can then attempt to parse through. I'm not familiar with the use of either, but I know it can be done, with mixed results depending on the size and complexity of the code being analyzed. Both tools are fairly expensive though, I believe.

Re:How / why did you get the job...

[Jeremi]

One might as well ask, why are you posting smarmy retorts when you clearly didn't understand the question? The question was about understanding the program, not the underlying language.

You must have inherited my old project

[theophilosophilus]

Sorry about that.

What I do

[laughing_badger]

SourceNavigator : A good visualisation package http://sourcenav.sourceforge.net/

ETrace : Run-time tracing http://freshmeat.net/projects/etrace/

This book is worth a read http://www.spinellis.gr/codereading/

Draw some static graphs of functions of interest using CodeViz http://freshmeat.net/projects/codeviz/

Write lots of notes, preferably on paper with a pen rather than electronically.

Answer

[hey!]

Yes. Understanding code is one of thing things you hire tools for.
...

Wait, were you talking about software?

doxygen - with full source option

[mhackarbie]

I agree with the previous recommendations for Doxygen. A while back I wanted to become familiar with the source code for a game engine and tried various tools to help with the 'grok' factor. I found the doxygen docs, with full source code generation in html, to be the fastest and most convenient way to walk around the code. After a while, it just clicked.

Creating small demo apps that use the code can also help.

mhack

Re:Doxygen, and Extracting Software Architectures

[Mr.Bananas]

I use Doxygen for C code, and it is really helpful. One of its most useful features is that it generates caller and callee graphs for all functions. You can also browse the code itself in the generated HTML pages, and the function calls are turned into links to the implementation. Data structures and file includes are also pictorially graphed for easy browsing.

If the system you need to understand has a really big undocumented architecture, then this presentation might be useful to you (there is a research paper, but it's not free yet). In it, the authors present a systematic method of extracting the underlying architecture of the Linux kernel.

GNU Global

[Masa]

GNU Global is able to generate a set of HTML pages from C/C++ source code. This tool has helped me several times. All member variables, functions, classes and class instances are hyperlinks. It provides an easy way to examine source code. It also provides tags for several text editors (for Vim and Emacs especially). http://www.gnu.org/software/global/

Umm.. documentation?

[Anonymous Coward]

Seriously folks, having spent large chunks of my working life having to decipher the mess of those who came before me I cannot stress enough the importance of clear comments, variable/function names, and consistent and readable syntax. AND WRITE F@#$%ing HUMAN READABLE DOCUMENTS DESCRIBING FUNCTIONAL REQUIREMENTS, ALGORITHMS USED, LESSONS LEARNED, ETC.
Calling all your variables "pook" or the like may be very cute, but does not help me figure out what the heck the function is supposed to do or why I would ever want to call it. Yes it's a pain. Yes we're all under time deadlines and want to get it working first and go back and document it later. And yes, it WILL bite you in the ass (ever heard of karma? your own memory can go and then you have to decipher your OWN code!).

That said, if you have inherited a code base from someone who ignored the above, go through and generate the documentation yourself. Write flow charts and software diagrams showing what gets called where and why. Derive the equations and algorithms used in each piece and figure out why the constant values are what they are. Finally, start at the main function or reset vector (I do a lot of microcontroller development) and trace the execution path.

Osmosis

[Greyfox]

If the original developer made useful comments that will help immensely. If there's a design document showing how the program fits together that helps a lot. If there's a process document explaining the business logic the application implements, that helps a lot. On average you'll start with a marginal code base with no comments, no design documents and no explanation of what the application is attempting to accomplish.

Get the guys who use it to explain what they're trying to do, read the code for a couple of days and then have them show you how they use the application. Then plan on six months to a year to get to the point where you can look at buggy output and know immediately where the failure is occurring. In the mean time just work in it as much as you can and don't try to redesign major parts of it until you know what it's doing.

Don't attempt the impossible...

[namgge]

and an implicit expectation that I'll grok it all Real Soon Now

It is unlikely that your job is really to 'grok it all'. Most likely there are specific issues that need to be solved - stop panicking and pick the simplest one on the list and start working on it.

In a similar position to you, I followed Brook's advice to study on the data structures and found it good. Also just running the application under a debugger, inserting breaks in important looking code and then having a look at the call stack when that code was used also proved enlightening. A good debugger also lets you explore the data structures.

When smart-asses tell you "Bill would have fixed that in ten minutes." I recommend replying "I never met Bill, why do you think he left?"

Namgge

Etags

[__david__]

Emacs and etags are your friend. Meta-. zips to the function under the cursor. C-s for incremental search. Meta-x grep-find for any other search.

Also, run the program with a debugger and step through it. Or put some print statements in key places and see what it produces.

I find that's all I ever need.

Understand the design first, then the code

[Anonymous Brave Guy]

I'm afraid you've set yourself an almost impossible task. IME, there are no shortcuts here, and it it's going to take anywhere from a few months to a couple of years for a new developer to really get their head around a large, unfamiliar code base.

That said, I recommend against just diving in to some random bit of code. You'll probably never need most of it. Heck, I've never read the majority of the code of the project I work on, and that's after several years, with approx 1M lines to consider.

You need to get the big picture instead. Identify the entry point(s), and look for the major functions they call, and so on down until you start to get a feel for how the work is broken down. Look for the major data structures and code operating on them as well, because if you can establish the important data flows in the program you'll be well on your way. Hopefully the design is fairly modular, and if you're in OO world or you're working in a language with packages, looking at how the modules fit together can help a lot too. Any good IDE will have some basic tools to plot things like call graphs and inheritance/containment diagrams, if not there are tools like Doxygen that can do some of it independently.

If you're working on a large code base without a decent overall design that you can grok within a few days, then I'm afraid you're doomed and no amount of tools or documentation or reading files full of code will help you. Projects in that state invariably die, usually slowly and painfully, IME.

Re:The best tool

[Anonymous Coward]

The best programmers I've ever worked with didn't have degrees. But some of the worst ones did.

Look at doxygen/umbrello

[Yiliar]

See:
http://www.stack.nl/~dimitri/doxygen/
and:
http://uml.sourceforge.net/index.php

These tools allow you to 'visualize' a codebase in several very helpful ways.
One important way is to generate connection graphs of all functions.
These images can look like a mess, or a huge rail yard with hundreds of connections.
The modules, libraries, or source files that are a real jumble of crossconnected lines are a clear indication of where to start clean up activities. :)

Good luck!

Wait 'till you get to reading the specs...

[crovira]

That should be good for a laugh or three.

They'll be out of date, full of inconsistencies and incomplete.

Then you'll be reading the code only to discover that people's idiosyncrasies and personalities definitely affects their coding styles. (There's even some gender bias where women tend to set a lot of flags [sometimes quite needlessly] and decided what to do later in the execution while men code as if they knew where they were going all the time, just that when they get there, they're missing some piece of information or other.)

If you read code developed by a whole team of people, you'll get to know them, intimately.

Good luck. You'll be at the bar in no time... I kept the stool warm for you.

The Slashdot attitude

[gaspyy]

I'm appalled by some of the comments that imply that the poster may not be fit for the job.

A few years back I had to maintain a large module written in C#. I had about 200K lines of code, 50 classes, zero documentation, zero comments, zero error logging support, and I was expected to find and fix bugs and add functionality the day after the module was handled over.

So if you were never in this position, just STFU. Yeah, the code is there, but is this flag for? Is this part really used, or is obsolete? What are the side-effects of using that method? And so on...

Eventually, I learned it, especially after some intensive debugging sessions, but it was frustrating to say the least. I would have loved to have some aiding tools.

Re:Umm.. documentation?

[Skewray]

Why? I can write crap and you can clean it up. This is Division of Labor, which is the basis of our civilization.

Where be dragons?

[mm4]

Apart from Understand for C++, I'd also suggest SourceMonitor - http://www.campwoodsw.com/sm20.html It will at least quickly point you to potentially problematic parts (long functions, deep nesting, etc.).

Mod parent up

[mccrew]

Sorry, no points today to mod you up myself.

I would suggest a slight variation on the theme. Fire up the application, start it on one of its typical tasks, and then interrupt it in the debugger to catch it. While the process is stopped mid-flight, take note of the call stack to see which classes and methods are being used. Maybe step through a few calls, then let the program run some more.

By doing this repeatedly, you will quickly get a sense for which parts of the code see the most action, and would provide the most obvious places to start studying the code base, and provide the best bang-for-buck return on your time.

Re:Mod parent up

[Lazerf4rt]

Fire up the application, start it on one of its typical tasks, and then interrupt it in the debugger to catch it. While the process is stopped mid-flight, take note of the call stack.

Good advice -- breaking randomly. However, it works best in CPU-intensive applications. If the app is mostly idle and event-driven, you're best off searching the code and looking for a place to set breakpoints.

Also, when I use the debugger to help understand some new code, often I'll open a text file and build a "trace" as I go. As I explore things in the debugger and find new call stacks, I add more detail to the trace, in a hierarchical (indented) style. Then I save the traces in case I forget something later.

As for the original question, I would recommend staying focused. Don't go all over the program trying to understand every system at once. Pick a specific part you really need to understand (say, based on a task you have to do) and focus on understanding that.

Unfortunately, the best tool for understanding code is experience. Not theory and not some fancy visualization program. Once you've seen a lot of different code, you come to recognize what each person was thinking when they wrote it. Once that kind of thing comes easily, you no longer find it necessary to bitch about each different programmer's coding style (as some do). So in a way, the guy who posts this question is lucky to have such a big pile of code in front of him.

Re:Mod parent up

[ckaminski]

The best way to learn the code is to start fixing some low or medium severity bugs. Something that's not a sev1 is either not so endemic to the system that changing it breaks everything, nor is it likely to be some random data corruption issue that will be impossible to find. It will be stupid user-input problems, or interaction issues.

Most of my productive code learning was in the first three months of bug-fixing. I think that's why most newhires end up on bug fixing as a rule - it's the fast-path to comprehension.

Re:Mod parent up

[Just Some Guy]

By doing this repeatedly, you will quickly get a sense for which parts of the code see the most action, and would provide the most obvious places to start studying the code base, and provide the best bang-for-buck return on your time.

If only there were some way to automatically generate this information, this "profile" of the running code, if you will.

Re:How / why did you get the job...

[PetriBORG]

Only 1600 lines?

I used to work at a company with a lot of Pascal and C code... It was extremely common (as in, all but a few) for programs to be written entirely in one code file. These files would go on for 20,000 lines or more. So many lines in fact that after the compiler had imported the header files at the top of the file that they would be over 65,000 lines long and the debugger would crap out because it had exceeded the int that it used for line number counting.

Sadly this isn't a joke.

HTML based cross reference

[NullProg]

Run these commands (or put them in a script):

ctags *
gtags
htags -Fan

It will create a ~\HTML folder with all the function/variables cross-referenced. Open the file index.html or mains.html in your browser. If your not running Linux, I think these utilities are included in cygwin http://www.cygwin.com/

Enjoy,

Browse-by-Query

[mmacdona86]

I'll plug my own open-source project for this:
Browse-by-Query-- it won't help with C/C++(sorry for the original questioner), but it will handle Java or C#.
It dumps the code into a database and lets you query it to find the relationships.
I'm biased, of course, but I've found it's just the thing to understand how a particular piece of functionality in an unfamiliar code base fits into the big picture.

Not for a "large" codebase...

[smitth1276]

That doesn't always work for a code base with millions of lines of atrociously written code. I've worked with code where it is absolutely not feasible to step through everything.

It seems like in those cases I end up working from effects... I note some program behavior and then try to find exactly what causes that behavior, which can be surprisingly difficult if you are dealing with the "right" kind of code. After a while, though, the patterns begin to emerge in the system as a whole.

Re:Not for a "large" codebase...

[ChrisA90278]

"That doesn't always work for a code base with millions of lines of atrociously written code. I've worked with code where it is absolutely not feasible to step through everything"

You are correct. All these people talking about using a debugger and so on... That does NOT work on larger projects any on fairly simple ones. "Large" projects might have 250 source code files and thousands of functions or classes and likely a dozen or so interacting executable programs. I've seen print outs of source code that fill five bookcase shelves. No one could ever read that.

I've had to come up to speed on million+ lines of code projects many times. The tool i use is pencil and paper

The first step is to become an expert user of the software. Just run the thing, a lot and learn what it does. Looking at code is pointless untill yu know it well as a user.

Re:Not for a "large" codebase...

[no-body]

Run it through a profiler - giving function names, times called, cpu time used, calling hirarchy/tree

if... there is such an animal around still for the environment in question.

Absolute tosh !

[golodh]

An interesting post, even if it's absolute tosh. No-one in his right mind tackles a new code-base of any size or complexity with nothing but a printout. Not if he's expected to understand how it works and/or maintain it in a responsible way.

In fact, it nicely highlights the difference between "software engineers" and "code monkeys". Code monkeys just dive in; they never pause to think. In fact ... they tend to avoid thinking. It's not their strong point. After all ... they're paid to code, right? Not to think. Software engineers on the other hand, look before they leap and spot the places where they need to pay attention first. And they're systematic about it.

In fact, a software engineer will happily spend a day or two putting the right tools in place, *including* a full backup and a proper version management system for when he's going to have to touch anything.

The first thing you want to know about a new code base (after you find out what it's supposed to be doing) is its structure. Tools like Doxygen (see previous posts) show you that structure *far* quicker and *far* more reliably than any amount of dumb code-browsing can. And besides ... once you do it, you've got that documentation stashed away securely instead of milling around incoherently in your head (you'll have completely forgotten most of what you read by next month) or on disorganised pieces of note paper.

The second thing is to figure out if it calls any "large" functionalities like subroutine libraries or even stand-alone programs like databases, let alone if it makes operating system calls. The call-tree will give you an excellent view, and the linker files can complete the picture. You wouldn't be the first maintenance programmer who found out after months that his application critically depends on some other application he wasn't told about.

The third thing is to see where your code does dirty things. Let the compiler help you. Just compile your application with warnings on and have a look at what the compiler comes up with. You might be surprised (and horrified). Then compile with the settings used by your predecessor and check that your executable is bit-for-bit identical to what's running (you wouldn't be the first sucker who's given a slightly-off code base).

If performance is at all important, then running the whole thing for a night on a standard case under a good profiler will also tells you lots of important things. Starting with where your code spends its time, where it allocated memory and how much, and where the heavily-used bits of code are. All neatly written down in the profiler logs.

Finally, run your application with a tool to detect memory management errors the first chance you get. Useful tools are Valgrind (in a Linux environment), Purify (expensive, but probably worth it) under Windows, and sundry proprietary utilities under Unix. Just about 90% of the errors made in C programs come from memory management problems, and half of them don't show up except through memory leakage and overwritten variables (or stacks .. or buffers .. or whatever). You'll need all the help you can get here, and as far as these errors are concerned, dumb code browsing is useless. Just keep your head when looking at reports from such tools ... they can throw up false positives. Ask around on a forum with specific questions if you're allowed, or ask your supervisor. After all ... you showed due dilligence.

When you know all that (if you have the tools in place, all of this can be done within 1 day + 1 overnight run + 1 hour reading the profiler output), go ahead and trace through the code in a debugger. You'll be in a *far* better position to judge what you should be reading.

Re:Absolute tosh !

[mabhatter654]

I'd agree. He's being considered a "code monkey" and not a software engineer. Typical situation is that they'll drop some random user problem on his desk after a week "to familiarize himself" then expect him to figure out what program it is and why it broke and suggest a process improvement. Then tell him he's all wrong because "they already tried that 5 years ago."

The question he's trying to answer is what does the code "do"? why does it exist? what problem does it solve? When you inherit some homegrown ERP system for example, it's easy to find a bug in a routine... not so easy is why input from program A is displayed wrong in program E that is processed by B, C, & D then stored for a week. He's looking for a quick picture of what it all looks like.. in 90% of cases nobody has that info for the CURRENT version of their homegrown system...they might have made the flowcharts, data dictionaries, and code books years ago, but nobody keeps them current.. and DOCUMENTED. How do you get enough info in a short amount of time?

Re:How / why did you get the job...

[swillden]

...if you don't understand the language?

Yes, it's hard to understand questions when you don't understand the language.

I'm sure you can find some remedial English classes if you look.

Use UML, and focus on the interfaces

[davide marney]

If your project is object oriented, you may be able to get your UML modeling tool to import the code and visualize the classes. When you do this, you'll probably get a HUGE diagram that seems just as unwieldy as looking at the code. The trick is to apply a filter to the model, so you're not overwhelmed with detail. Your UML tool should be able to do that for you.

I recommend focusing on all interface classes first. This can give you a remarkably sane picture of a system, and will help you divide up the code into more conceptually meaningful chunks.

The tool I use is Enterprise Architect, which does quite a lot of heavy lifting yet is still inexpensive enough for me to own a personal copy.

Solution

[Chapter80]

I've always found that the most effective method of learning code is to inject a random line of code somewhere, and see what breaks. Two techniques: 1) print some official-looking error message, and 2) add a large value (a million or greater) to a number somewhere. Keep a nice chart of what you added, where:

Error 'Format Conversion Error, converting from Y2K to Z2L' added to module x1
Error 'Out of Memory Banks' added to module x2
Error 'Object Expected; found adjective instead' added to module x3
Error 'bitbucket 95% full; please empty' added to module x4
Added 1,000,042 to some random value in module x5
Added 5,555,555 to some random value in module x6

Not only will you learn about the code, you'll make a great impression on your boss, when, within minutes, you are able to resolve some mysterious problem that has never happened before.

More than tools

[sohp]

The best tool is your brain, applied liberally. Here's some thoughts to put in it

Feathers, Michael. Working Effectively with Legacy Code, Chapter 16 especially.

Spinellis, Diomidis. Code Reading: The Open Source Perspective, Chapter 10 lists some tools for you.

My own thoughts now. First, don't trust the comments, they are probably outdated. Second, if it's a big code base, forget the debugger. Write some little unit test cases that exercise the sections of code you need to understand, and assert what you think the code is supposed to do.

Finally, unless you are cursed with a codebase which is not kept in version control (in which case, ugh, time to start the jobhunt up again maybe), then take a look at the revision history. See what changes have been made to the area you are working on. With luck, someone will have put in a revision message that points you towards greater understanding of why a change was made, which will in turn nudge you towards knowing the purpose of the section of code that was change.

That was the best reply in the entire thread!

[JonTurner]

An *excellent* stragegy and thorough explanation. Especially the bit about stopping to think and devise a plan rather than just diving in headfirst. All spot on!

The only thing I could possibly add is to say "gather resources to understand the *purpose* of the system", either through documentation or by speaking with project management and/or end users. If you can learn the business rules and processes, that will be an enormous help in understanding the code's design.

Been there...

[seanadams.com]

There are two kinds of hard problems in programming: problems that are hard because they require ingenuity and deep thought, and problems that are hard because they require weeks of unraveling someone else's garbage.

There are some horrible programmers out there and I have on many occasions been tasked with cleaning up their messes. In your situation I would suggest either a) try to figure out if it would take less time for you to implement it in a clean and maintainable way or b) find someone else you can hire who knows the code base or at least is more familiar with the specific problem.

If you can't do a or b then you're screwed. In that situation, personally, I would either quit, ask for a different project, or print out the whole source code and sit back with a pen and start studying and commenting - one of the few tasks for which I still prefer dead trees.

Re:Been there...

[skiflyer]

a) is so often the wrong choice and can really submarine a company because they keep getting a cycle of a)'s ... every 5th release becomes a complete rewrite as the new team says "we need a refactoring of the code, no one here is familar with it and/or it's spaghetti code, just give us 5 months we'll maintain the behavior 100% and we'll clean up a lot of bugs and we promise in the future maintenance will be a breeze"

Re:Been there...

[seanadams.com]

Not everything is a duality.

Ah, so really there are two kinds of things: those which are dualities and those which are not?

Re:Been there...

[SnowZero]

Ah, so really there are two kinds of things: those which are dualities and those which are not? Finally, someone who understands both sides of the sphere.

I had a pile of C++ dropped in my lap 2 years ago.

[Richard Steiner]

My main tool for figuring it all out was to use exuberant ctags to create a tags file, and Nedit to navigate through the source under Solaris, with a little grep thrown in. I also used gdb with the DDD front-end to do a little real-time snooping.

I've since added both cscope and freescope, as well as the old Red Hat Source Navigator for good measure.

Tests

[gerddie]

Tests are indeed very good to understand a code base- Nearly all the last year I was working on a code base that nobody understood completely, although I had someone to ask about the general code structure. Writing tests helped me to understand what some parts of the code actually do. And where I needed to change things I could make myself sure that I didn't break anything.

Another great tool is valgrind+KCachegrind - it gives you really nice call trees. Vtune can do something similar as well, but IMHO the output is not as good as in KCachegrind. The only problem, of course, is that valgrind makes your program very slow and, it is, AFAIK, not available on MS Windows.Vtune, OTOH, runs the program at normal speed, but it's calltree output is ugly, at least on Linux.

If these two options are not for you than you might add a trace output to each function. IMO this is better than using a debugger - especially in C++ with BOOST and STL, where a lot of stepping goes through inline functions.With proper logging levels you can get a very useful output to see what's going on. It helps to understand the code, and it also helps, if you hit a bug.

Re:hmm.

[SageinaRage]

It's more like a carpenter asking for a nail gun because it's quicker, less tiring, with less change of damaging themselves. Any carpenter with any sense would ask for one, just like any coder with any sense would ask for these tools.

Source Insight

[Effugas]

It's inexpensive, and scales astonishingly. I've spent the last two years in it, and it's just how I audit code nowadays.

Shameless plug for CodeSurfer

[mmcdouga]

My company makes a code understanding tool called CodeSurfer. It's not open source, and it's not free (though it is free for academic use).

You can browse your code, following dependences and definitions. You can also construct queries, do isolate what statements can affect a particular variable, and a bunch of other tricks based on static analysis. There's a programming interface too.

Other good ways to get your head around code (speaking as a software engineer, rather than a guy promoting his company):

• I agree with whoever suggested breaking in a random spot and stepping through the code.
  
• Talk to the other developers, if they are around. Don't suffer in silence for the sake of doing it on your own.
  
• Pick a minor throwaway feature (eg every button should be blue) and modify the code to add that feature. This forces you really learn the code, but without the pressure of making a real product-worthy feature.
  

About "new tools"

[golodh]

@KZigurs,

Well ... some good points, and some I'd say are too detailed at this point.

I totally agree with point (1). I forgot to mention it since I assumed (always a bad thing) that the author actually could compile and run the thing. An important point to keep in mind. Thanks for bringing it up.

Points (2)-(5) however all come after you've understood the basic structure of your code base.

Next, I'd say that a fairly junior software engineer trying to tackle a large unknown code-base without proper tools is doomed to failure no matter what. So the step from "If you're in a rush" and "You are in a paid job and expected to deliver predictable results." to "forget about tools you're not familiar with and just dive in" is an exercise in self-delusion and a recipe for disaster. Nothing less. It's like someone rushing out of the house and sprinting for work because they don't know where they put the car keys or their bus ticket and feel they are too much in a rush to search for them.

Besides, producing automated documentation is a good way to communicate. The tool communicates the structure of the code-base to you, and you can use e,g, the call-graphs to (efficiently) communicate the complexity (or otherwise) of the code-base to your supervisor. It also communicates to him how you are approaching the problem, which is likely to be a plus.

Now suppose the codebase is really difficult. A competent software engineer is, like any other kind of engineer, co-responsible for making actual and potential trouble spots *visible* to management. Preferably before they explode. Although it's popular wisdom to despise Management, if you, the hands-on person, don't tell Management of the problems, you ensure that they're driving blind. You rob them of the chance do do anything about it before the problem becomes so acute that even they'll have to notice. They will recognise it if you do and keep it in mind when they have to assess you. Depend on it. Besides, you just happen to be the only one who can tell them, and you fail in your responsability if you don't. Part of a software engineer's job is to *communicate*. Now you can't give your supervisor any honest estimate of how well you have the new code base under control before you get to know it. And tools really really help you save time and allow you get a much better overview.

Communication works both ways. If, with all the tools you use, you are unable to understand the code-base, you lack one or perhaps two elements that distinguishes a basic software engineer from a good or even a great one. Talent and experience. And you should be honest with yourself and your supervisor about that too. If the job really is too hard for you, have the guts to own up before you mess up and thereby save yourself and your company a lot of trouble. And believe me ... there are lots and lots of good jobs in software development / maintenance that can be done without a surfeit of either. Such is the power of engineering.

Now Doxygen (or similar tools) may be unfamiliar to the author, but such tools really work. Besides, I've seen students download, understand, and use Doxygen in less than 1 hour after they were told about it.

Been there, will never return

[owndao]

I had a very wise undergrad EE prof who said on the first day of design class that we needn't worry about the many "complicated" things that we would have to design during the course because we had already completed all of our circuit analysis courses. He said it's much harder to figure out the details of someone's design than to design it yourself. Same applies here in software. I've been there working with other's undocumented code and quite frankly it was infrequently that I left the project with more respect for the programmer. Here I'll just say what I learned from the experiences as useless as it might be.

If the coding style used is appropriate you stand some chance. Lines of code don't matter much when behavior is sufficiently complex that you cannot list the states and events that trigger execution and state change let alone keep track of them in your head long enough to understand their context.

I once had a similar problem with some legacy OS9 c code that performed a simple communication task and updated a monitor. With no documentation from the writers I was to "simply add some new data to be collected and display it." The problem with this 3000 loc was that it was written as a state machine with no modularization - next to impossible to follow in a debugger. What I wanted to do is run a performance analyzer along with the code but I was told that was "out of budget". This would have told me at least the parts of code being executed frequently and I could start to associate the external events with the code processing.

On very large applications like AT&T's RNS (residential account management for BellSouth) that exceed million-lines-of-c-code the only thing that made the application workable for new features was the fact that it was created in a CMM III product environment thus it was well documented in design, development, testing, feature changes, bug fixes, etc. Even with all of this the number of processes and related data stores still showed a lot of bleed over and function duplication (there was no simple way to determine if a function was in existence that already did what you needed and even harder to determine if it was state data dependent and thus unusable in certain other states. Attempts by us (contract coders mainly) to get the company to allow us build a function-finding-tool/database to eliminate this problem fell on mostly deaf ears.

Because of this we had to depend on the longer-lived of the system architects to get an idea of where functionality existed. There were many times though when no one knew and weeks had to be spent reverse engineering communication structures, what the heck undocumented stretches of code did, re-write the documentation correctly and then start to implement the feature or correct the problem that had "been there for years." Management did not like the time taken to repair poor coding as this was not included as one our trackable metrics and therefore not in our feature/bug's budget (since it was not considered to be either).

RNS sounds bad but it was a breeze compared to that tightly optimized state machine code without documentation. So, my recommendations are:

1) If it is stream-of-thought-code (kind of like Faulkner's The Sound and the Fury), not modularized, not documented Tell your manager that it most likely will have to be re-designed to understand it fully. That means do an essential model of it's processing and data stores, use-cases, objects and events or whatever rigorous methodology you prefer. Then use that to re-write it. If management doesn't want to do that then you do not work for a company interested in maintainable code but wants a cheap fix. I would leave as soon as you get from them what they took from you in suckering you into the place.

2) If it is structured and/or developed in a "self-documenting-language" like Ada, Modula, Eiffel, etc. that forces structure (or at least makes it easier to write structured rather than unstructured), finish documenting it properly a