Slashdot Mirror
Memory Checker Tools For C++?
An anonymous reader writes "These newfangled memory-managed languages like Java and C# leave an old C++ dev like me feeling like I am missing the love. Are there any good C++ tools out there that do really good memory validation and heap checking? I have used BoundsChecker but I was looking for something a little faster. For my problem I happen to need something that will work on Windows XP 64. It's a legacy app so I can't just use Boosts' uber nifty shared_ptr. Thanks for any ideas."
boost::shared_ptr is not a memory checker, it's a reference-counted smart pointer, and works fine in legacy apps (such as compiled under VC++ 6).
This isn't really an answer to your question, but it's on topic and there's some questions I wanted to get answered myself.
First of all, shared_ptr is going into the standard library as part of TR1. Does anyone know when common development environments, i.e. GCC and MSVC, will start including TR1?
Second, I believe that there are a number of garbage collectors available as libraries for C++. I've heard boehm's garbage collector mentioned numerous times. My question is, are any of these libraries any good? Are they really practical to use in real world applications? Do you have to modify all of your types to use it, or can primitive and stl types work with it?
Some time ago i was using valgrind, but afaik it does not run under windows. I think that MS Dev has some memory leak detection built in, but it is far behind valgrind. Besides, who codes stable stuff for windows? :)
Yep, Paul Nettle's little memory manager rocks. It WILL find leaks in your program. http://www.paulnettle.com/ (Yes, you have to navigate through a horrible flash site to get it, but it's worth it).
Use valgrind. It's for Linux only, but what it does is invaluable for most of the tasks. I don't know of any other tool of such help.
Note this is not 'memory management tool', but one to help you find and clean up the memory leaks. There is no way to do proper garbage collection using the STL's allocators, though there is a 'gc' library http://www.hpl.hp.com/personal/Hans_Boehm/gc/ which tends to do the job. Haven't used it, though projects like Mono http://www.mono-project.org/ use it extensively.
Purify works quite well. It is big and slow, and doens't play nicely with ACE_Tao stuff sometimes (tao do cleaver things that scare s purify sometimes)
I know this isn't exactly what the article is looking for. But, if you are using the STL (which you SHOULD be!) you may be interested to know that the STLPort STL implementation includes a debug mode which contains loads of error checking to make sure you aren't misusing STL.
They might be useful for small apps but if you have a massive app they are almost more trouble than they are worth.
It's hard to say what you can do except foster safe coding practice and highlight the common pitfalls such as memory leaks, buffer overflows etc. Many compilers can help detect heap / memory overruns because the debug libs put guard bytes on the stack & heap that trigger exceptions when something bad happens. There are also 3rd party libs such as Boehm which help with memory leeak / garbage collection issues and dump stats. I'd say using STL & Boost is also a very good way of minimizing errors too simply because doing so avoids having to write your own implementations of arrays, strings etc. which are bound to be less stable.
Talk about a sledgehammer to crack a nut. Boost strikes me as the sort of library used by people who want to show off how up to date their skills are , not people who really need to write a program to get a job done. Its bloated , has a wierd syntax that differs from the C++ norm and doesn't solve any problem that isn't already solved or could be done quite easily by standard C++ anyway. What is its point except as intellectual masturbation by its authors? No this isn't a Troll, this is a post by someone who was forced to use Boost for a year and I loath it. Yeah , mod me down , whatever...
PageHeap is a debugging tool for Windows created by Microsoft. It does what you want.
For more information look here:
http://support.microsoft.com/kb/286470
Use valgrind : www.valgrind.org It is (in my opinion) the best tool available for this purpose. In fact, I develop C++ exclusively on linux first because of valgrind, then port to Win32 later. By the way, you're not missing out on anything special by not programming in Java or C#. Both of those languages are slow, and introduce their own language complexities. -bms20
I haven't used it for a while, but I used to use Bruce Peren's efence for bits of malloc debugging, it hasn't been actively developed for ages but it's pretty light weight if that's what you need. There appears to be an up to date branch DUMA which I haven't tried. As far as I remember you can use efence under WIN and DUMA claims to work......
Unfortunately, what you prolly want is valgrind or purify.
Hmmmmmm
I have tried mainly Boundschecker and Purify, and they were usually quite slow and difficult to set up and produced lots of spurious results. Also, quite often they simply didn't work at all and refused to run certain programs. A few years ago I reduced the problem to a 10 line C++ program that would crash Boundschecker or Purify, can't remember anymore which one it was.
o r.asp
In any case, Visual Leak Detector is a free memory checking tool. It's only for Windows / Visual Studio, but if you are using that, VLD is awesome: http://www.codeproject.com/tools/visualleakdetect
It's super easy to set up, just #include "vld.h" somewhere in your program, and then run the debug mode. No need to rebuild everything in instrumented mode, and no false results (at least I haven't got any). Real memory leaks will be reported in the output window of the IDE.
I used to select tools for my team. For UNIX, I selected Purify and for Windows, BoundsChecker. After a few days with BoundsSlacker, **all** and I mean every one of my team was asking for Purify for Windows licenses. 17 licenses later and our code crashes have dropped byat least 90%.
Yes, it isn't cheap. Just do it. You'll thank me.
The productivity increase alone will make it worthwhile for management.
Valgrind's default memcheck tool is an excellent way of finding memory errors - ranging from extremely subtle to obvious. In addition, Valgrind can be used as a code profiler, cache simulator and many other things. It really is an excellent tool - I recommend it to anybody writing C++.
"It is better to die for an idea that will live than to live for an idea that will die" - Steve Biko
Last week on Gamasutra was a good article on memory leak detection, and how to role your own tool:
"Monitoring Your PC's Memory Usage For Game Development"
While the title says it's for game development, I found that the meat of the article applies to any windows developer.
For example, this code has serious issues: extern string method_that_returns_string_object();
char *ptr;
.
.
.
ptr = method_that_returns_string_object();
.
.
And FWIW, I've used Purify on massive apps, and found huge problems that the developers didn't even know were there. On one project, they couldn't explain why their "perfect" app kept crashing, either. Worse for them, I had been hired as a consultant to fix their problems that they couldn't seem believe existed (HINT: your boss hired someone from the outside...), and after watching the team flail and spend literally almost a man-year trying to find one memory bug, I finally had enough of "advice giving" being ignored and got on their system, linked their app under Purify, ran it, and found the bug - a double delete of an object from two different threads. It all took me about fifteen minutes. I did that in front of their management. I made my point.
Purify (and like tools) are a great help. Not using them is like trying to build a house without power tools. Yeah, it can be done. But what would you think if hired a builder to make your house and his team showed up carrying hand saws? Oh, and you are paying that team to hand-saw all the lumber...
What would you think of that builder?
Yet, when a developer asks for tools like Purify, management often balks. Because 1) they're shortsighted, and 2) developers don't know how to use such tools.
Like I said - what would you think of a construction company where the workers don't know how to use modern power tools to help their productivity?
Well, you just put yourself in that category.
Yes, Purify is somewhat slower than running without Purify. But it's a lot faster than most other full-memory checking methods. If you're worried about speed, link against the Win32 debug libraries - they'll at least show problems with double free() calls, access of free()'d and deleted objects, etc. And without too much performance problems.
We are running many high speed financial message processing applications. A crash for any reason (including a leak) would be very costly for us.
.NET. Both sets of colleagues have had major performance problems caused directly by the garbage collectors kicking in and consuming vast CPU power while they did their thing. The result was a failure to process messages in a timely manner in our high speed environment. The solution in both languages was to use pools of reusable objects and never cause their reference counts to drop to 0. Thus they implemented the very same mechanism that we use in C++ and avoided the garbage collectors.
We pre-allocate pools of objects at startup and then re-use them. No other memory is allocated or freed while the process is running. Our pools of reusable objects are monitored very carefully as an object that isn't release back to its pool when the job is done is akin to a memory leak. Use of sentries to automatically release objects back to the pools when they fall out of scope is mandatory.
So my answer is to the problem is:
1. Use sentries (or some other mechanism) to guarantee memory is released.
2. Don't allocate except at startup.
3. No need for elaborate tools due to the above.
I'm sure that not all applications data usage would fit into this model, but it is surprising how many can.
We have seen some leaks in our applications. These were tracked down to STL internally leaking. They weren't generally very large and therefore we continue to live with them.
On the subject of garbage collectors, some of our colleagues use Java and
So don't think that a garbage collector is the solution. Perhaps in less demanding applications it is a potential answer.
Lastly, I strongly dislike anything from Rational. I find them overpriced unreliable bloatware (YMMV). Purify used to be good some time ago, but those days are long gone.
I echo what others have said above. You are a developer. You know your requirements. Build a simple tool to monitor and check your usage. For us it was managed pools of re-usable objects.
Have a look at memory validator. I don't know if it supports 64 bit applications, but it has a great list of features and is the only decent memory validation tool I've ever seen on Windows.
- Avoid pointer arithmetic.
But why use C++ if you're not using pointer arithmetic? If you're not doing that, go to Java. I know this sounds silly, and it opens religious issues, but (aside from legacy apps, like this one) why would I want to use C++ if I'm not doing that?
I use C because it's small, fast, convenient and portable. I can code something tiny quickly.
I use Java because it's an object oriented language that helps with complex app coding.
I use C++ because I want some of that Java stuff, but I want it to bind to my memory model more realistically. ie. Pointer arithmetic.
I have had pretty results from Memory Validator from Software Verify. (m l ) It'll slow down your app but I think it does a better job keeping things close to real time then purify.
http://www.softwareverify.com/cpp/memory/index.ht
-----------
Fight Entropy!!! Fight Entropy!!! Figth Etnropy! !
iFgth Etnrop!y ! giFth tErno!py ! giFt htrEno!p y! --- Well maybe
not...
Use a testing framework like Parasoft's CPP stuff http://www.parasoft.com/jsp/products/home.jsp?prod uct=CppTest
Australian running a company that does C# / C++ / Java / SQL / Python / Mathematica
Use valgrind.
If you can afford to make your app compile with mingw's gcc (or cygwin's gcc but mingw should be easier), then you may be able to use mudflap, which is a memory debugging system integrated into gcc. You just need to pass -fmudflap, and gcc will instrument the program at compilation time.
i ze-Options.html#Optimize-Optionsg ing (maybe slightly outdated)
One thing is that it used not to properly instrument some really basic C++ operation in gcc 4.1 (I don't remember what exactly, something like copy construction of an object containing pointers) and was reporting spurious leaks because of that. It may have been fixed in 4.2.
Search for "mudflap" in the following page: http://gcc.gnu.org/onlinedocs/gcc-4.2.0/gcc/Optim
As well as http://gcc.gnu.org/wiki/Mudflap%20Pointer%20Debug
Exactly my experience. I didn't know Electric Fence existed (and it may not have, this was back in '92-'93), so I wrote my own malloc replacement with bounds checking. It didn't eat up much more memory (I think around 64 bytes/allocation) or use a whole lot of CPU (basically, it walked the heap checking for corruption every N allocations, and N was configurable down to 1).
I still remember the first time I tested it; I allocated some memory, then dumped the heap. I saw the block I had allocated, but there was another 2K block allocated that I hadn't! Fortunately I was dumping the contents along with the size, and I quickly figured out that printf() was calling my allocator too! (I had written replacements for [mc]alloc() and free(), and used the same names so I could instrument existing code w/o having to rewrite it.)
Just junk food for thought...
Application Verifier comes in x86, ia64 and amd64 flavors.
This tool allows you to enable PageHeap for your process, which is heap corruption detection built into the OS heap implementation. Upon freeing a block of memory, PageHeap will break into your debugger spewing tracing that a block has been corrupted. It can also provide the call stack when the block was allocated. Newer heap validation features are available in progressively more recent OS releases.
GLIBC allows you to create hooks for the standard mem functions (malloc/realloc/free). Remember that g++ still calls these under new/delete so it works for C++ also.
One of our guys coded up a simple shared lib that can be loaded with LD_PRELOAD that sets simple hooks of printing memory locations for new/realloc/delete. He then wrote a perl script that kept track of these things and spit out anything that was malloc'ed and not realloc'ed or free'd.
I can't post it, because technically it's not my code it's my company's. But his shared lib code is just 300 lines long, and shouldn't be hard to duplicate. The perl log filter is even more straighforward. Each malloc gets saved. Each free removes the malloc. Each realloc removes the old malloc and adds a new one. Anything left over is a leak.
Override __malloc_initialize_hook with a pointer to your init_function. In your init_function, save the old functions at __malloc_hook __free_hook __memalign_hook and __realloc_hook and substitute your own. Now write your replacement functions, in it, do your logging and temporarily replce the old hooks and call the original functions, replace with your hook on the way out to get the next call. All of the hooks should be wrapped in a mutex to help re-entrancy problems.
It's not a full memory detector, just does leaks, but it's non-intrusive, requires no recompiles, and is the best way we have to leak detect our huge server long running code.
I can attest to this. A few months back I was doing cross development for FreeBSD/Linux and maintaining a development version for Windows.
My code worked fine in Windows - mostly, but still a bug remained that I suspected was in the renderer but couldn't prove it.
The code ran fine on FreeBSD without any problems but I couldn't run it for more that a few minutes on Linux without it crashing hard but at random times.
I ran my Windows code through BoundsChecker (which I've used for years and have found to be very effective). Came up fairly clean, I fixed a few things but I could not find the bug.
I wasn't familiar with any Linux debug tools but eventually tried Valgrind. It found the problem after the first run, and BoundsChecker didn't even give me a peep. I found I was stepping just outside a graphics buffer at certain times and writing in memory I didn't own. It was enlightening the bug was in the same code for FreeBSD/Linux/Windows but displayed completely different behavior.
Anyway, I have respect for Valgrind. Now, if it was only available for FreeBSD.
I don't mind if an implementation of STL has some hidden/private/singleton allocation pools behind the scenes to speed things up. What I find really friggin' annoying is that they never track those allocations and offer any form of "reset" function that you can call before exiting your app, so that the simplest global malloc-counting methods can audit for leaks. You shouldn't need an "SGI-STL-aware-and-compatible leak detector," you should only need a malloc leak detector.
[
>>If you write a lot of code, you WILL make mistakes like memory leaks. A lot of them. If you don't think so, you're living in
>>fantasy land and you're nowhere near as good a coder as you think you are.
Pfft.
Actually, good coding habits will indeed work.
We were three people coding a 100,000 line program, 0 memory leaks. C.
Why should I ever have to do that? Either I'm checking the correctness of operator+, then I'm interested in the implementation of it, and don't care at all about who might call it where for whatever reason. Or I'm checking the correctness of code which uses operator+, and then I already know where it is used (in the code I'm checking right now, and I don't care it it is called from anywhere else).
Indeed, with generic programming, the same code may call several different implementations of operator+, depending on what type it is used on. The same goes BTW for normal named functions. And the same is also true for virtual member functions (operator or not) in an OOP context.
I'd say if you have to find the callers of operator+ in order to check if it is implemented correctly, there's something fundamentally wrong with your code.
The Tao of math: The numbers you can count are not the real numbers.
Meanwhile , back in the real world...
No, not at all. The current C/C++ specifications permit compilers to transform code in ways that can interfere with a garbage collector. Fortunately compilers do not do that as often as they could, but it seems like something important that should be addressed.
See Hans' paper Simple Garbage-Collector Safety for details.
DNA just wants to be free...
I was a developer for a "top 5" web browser for 6 years. I often worked on porting to small platforms and I also focused on how to improve performance on these platforms.
Before I go into a rant, I must first rant about how much I hate Java. I feel that it was a great proof of concept and that they should have taken what they learned and went back and did it better. Java is a lot of great ideas implemented poorly due to lack of experience. I think they should have spent more time with the SmallTalk guys who actually almost had it right to begin with. Hell, evolving SmallTalk would have been far more intelligent than turning C++ into SmallTalk.
Ok... here's the thing. I tried a lot of competing products. I tried memory checkers, memory allocators (and SmartHeap is the shit!), I tried memory profilers, hand instrumented memory logging, etc... what I've learned are a few things...
Garbage collection (even reference counting) can improve performance greatly, but it had little or no impact on fragmentation. A system that I slapped together as a malloc/free new/delete override proved quite successful at drasticly improving browsing performance. What I did was that instead of deleting memory, I queued deletes and when the pool needed to be grown, I would process the deletes or I would process the deletes during idle cycles.
This just made the program seem faster during runtime... obviously, the added overhead just made it slower.
To explain why a web browser is one of the most rigorous tests of a memory environment... just think of the hundreds to thousands of DOM nodes/elements/etc..., script objects, images, etc... there are in sinlge page. Add that each element is typically represented by a single allocated object. Consider that images can decode to 100 megs in size (yes, it happens), most often closer to 2-3 megs for background images.
A web browser can't use a memory system optimized for specific object sizes.
Due to the dynamic nature of the objects, object reusability is not really an option
Scripts can grow or shrink memory usage thousands of times per second
Browsers typically contain 3rd party code from plugins which need to interact with the browser
I can go on and on... a web browser is possibly the worst memory management nightmare on the planet. Often I worked with customers that were developing their own operating system. They used to tell me that my web browser must suck because making it stable on their system was a pain in the ass and often required them to either change or rewrite their entire memory management system to get good performance on embedded devices. Then I'd explain to them that up until now, their memory manager has been having a friendly snow-ball fight with a penguin, now it's running for it's life from an avalanch caused by a mean Yeti.
So here's the deal, GC really didn't pay off for us... helped a little, but I have to say that once we started using reference counting and simple GC, the quality of the code got really poor. Just look at Symbian for an example of a product that suffers from using great memory management system that increases coding complexity 10 fold. It makes it so that you spend all your time coding for the memory manager and you run out of time to make the program itself work.
Now on the other hand, I played with a few Java web browsers and learned something important. Java is made for phones. If it has no other purpose (and I'm convinced it doesn't), it's for embedded devices. Because of relocatability, fragmentation doesn't occur (in a good VM) and applications run much better. The GC + Relocation system is REALLY REALLY REALLY good, if I were to start writing a new web browser today, I'd find a good alternative to Java and get moving on it.
Oh... last thing about auto-pointers, they're a blessing and a curse. For the most part, I find the best solution to be to use a proper system library like Qt instead of boost or STL. Qt seems to actu