Slashdot Mirror
Memory Checker Tools For C++?
An anonymous reader writes "These newfangled memory-managed languages like Java and C# leave an old C++ dev like me feeling like I am missing the love. Are there any good C++ tools out there that do really good memory validation and heap checking? I have used BoundsChecker but I was looking for something a little faster. For my problem I happen to need something that will work on Windows XP 64. It's a legacy app so I can't just use Boosts' uber nifty shared_ptr. Thanks for any ideas."
boost::shared_ptr is not a memory checker, it's a reference-counted smart pointer, and works fine in legacy apps (such as compiled under VC++ 6).
This isn't really an answer to your question, but it's on topic and there's some questions I wanted to get answered myself.
First of all, shared_ptr is going into the standard library as part of TR1. Does anyone know when common development environments, i.e. GCC and MSVC, will start including TR1?
Second, I believe that there are a number of garbage collectors available as libraries for C++. I've heard boehm's garbage collector mentioned numerous times. My question is, are any of these libraries any good? Are they really practical to use in real world applications? Do you have to modify all of your types to use it, or can primitive and stl types work with it?
Some time ago i was using valgrind, but afaik it does not run under windows. I think that MS Dev has some memory leak detection built in, but it is far behind valgrind. Besides, who codes stable stuff for windows? :)
Yep, Paul Nettle's little memory manager rocks. It WILL find leaks in your program. http://www.paulnettle.com/ (Yes, you have to navigate through a horrible flash site to get it, but it's worth it).
Use valgrind. It's for Linux only, but what it does is invaluable for most of the tasks. I don't know of any other tool of such help.
Note this is not 'memory management tool', but one to help you find and clean up the memory leaks. There is no way to do proper garbage collection using the STL's allocators, though there is a 'gc' library http://www.hpl.hp.com/personal/Hans_Boehm/gc/ which tends to do the job. Haven't used it, though projects like Mono http://www.mono-project.org/ use it extensively.
Purify works quite well. It is big and slow, and doens't play nicely with ACE_Tao stuff sometimes (tao do cleaver things that scare s purify sometimes)
After seeing some of http://www.worsethanfailure.com/ OMGWTF C Calculator Contest submissions today, I do believe you could get some fine examples of what you're looking..
*hides*
Use valgrind to find the bugs and Hans Boehm's GC to not have to fix them.
If you didn't already know of such tools then go do some research; what you want probably already exists.
I know this isn't exactly what the article is looking for. But, if you are using the STL (which you SHOULD be!) you may be interested to know that the STLPort STL implementation includes a debug mode which contains loads of error checking to make sure you aren't misusing STL.
They might be useful for small apps but if you have a massive app they are almost more trouble than they are worth.
It's hard to say what you can do except foster safe coding practice and highlight the common pitfalls such as memory leaks, buffer overflows etc. Many compilers can help detect heap / memory overruns because the debug libs put guard bytes on the stack & heap that trigger exceptions when something bad happens. There are also 3rd party libs such as Boehm which help with memory leeak / garbage collection issues and dump stats. I'd say using STL & Boost is also a very good way of minimizing errors too simply because doing so avoids having to write your own implementations of arrays, strings etc. which are bound to be less stable.
Talk about a sledgehammer to crack a nut. Boost strikes me as the sort of library used by people who want to show off how up to date their skills are , not people who really need to write a program to get a job done. Its bloated , has a wierd syntax that differs from the C++ norm and doesn't solve any problem that isn't already solved or could be done quite easily by standard C++ anyway. What is its point except as intellectual masturbation by its authors? No this isn't a Troll, this is a post by someone who was forced to use Boost for a year and I loath it. Yeah , mod me down , whatever...
Try Purify: http://www.ibm.com/software/awdtools/purify/
PageHeap is a debugging tool for Windows created by Microsoft. It does what you want.
For more information look here:
http://support.microsoft.com/kb/286470
Use valgrind : www.valgrind.org It is (in my opinion) the best tool available for this purpose. In fact, I develop C++ exclusively on linux first because of valgrind, then port to Win32 later. By the way, you're not missing out on anything special by not programming in Java or C#. Both of those languages are slow, and introduce their own language complexities. -bms20
I am not sure about its availability for Windows XP, but I have used the Windows Server 2003 Resource Kit tools to successfully locate memory leaks in Windows dlls. First, you have to link with /DEBUG to get any meaningful results. Then run memtriage on the pid to get the heap address range and size allocation of all dlls attached to the given process, then use dh (dump heap) to determine which of the address ranges is for your resource.
If you see growth in successive executions of memtriage (which can be set to sample the heap over any defined time interval and for any number of iterations and all heap data is written to a log file) then analysis of corresponding executions of dh will tell you where memory is being allocated and not being freed.
I haven't used it for a while, but I used to use Bruce Peren's efence for bits of malloc debugging, it hasn't been actively developed for ages but it's pretty light weight if that's what you need. There appears to be an up to date branch DUMA which I haven't tried. As far as I remember you can use efence under WIN and DUMA claims to work......
Unfortunately, what you prolly want is valgrind or purify.
Hmmmmmm
Purify has been considered the best solution for this problem for many years. (How many readers have their 'Purify Mug'?)
Linux developers can use Valgrind, which is also very good and is free. But it won't run on your platform.
Then there are the static checking tools like Coverity. I believe that they do great things, though I have never used them. If you are a big company I think it would be well worth getting them to talk to you; you would probably find it intellectually interesting, if nothing else. There are other tools in the same field; Wikipedia has a list.
You may find that Purify is too slow. It has various options that you can tweak. It also benefits from having loads of RAM (steal it from your colleagues while they have lunch). But basically you need to live with the speed and either be patient or hack your application to go straight to the problematic bit.
In my experience this sort of debugging is always painful, and the lesson it teaches us is to *not put the f***ing bugs in the code in the first place!* By that I mean:
- Avoid dynamic memory allocation when possible (i.e. use std containers instead).
- Every time you type 'new' or 'malloc', think to yourself "where does this get deleted/freed?"; ideally the call to delete or free should be a few lines away from the call to new or malloc and it should be blindingly obvious that they occur in pairs.
- Be really clear about ownership of pointers.
- Use smart pointers (like the Boost scoped_ptr and shared_ptr) when appropriate.
- Avoid pointer arithmetic.
- Don't use a NULL sentinel value.
Every time you find yourself doing one of these "bad" things, try to remember your last epic all-night debug session with Purify and fix it....
By following these sorts of practices, I have managed to avoid any nasty memory-allocated related bugs for a few years. But of course it doesn't help with your legacy codebase.
I have tried mainly Boundschecker and Purify, and they were usually quite slow and difficult to set up and produced lots of spurious results. Also, quite often they simply didn't work at all and refused to run certain programs. A few years ago I reduced the problem to a 10 line C++ program that would crash Boundschecker or Purify, can't remember anymore which one it was.
o r.asp
In any case, Visual Leak Detector is a free memory checking tool. It's only for Windows / Visual Studio, but if you are using that, VLD is awesome: http://www.codeproject.com/tools/visualleakdetect
It's super easy to set up, just #include "vld.h" somewhere in your program, and then run the debug mode. No need to rebuild everything in instrumented mode, and no false results (at least I haven't got any). Real memory leaks will be reported in the output window of the IDE.
valgrind...oh but you're using windows.
I write C++ for embedded systems, and I needed a decent heap checker, so I rolled my own. It only took a few hours to write.
Of course, I basically copied the mechanism of tools like efence or DUMA, but I was able to tweak the tool to do exactly what I needed it to do. You may want to try the same approach.
-Seth
I used to select tools for my team. For UNIX, I selected Purify and for Windows, BoundsChecker. After a few days with BoundsSlacker, **all** and I mean every one of my team was asking for Purify for Windows licenses. 17 licenses later and our code crashes have dropped byat least 90%.
Yes, it isn't cheap. Just do it. You'll thank me.
The productivity increase alone will make it worthwhile for management.
... not
BoD
Does anyone know of any good memory checkers that can understand python's weird ass memory manager from back in '92.
Glowcode is a memory allocation watchdog and code profiler rolled into one. IMO, it has the best implemented features of its class, and is speedy and lightweight to boot. On top of that, it's useful for checking run-time memory leakage that may be masked by the deallocations that happen at the end of your program. I use it to find those odd leaks that only happen when you do a specific action, for example. A very useful tool.
node-def: a tactical hacking sim. Now in open beta.
Valgrind's default memcheck tool is an excellent way of finding memory errors - ranging from extremely subtle to obvious. In addition, Valgrind can be used as a code profiler, cache simulator and many other things. It really is an excellent tool - I recommend it to anybody writing C++.
"It is better to die for an idea that will live than to live for an idea that will die" - Steve Biko
How fast does Java run on a handheld device with a 67 MHz CPU and 4 MB of RAM? Over forty million of these devices have already been deployed.
Valgrind is, by far, one of the most useful tools i've ever came across for C/C++ developing. It's a shame it runs only on Linux - i bet a lot of Windows software would be much less leak-proof if such an excellent (and free) tool were available for them.
Last week on Gamasutra was a good article on memory leak detection, and how to role your own tool:
"Monitoring Your PC's Memory Usage For Game Development"
While the title says it's for game development, I found that the meat of the article applies to any windows developer.
Expensive little toy from Rational.
It profiles memory allocation on existing executables.
Produces thousands of lines of output, most of which you can ignore after revewing the code, but it does catch almost everything.
Old COBOL programmers never die. They just code in C.
Provides a lot more statistics than Purify or BoundsChecker.
Works with Visual Studio (C++ and VB), Metrowerks, Delphi, Fortran and third party allocators such as CherryStone (or even an allocator you wrote yourself).
Very customizable.
Some people run sessions with billions of allocations tracked by Memory Validator
http://www.softwareverify.com/testimonials.html
Stephen
(Disclaimer: I work for SVL)
Disclaimer - I have no connection with Gimpel (http://www.gimpel.com), but their tool PC-Lint is very good and quite cheap. I swear by it (not at it!). It's the best development tool I've bought, IMHO. The people at the forum are generally very handy. In case you're not aware it's a very powerful static analyser.
/analyze command line option (aka PreFast). This provides vaguely lint like static analysis and might be of some help - although I've found it to be pretty poor compared to PC Lint.
It can find a significant number of memory "abuses" in version 8 - if you get access to it/buy a copy, ensure you switch on at least the following warnings in your lint options file:
415
416
419
420
421
423
429
433
661
662
669
670
672
796
797
803
804
449
Also if you have the high end version of Visual Studio 2005 (Team Edition For Software Developers is the only one that has it), have a look at its
Mike
Linux fan and Win32 developer
I've done it in C before. Helps if you have already redefined malloc/calloc/realloc/free, if not then replace all calls to these (and new, delete etc) with calls to your versions. Make the code call macros which record the line and file number. IE -
void* my_malloc(size, linenum, filename);
#define my_malloc_macro(size) my_malloc(size, __LINE__, __FILE__)
my_malloc_macro is what code calls when it wants a chunk of heap.
When new memory is allocated my_malloc (or whatever) adds a copy of the calling line and file name to a list or hash table along with the pointer value and the size.
Then free searches the list of allocated memory for the relevant pointer value and removes it.
Then simply add another function to display the memory usage. You can see what's in place at any one time or (what I found useful) get it to dump out everything still allocated just before the program ends. Assuming you have a clean shutdown process.
Of course you get into problems with threads, I was lucky enough to be using a set of runtime libraries that provided a "Stop all other threads" and a "restart all other threads" call. It slows your program down significantly, but it still works well.
For example, this code has serious issues: extern string method_that_returns_string_object();
char *ptr;
.
.
.
ptr = method_that_returns_string_object();
.
.
And FWIW, I've used Purify on massive apps, and found huge problems that the developers didn't even know were there. On one project, they couldn't explain why their "perfect" app kept crashing, either. Worse for them, I had been hired as a consultant to fix their problems that they couldn't seem believe existed (HINT: your boss hired someone from the outside...), and after watching the team flail and spend literally almost a man-year trying to find one memory bug, I finally had enough of "advice giving" being ignored and got on their system, linked their app under Purify, ran it, and found the bug - a double delete of an object from two different threads. It all took me about fifteen minutes. I did that in front of their management. I made my point.
Purify (and like tools) are a great help. Not using them is like trying to build a house without power tools. Yeah, it can be done. But what would you think if hired a builder to make your house and his team showed up carrying hand saws? Oh, and you are paying that team to hand-saw all the lumber...
What would you think of that builder?
Yet, when a developer asks for tools like Purify, management often balks. Because 1) they're shortsighted, and 2) developers don't know how to use such tools.
Like I said - what would you think of a construction company where the workers don't know how to use modern power tools to help their productivity?
Well, you just put yourself in that category.
Yes, Purify is somewhat slower than running without Purify. But it's a lot faster than most other full-memory checking methods. If you're worried about speed, link against the Win32 debug libraries - they'll at least show problems with double free() calls, access of free()'d and deleted objects, etc. And without too much performance problems.
Why not write memleak free software ?? .. ojeah .. overloading the new and delete so you can track your own stuff isn't that hard.
If your app is already reasonably portable, consider porting to Linux solely to use Valgrind. It's that good. Otherwise, have you looked at Rational's Purify? I haven't used it myself but hear it has some pluses and minuses as compared to Valgrind, but is worth a look.
I develop cross platform applications, and I find valgrind incredibly helpful, especially when integrated into my unit tests and combined with a suppressions file that hides any false positives from platform libraries etc.
As for shared_ptr, please DO NOT think of shared_ptr as a complete memory management and leak protection solution. Smart pointers in general and shared_ptr in particular are NOT a magic band aid. Slapping shared_ptr use into a design won't necessarily fix your lifetime & memory management issues, it's quite likely to just hide them and introduce new and fun bugs where object 1 *thought* it was manipulating object 2's fred, but it was actually manipulating an obsolete and unreferenced copy of object 2's fred. And so on. Ownership and lifetime still need careful thought, and it can be better to use good old auto_ptr<>, direct membership, etc rather than shared_ptr in many situations. Also remember that the use of dtors that affect anything outside the object its self is likely to lead directly to foaming insanity when combined with shared_ptr use.
Get your ownership and lifetime issues thought out well, and *then* think about introducing a shared ownership model if the situation warrants. Don't just slap shared_ptr<foo> in everywhere you use foo* and expect magic.
We are running many high speed financial message processing applications. A crash for any reason (including a leak) would be very costly for us.
.NET. Both sets of colleagues have had major performance problems caused directly by the garbage collectors kicking in and consuming vast CPU power while they did their thing. The result was a failure to process messages in a timely manner in our high speed environment. The solution in both languages was to use pools of reusable objects and never cause their reference counts to drop to 0. Thus they implemented the very same mechanism that we use in C++ and avoided the garbage collectors.
We pre-allocate pools of objects at startup and then re-use them. No other memory is allocated or freed while the process is running. Our pools of reusable objects are monitored very carefully as an object that isn't release back to its pool when the job is done is akin to a memory leak. Use of sentries to automatically release objects back to the pools when they fall out of scope is mandatory.
So my answer is to the problem is:
1. Use sentries (or some other mechanism) to guarantee memory is released.
2. Don't allocate except at startup.
3. No need for elaborate tools due to the above.
I'm sure that not all applications data usage would fit into this model, but it is surprising how many can.
We have seen some leaks in our applications. These were tracked down to STL internally leaking. They weren't generally very large and therefore we continue to live with them.
On the subject of garbage collectors, some of our colleagues use Java and
So don't think that a garbage collector is the solution. Perhaps in less demanding applications it is a potential answer.
Lastly, I strongly dislike anything from Rational. I find them overpriced unreliable bloatware (YMMV). Purify used to be good some time ago, but those days are long gone.
I echo what others have said above. You are a developer. You know your requirements. Build a simple tool to monitor and check your usage. For us it was managed pools of re-usable objects.
If you write a lot of code, you WILL make mistakes like memory leaks. A lot of them. If you don't think so, you're living in fantasy land and you're nowhere near as good a coder as you think you are.
I'd say that one of the very first steps in making yourself into truly a good coder is realizing that you WILL make LOTS of mistakes. Which is why every truly good coder I've ever met has absolutely no qualms about letting anyone else see their code - good coders want more eyes to look for the mistakes they know they made.
Of course, when you termed them "skilled coders", you didn't characterize the level of skill. "Poorly skilled" is still a subset of "skilled"...
Can't find the best place to shout this loud enough "Valgrind is AMAZING!" :-)
It really is awesome. I used it on my library of portable data structures (C++ XML parser, compression and encryption stuff). I tested on Linux and still got the benefit on Symbian and Win Mobile when I recompiled it for those platforms.
I had a test suite that exercised a fair it of the library and ran the tests on Linux under Valgrind. The test suite would often run perfectly but Valgrind would spot memory handling errors that didn't have any consequences in that particular test but would have in other circumstances.
You don't have to compile special support for it into your programs. It produces some useful output even for non-debug versions of your code.
What Valgrind does is take you 50% of the way towards the convenience of a garbage collected language. You spend a lot less time farting around with minor details but still more than if you were using a GC langage. Valgrind generously rewards people who write representative test code.
This is all just my personal opinion.
Have a look at memory validator. I don't know if it supports 64 bit applications, but it has a great list of features and is the only decent memory validation tool I've ever seen on Windows.
Seeing as your targeting Windows, why not use C++/CLI and get the very same memory management as C# ?
It is not immoral to create the human species - with or without ceremony, Samuel Clemens.
- Avoid pointer arithmetic.
But why use C++ if you're not using pointer arithmetic? If you're not doing that, go to Java. I know this sounds silly, and it opens religious issues, but (aside from legacy apps, like this one) why would I want to use C++ if I'm not doing that?
I use C because it's small, fast, convenient and portable. I can code something tiny quickly.
I use Java because it's an object oriented language that helps with complex app coding.
I use C++ because I want some of that Java stuff, but I want it to bind to my memory model more realistically. ie. Pointer arithmetic.
Poor you. You raised on C++ and naturally feel no love. You get your memory screwed up. You have no good languages to program in. Worst of all, you're targeting XP.
Poor you.
We have seen some leaks in our applications. These were tracked down to STL internally leaking. They weren't generally very large and therefore we continue to live with them.
Just FYI - what you most likely observed was not a memory leak. Various STL implementations will use their own pooled memory allocators where they won't release memory back to the free store so it can be subsequently reused for without the overhead of allocating the memory again.
However, I've also run across a pathological case where perfectly valid C++/STL code did cause the gcc (something like v3.3 or v3.2) STL implementation to never reuse the memory it had allocated so eventually the process would drive the machine into swap until the OOM killer kicked in. I was able to confirm this be running the application for as long as I wanted to with the GLIBCXX_FORCE_NEW/GLIBCPP_FORCE_NEW environment variables defined.
----------------------------------- My Other Sig Is Hilarious -----------------------------------
I have had pretty results from Memory Validator from Software Verify. (m l ) It'll slow down your app but I think it does a better job keeping things close to real time then purify.
http://www.softwareverify.com/cpp/memory/index.ht
-----------
Fight Entropy!!! Fight Entropy!!! Figth Etnropy! !
iFgth Etnrop!y ! giFth tErno!py ! giFt htrEno!p y! --- Well maybe
not...
Use a testing framework like Parasoft's CPP stuff http://www.parasoft.com/jsp/products/home.jsp?prod uct=CppTest
Australian running a company that does C# / C++ / Java / SQL / Python / Mathematica
Use valgrind.
SmartHeap
> shared_ptr is not thread safe
m
Not true, as I understand it (but please correct me if I'm wrong). See the section titled "thread safety" on this page: http://www.boost.org/libs/smart_ptr/shared_ptr.ht
"shared_ptr objects offer the same level of thread safety as built-in types."
It goes on to give various examples. This means that you only need to add your own locking in cases where you would also need to lock a built-in pointer type. Well, hmm.... a question that just occurs to me is whether a T* is small enough to be atomic on common architectures, while a shared_ptr is not atomic. It may be that code that relies on writes to pointers being atomic (which is true on most machines, but would not be on e.g. a 32-bit processor with a 64-bit address space) would not work with shared_ptr. But the rest of the time, I believe shared_ptr really is safe.
The following CRT/Win32 functions can come in handy when diagnosing memory leaks and heap corruption.
Use _CrtSetDbgFlag to get the memory manager to test the heap periodically during use among other things. Or use _CrtCheckMemory to do it strategically.
Using _CrtMemCheckpoint and _CrtMemDumpAllObjectsSince can check to see if any heap blocks have been left on the heap in a range of code. You can use _CrtSetBreakAlloc to break on the allocation to locate the point where a widowed block was allocated.
You can also use GetProcessHeaps and HeapValidate to check heap integrity. In particular, it can be used to check all heaps in the process.
Yes, absolutely true that error messages are useless. The blame for this can be shared between the language, the compiler, and the library. The worst I have encountered is Boost.Spirit where it would churn out messages hundreds of lines long with no clue whatever what the real problem was; you might as well just make random changes until the message goes away. In the end this makes you code in a very defensive style, not deviating far from the examples that you've copied-and-pasted from the documentation.
I would say that this is a serious enough problem that we ought to stop and fix it before developing yet-more-complex libraries. One attempt to fix it at the language level is the introduction of 'concepts' in the next version of C++, which allows template classes to specify properties that their parameters must have - and which presumably allows more sane error messages when the properties do not hold. An attempt to fix it at the library level can be seen in the message that you cite: "property map not found". Yes, it's embedded in a load of stuff from the compiler, but maybe that's the best it can do.
I'd be very interested to know whether any of the other compilers give more comprehensible error messages than g++.
Rational Purify is probably the best tool on the market.
I understand the need for good C/C++ tools to check for memory leaks, performance bottlenecks, profiling, et cetra, but the idea of using garbage collection/memory management libraries in these languages seems a tad silly.
If you can afford to make your app compile with mingw's gcc (or cygwin's gcc but mingw should be easier), then you may be able to use mudflap, which is a memory debugging system integrated into gcc. You just need to pass -fmudflap, and gcc will instrument the program at compilation time.
i ze-Options.html#Optimize-Optionsg ing (maybe slightly outdated)
One thing is that it used not to properly instrument some really basic C++ operation in gcc 4.1 (I don't remember what exactly, something like copy construction of an object containing pointers) and was reporting spurious leaks because of that. It may have been fixed in 4.2.
Search for "mudflap" in the following page: http://gcc.gnu.org/onlinedocs/gcc-4.2.0/gcc/Optim
As well as http://gcc.gnu.org/wiki/Mudflap%20Pointer%20Debug
At the risk of being beaten into oblivion I would like to submit that often times when garbage collection causes a detectable performance penalty is due to poorly written Java code. The code generally looks more like C++ than it does Java. I am not saying that garbage collection is cost free, but can be effectively marginalized, even in large scale and high performance applications.
Valgrind is fantastic for something like this.
Following MS tools are absolutely free:i nstallx86.mspx. Following article explains how to use it http://support.microsoft.com/kb/268343/en-us.d ebugstart.mspxo ws/appcompatibility/appverifier.mspx.
- Use umdh that is a part of the Debugging tools for windows to track memory leaks. http://www.microsoft.com/whdc/devtools/debugging/
- Whatever you do make sure you have proper symbols. Following article explains how to get symbols from MS symbol server. http://www.microsoft.com/whdc/devtools/debugging/
- Use paged heap to track all other issue like memory overruns, double free and all other sorts of heap corruption. http://www.microsoft.com/technet/prodtechnol/wind
- Please note that if you run you application with app verifier checks on you need to run it under a debugger. I would strongly suggest windbg or cdb instead of visual studio because it has extensions that would greatly help you to track down the issue ("!analyze -v" "!avrf" "!heap -p -a "). For more details see windbg help. If your application is a service then you might consider running your machine under kd, which would trap all unhandled exceptions and application verifier reports.
- Following link has a very good windbg tutorial http://www.codeproject.com/debug/windbg_part1.asp
That is all you need to debug any kind of heap issues.
Application Verifier comes in x86, ia64 and amd64 flavors.
This tool allows you to enable PageHeap for your process, which is heap corruption detection built into the OS heap implementation. Upon freeing a block of memory, PageHeap will break into your debugger spewing tracing that a block has been corrupted. It can also provide the call stack when the block was allocated. Newer heap validation features are available in progressively more recent OS releases.
Unfortunately, I don't think it'll help you in the Windows world. But for *nix, I've found dmalloc to be quite simple to use but very effective.
http://dmalloc.com/releases/
Following MS tools are absolutely free:i nstallx86.mspx. Following article explains how to use it http://support.microsoft.com/kb/268343/en-us.d ebugstart.mspxo ws/appcompatibility/appverifier.mspx.
- Use umdh that is a part of the Debugging tools for windows to track memory leaks. http://www.microsoft.com/whdc/devtools/debugging/
- Whatever you do make sure you have proper symbols. Following article explains how to get symbols from MS symbol server. http://www.microsoft.com/whdc/devtools/debugging/
- Use paged heap to track all other issue like memory overruns, double free and all other sorts of heap corruption. http://www.microsoft.com/technet/prodtechnol/wind
- Please note that if you run you application with app verifier checks on you need to run it under a debugger. I would strongly suggest windbg or cdb instead of visual studio because it has extensions that would greatly help you to track down the issue ("!analyze -v" "!avrf" "!heap -p -a "). For more details see windbg help. If your application is a service then you might consider running your machine under kd, which would trap all unhandled exceptions and application verifier reports.
- Following link has a very good windbg tutorial http://www.codeproject.com/debug/windbg_part1.asp
That is all you need to debug any kind of heap issues.
deleaker, from www.deleaker.com
This is one of the best Windows C++ specific leak detectors out there. It is VERY simply to use, and integrates into Visual Studio in a very simple manner.
-- I'm the root of all that's evil, but you can call me cookie..
GLIBC allows you to create hooks for the standard mem functions (malloc/realloc/free). Remember that g++ still calls these under new/delete so it works for C++ also.
One of our guys coded up a simple shared lib that can be loaded with LD_PRELOAD that sets simple hooks of printing memory locations for new/realloc/delete. He then wrote a perl script that kept track of these things and spit out anything that was malloc'ed and not realloc'ed or free'd.
I can't post it, because technically it's not my code it's my company's. But his shared lib code is just 300 lines long, and shouldn't be hard to duplicate. The perl log filter is even more straighforward. Each malloc gets saved. Each free removes the malloc. Each realloc removes the old malloc and adds a new one. Anything left over is a leak.
Override __malloc_initialize_hook with a pointer to your init_function. In your init_function, save the old functions at __malloc_hook __free_hook __memalign_hook and __realloc_hook and substitute your own. Now write your replacement functions, in it, do your logging and temporarily replce the old hooks and call the original functions, replace with your hook on the way out to get the next call. All of the hooks should be wrapped in a mutex to help re-entrancy problems.
It's not a full memory detector, just does leaks, but it's non-intrusive, requires no recompiles, and is the best way we have to leak detect our huge server long running code.
So you think you're not allocating parts of that pool internally? Are you not marking pieces of the pool to be in use? Personally, much greater problem is buffer overrun than memory leaks. Memory leaks can be tracked, and fixed eventually. But how do you know where did a program do a buffer overrun? It's easy to see "aff, that was caused by a buffer overrun", but where?
Microsoft has one of the best debuggers integrated into Visual Studio. It can find memory leaks and bugs in your code very easily and it just requires a couple of lines of code. you just surround all your code with the debugging function calls, and it will compare the before and after snapshots of your memory and tell you if you have a leak or not. Behold:
#include <crtdbg.h>
int main()
{
#ifdef _DEBUG
_CrtSetDbgFlag(_CRTDBG_LEAK_CHECK_DF | _CRTDBG_ALLOC_MEM_DF | _CRTDBG_CHECK_ALWAYS_DF);
#endif
int *p = new int;
#ifdef _DEBUG
if (_CrtDumpMemoryLeaks())
{
printf("You have a Memory Leak!");
}
#endif
}
A morning without coffee is like something without something else.
I can attest to this. A few months back I was doing cross development for FreeBSD/Linux and maintaining a development version for Windows.
My code worked fine in Windows - mostly, but still a bug remained that I suspected was in the renderer but couldn't prove it.
The code ran fine on FreeBSD without any problems but I couldn't run it for more that a few minutes on Linux without it crashing hard but at random times.
I ran my Windows code through BoundsChecker (which I've used for years and have found to be very effective). Came up fairly clean, I fixed a few things but I could not find the bug.
I wasn't familiar with any Linux debug tools but eventually tried Valgrind. It found the problem after the first run, and BoundsChecker didn't even give me a peep. I found I was stepping just outside a graphics buffer at certain times and writing in memory I didn't own. It was enlightening the bug was in the same code for FreeBSD/Linux/Windows but displayed completely different behavior.
Anyway, I have respect for Valgrind. Now, if it was only available for FreeBSD.
and use Java
Partial implementation is available from Boost.
A commercial full implementation can be purchased from DinkumWare.
Probably not what you were intending.
I don't mind if an implementation of STL has some hidden/private/singleton allocation pools behind the scenes to speed things up. What I find really friggin' annoying is that they never track those allocations and offer any form of "reset" function that you can call before exiting your app, so that the simplest global malloc-counting methods can audit for leaks. You shouldn't need an "SGI-STL-aware-and-compatible leak detector," you should only need a malloc leak detector.
[
People screw up when they use this. Oh, maybe you're a genius who gets it right (just like you're a better than average driver) but have some pity on the other developers.
You get leaks that aren't leaks. The memory gets freed when the app exits. That's useless, because the OS will free it then. What failed to happen was freeing while the app was busy doing stuff. It's not OK to have objects just hanging around, all tied to each other, until the app exits.
You get bad interactions with regular pointers. Regular stuff gets leaked because you relied on the dumb pointers. Regular stuff gets double freed. Note: the regular stuff need not be pointers. It could be database connections, file handles, etc.
This is what I posted: and after watching the team flail and spend literally almost a man-year trying to find one memory bug, I finally had enough of "advice giving" being ignored Now, what part of '"advice giving" being ignored' is too complex for you to comprehend?
I had some problems with memory invasion and leaking in a game that I'm working. At that time I coded MemDebug.
Its a pretty much simple memory debugger, detect leaks and comes with a pointer wrapper that detects memory invasion.
You can get it here
http://sourceforge.net/projects/memdebug
I'll be the first to admit that static (as opposed to dynamic} array allocation is both quicker and less error-prone. Provided, as you already mentioned, that your application allows it. Your "pools" are what Fortran calls "shared common blocks".
The problem that I see is that whenever you are re-using part of your "pools", you are in fact doing something akin to freeing and allocating memory as far as the statespace model of your software is concerned. Only you do it "internally", within the memory space allocated to you by the operating system.
Of course you will avoid your programs crashing due to the operating system shooting them down for accessing memory that was not allocated to them, or from attempting to return memory to the operating system that the program doesn't "own" in the first place.
This is essentially what memory-allocation and memory-freeing crashes amount to, but as far as I can see from the description you give, you can still commit grave "memory allocation" errors in that the variable you use at some place contains the data of some other variable. You just make sure that it happens where no operating system can possibly take offense. That doesn't necessarily mean that your code doesn't contain allocation/deallocation errors.
If any such errors are present, your code will stay up and silently return the wrong answers. Is that an improvement? You be the judge.
Multi-platform, memory debugger with coverage analysis. It is NOT free and it works very well.
There are several types of memory debuggers:
source code,
object code,
linked library wrappers.
Insure++ is Source Code instramentation. It wraps code with its own trap routines and finds a LOT of errors.
Purify (I used it a bit ago, was not on Linux at the time) is a object code instramentation and it takes the object code and wraps it. It finds some stuff.
Electric Fence is a FREE library that basically wraps malloc and traps some basic systems.
Insure++ is the best that I have found. You can try it for a time if you wish at
www.parasoft.com
I can program myself out of a Hello World Contest!!
begin w/ "MZ", your Klingon must be rusty ad that would be an invalid EXEcutable.
"Happy families are all alike; every unhappy family is unhappy in its own way." -- Anna Karenina by Leo Tolstoy
Just released today, C++Builder 2007 fully support Windows Vista and includes the tightly bound CodeGuard memory checker. CodeGuard will monitor overwrites, pointer usage, etc.
C++Builder 2007 is the only C++ development environment that supports Windows Vista development (Visual C++ Orcas is still beta, might ship this year). But it supports Vista, XP, and 2000 all simultaneously. It's also the only RAD native compiling C++ development tool (Visual C++ is still MFC based... still uses resource files... not visual tools, components, or two-way designers).
Check it out. http://www.codegear.com/products/cppbuilder
The Conservative Garbage Collector http://www.hpl.hp.com/personal/Hans_Boehm/gc/ will make most memory issues go away. It is robust and is used in GCC for example.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
That example happened something like 5 or 6 years ago. Maybe try it on MSVC++ with CString?
Leaks usually don't result in crashes unless they are severe or very long-lived. A more robust solution might be multiple, redundant, somewhat shorter-lived processes.
Lastly, I strongly dislike anything from Rational. I find them overpriced unreliable bloatware (YMMV). Purify used to be good some time ago, but those days are long gone.
IBM/Rational has bought up quite a bit of software over the years. Not all "Rational" software is equal. It is true that Purify is not going to help with an application such as yours that uses a memory pool. But for really big applications, with many developers (not all of whom are equal), Purify is just wonderful. It can, and will, track down leaks and other serious problems. When used with PureCoverage, it can give you confidence that your application doesn't, and won't ever, leak memory or crash
I find valgrind invaluable; a lot of people swear by Purify instead. They sound like similar tools. I've never used the later but would love a comparison by somebody who's used both.
What (think) I know :- valgrind is free, purify is proprietary
- purify runs on some platforms that valgrind doesn't (BTW, I use valgrind on FreeBSD too)
- purify requires to link your program with its library, valgrind runs unmodified binaries on a virtualized processor
- valgrind has additional modules, like a profiler or a thread race condition detector
What I'd love to know :Since JDK 1.4.1.something Java has shipped with multiple other memory management options specifically designed to minimize problems caused by stop-the-world garbage collection. Tell your colleagues to upgrade their JDK then activate the "CMS" garbage collector via command line switches.
is competition good, or is duplication of effort bad?
Here's a writeup on a tool someone wrote for this very purpose:
r ing_your_pcs_memory_usage_.php?print=1
http://www.gamasutra.com/view/feature/1430/monito
-S
Yes, it was exactly what I intended. I was implying the poster was an asshat and wanted his meager intellect to be represented by his poor grammar and limited vocabulary. Surely I couldn't call him stupid and let him sound more than a dolt, could I?
One man year - my ass. You actually fell for the bait of an AC's lie on Slashdot. You should feel ashamed of yourselves.
Give LeakTracer a try. I uses LD_PRELOAD to avoid needing a recompile, and hooks into the new and delete operators to provide a concise leak report.
Microsoft's Application Verifier is a great tool for diagnosing lots of problems (on Windows mind you) including doing verification on your not overflowing bounds. Best part is you don't even have to change your code or recompile, it does completely run-time diagnostics. http://www.microsoft.com/technet/prodtechnol/windo ws/appcompatibility/appverifier.mspx
For the memory validation, every time you allocate memory it'll put it in it's own page with guard pages around it such that if you write past your buffers even by a single byte you'll immmediately AV inside the offending code (rather than crashing later on in some unrelated code or inside the heap manager).
Dead simple to use, and has saved me countless hours. A similar tool is the Driver Verifier, which is actually included with most releases of Windows under C:\Windows\System32\verifier.exe. It does essentially the same kinds of things but for drivers, so if you've been getting lots of blue screens with stuff like BAD_POOL_HEADER etc, run verifier, wait a few minutes for the driver to do it's dirty deed, and take a look at the crash dump to see which driver is to blame.
AppVerifier will do memory stuff as well as a few other things.
6 3.aspx
http://technet.microsoft.com/en-us/library/bb4570
I've used Purify, Boundschecker, Efence, various hand-rolled stuff and by far the best tool for checking memory access and for leaks is Valgrind. Because of the way it works, it finds all sorts of tricky memory problems at the point that they happen without *any* instrumentation of your program before hand.
The downside is that it only works on linux (and x86 at that), but it kicks ass. It's one of the unsung heros of open source.
Try if you can.
We also found BoundsChecker to be too slow to be very useful. AQTime (AutomatedQA), on the other hand, is relatively cheap and fast. There's a full featured (time limited) trial version that you can download.
There are many many references all over the internet for why using auto_ptr (which is boost's shared_ptr) is philospohically a bad idea.
Ultimately it comes down to being careful when you write your code that you clean up. There are a few things you can do to help:
1) Use containers wherever possible rather than create clouds of dynamically created objects.
2) Don't expect the user of your function to know to clean up after you, even if the user of your function is you. (e.g. pass in an empty container as a parameter that the function fills, rather than dynamically create/populate memory in the function then return it). This approach also helps reduce the number of calls to malloc()/new() as the caller can statically create an empty container.
How hard is it for these "equally skilled programmers" to port the J2ME runtime environment to a platform that has never seen Java before? How hard is it for them to do so and make a game in Java vs. to make a game in C++?
If you haven't guessed, the platform in question is Nintendo DS.
Why is everybody so obsessed with doing things manuallly? Why do you insist on creating your own private memory allocation instead of addressing the root problem? Doing allocation yourself just makes the whole thing more complicated, and makes your code even more prone to bugs.
Screw memory allocation. Ban pointers. Just allocate everything on the stack, forget the heap. Pass by value instead of reference. If you are working within a single thread, use references freely, just never pointers. If you are multi-threaded, use established and well-tested message passing systems instead of pointers to objects on the heap (or, God forbid, the stack!).
It is entirely possible to write low-memory and highly-performant systems by eschewing pointer use. In fact, it allows the compiler much greater flexibility in optimization because all the zillions of problems with aliasing vanish, making ambiguity almost non-existent.
Why does everybody assume without question that pointers are the only way of writing quality C++ code?
I don't know if you've used or seen fortify but I found it a wonderful tool to not warn and resolve memory issues in C++.
It is very fast, compiles switches into nothing and incredibly helpful.
-- Snip
Fortify is a powerful C++ debugging aid. Detects and pinpoints memory related bugs. It supports alloc/calloc/realloc/strdup/free and new/delete. It traps memory leaks, writes beyond and before memory blocks, writes to freed memory, free twice, freeing memory never allocated, and removes all randomness from reading uninitialized or free memory.
Not just where its defined, where its used! Try grepping for a "+" in any significant program!
Not directly related to your problem, but just FYI...
http://www.digitalmars.com/d/index.html - D language, free, memory-managed and compiles to native code. Perhaps worth checking out.
All they would need is a more robust run-time type information system. Something more along the lines of reflection in .Net would be amazing (as long as it's optional for the core language). I would love something like that to help automate serialisation, garbage collection, inter-language interfaces, etc. That, and a fancy multipass compiler to get rid of headers and the order of definition nonsense are on my wishlist.
I think for either to succeed though, you'd need to build them as a frontend for existing C++ compilers, instead of waiting for support.
I was a developer for a "top 5" web browser for 6 years. I often worked on porting to small platforms and I also focused on how to improve performance on these platforms.
Before I go into a rant, I must first rant about how much I hate Java. I feel that it was a great proof of concept and that they should have taken what they learned and went back and did it better. Java is a lot of great ideas implemented poorly due to lack of experience. I think they should have spent more time with the SmallTalk guys who actually almost had it right to begin with. Hell, evolving SmallTalk would have been far more intelligent than turning C++ into SmallTalk.
Ok... here's the thing. I tried a lot of competing products. I tried memory checkers, memory allocators (and SmartHeap is the shit!), I tried memory profilers, hand instrumented memory logging, etc... what I've learned are a few things...
Garbage collection (even reference counting) can improve performance greatly, but it had little or no impact on fragmentation. A system that I slapped together as a malloc/free new/delete override proved quite successful at drasticly improving browsing performance. What I did was that instead of deleting memory, I queued deletes and when the pool needed to be grown, I would process the deletes or I would process the deletes during idle cycles.
This just made the program seem faster during runtime... obviously, the added overhead just made it slower.
To explain why a web browser is one of the most rigorous tests of a memory environment... just think of the hundreds to thousands of DOM nodes/elements/etc..., script objects, images, etc... there are in sinlge page. Add that each element is typically represented by a single allocated object. Consider that images can decode to 100 megs in size (yes, it happens), most often closer to 2-3 megs for background images.
A web browser can't use a memory system optimized for specific object sizes.
Due to the dynamic nature of the objects, object reusability is not really an option
Scripts can grow or shrink memory usage thousands of times per second
Browsers typically contain 3rd party code from plugins which need to interact with the browser
I can go on and on... a web browser is possibly the worst memory management nightmare on the planet. Often I worked with customers that were developing their own operating system. They used to tell me that my web browser must suck because making it stable on their system was a pain in the ass and often required them to either change or rewrite their entire memory management system to get good performance on embedded devices. Then I'd explain to them that up until now, their memory manager has been having a friendly snow-ball fight with a penguin, now it's running for it's life from an avalanch caused by a mean Yeti.
So here's the deal, GC really didn't pay off for us... helped a little, but I have to say that once we started using reference counting and simple GC, the quality of the code got really poor. Just look at Symbian for an example of a product that suffers from using great memory management system that increases coding complexity 10 fold. It makes it so that you spend all your time coding for the memory manager and you run out of time to make the program itself work.
Now on the other hand, I played with a few Java web browsers and learned something important. Java is made for phones. If it has no other purpose (and I'm convinced it doesn't), it's for embedded devices. Because of relocatability, fragmentation doesn't occur (in a good VM) and applications run much better. The GC + Relocation system is REALLY REALLY REALLY good, if I were to start writing a new web browser today, I'd find a good alternative to Java and get moving on it.
Oh... last thing about auto-pointers, they're a blessing and a curse. For the most part, I find the best solution to be to use a proper system library like Qt instead of boost or STL. Qt seems to actu
Avoid heap allocation if at all possible.
Not an end all cure, and won't help in the cases where you absolutely NEED dynamic allocation... but as a practice you will find this to greatly decrease your memory woes.
When a junior dev shows me C++ code using pointers I tell him to rethink his implementation so that it doesn't use pointers.
In the hands of a master pointers an dynamic memory allocation are great.... but judging by the way everything seems to leak and leak copiously there are very few "masters" out there.
I am very small, utmostly microscopic.
The tool created at Google and released as open-source: http://google-perftools.googlecode.com/svn/trunk/d oc/heap_checker.html
Try a static analysis tool like Coverity... http://scan.coverity.com/
Sounds like what you need is actually an obstack. These come with glibc.
We at slashdot are scientists, specialists and kernel hackers. Your FUD will be found out.
True... but, the thing is, there are a lot of systems out there where the code is ugly, but, mostly works. Sure, it is less expensive to design a system right the first time around, but what often happens is that the system is well designed, but the company has already spent a giant chunk of money on all the testing needed to overcome the bad design. So yeah, you have a bad system, but one that does work. In their eyes, a rewrite is to incur the risk of going back to the beginning of a project. For everyone that rolls those dice and gets a clean, well designed system, there's another couple that get more delays brought on by a design as bad as the original one was to begin with, just in different ways.
While it is true that 90% of the cost of a program is in "maintenance", a lot of time, maintenance is a much smaller team than the original development team was. You can quickly go from a dozen developers down to 2, so, from a budget perspective, that doesn't seem so bad. Maintenance too, is often an addition of features and screens and tables. You aren't rewriting or refactoring existing code as much as you are tweaking it to allow for the big new thing to work. In the end, you can put your "good design" into your new stuff, and leave the old stuff alone. Then, the next guy comes along and puts his "good design" in. Quite often you wind up with a project that has a number of different personalities to it.
This is my sig.
Replace *alloc with a garbage collection library and forget manual memory management. Arguing manual memory management versus garbage collection is like arguing 2^n versus 128*n Yes in a small area and for a period of time you can manage memory collection manually, the problem comes after your application has matured and you're a year down the road(n = 11) Now it's more complex to manage from a human standpoint and often slower in terms of CPU and memory footprint to manage reference counts. First rule of optimization, find a better algorithm.
Maybe it's time to switch over to one of these newfangled memory-managed languages? It's a tiny bit like saying, "My Pinto is a huge death trap! Anyone know of some after-market air-bag kits for it?"
Patrick Doyle
I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
We pre-allocate pools of objects at startup and then re-use them. ...
I knew when I read that I was reading an intelligent post. Your application probably even runs faster too, not having to do all the new/delete or malloc/free or that real slow system choking GC stuff.
I first did this with embedded systems that had to be up for a very long time and memory leaks were just not an option. Pre-allocation stopped it. Always use this method of design when practical. When not, I am careful with memory management and create a special object class to track it.
But looking at the posts in this thread, I guess there is a shortage of seasoned and skilled C/C++ programmers with regards to memory management. I have yet to see a 100% capable tool for this issue. Either it misses it, takes too long, or more often gives so many false positives it is useless.