Slashdot Mirror
C# Memory Leak Torpedoed Princeton's DARPA Chances
nil0lab writes "In a case of 20/20 hindsight, Princeton DARPA Grand Challenge team member Bryan Cattle
reflects on how their code failed to forget obstacles it had passed. It was written in Microsoft's C#, which isn't supposed to let you have memory leaks. 'We kept noticing that the computer would begin to bog down after extended periods of driving. This problem was pernicious because it only showed up after 40 minutes to an hour of driving around and collecting obstacles. The computer performance would just gradually slow down until the car just simply stopped responding, usually with the gas pedal down, and would just drive off into the bush until we pulled the plug. We looked through the code on paper, literally line by line, and just couldn't for the life of us imagine what the problem was.'"
The linked "article" is just a "sponsored review" for a C# profiler...
Just because a language is garbage collected doesn't mean you can't "leak" memory (in the more standard definition of "waste memory over time"), it only means you can't completely lose track of references to objects (which is often used as a more technical definition of "leak"). It is quite common for people coding in such languages to accidentally generate live object structures that are mostly made up of garbage that they should have released their references to. Put another way: these people's program was legitimately claiming memory and never releasing it due to their limited understanding of how event handlers work.
This is a programming error, plain and simple. From TFA:
Though we thought we had cleared all references to old entries in the list, because the objects were still registered as subscribers to an event, they were never getting deleted.
So references were held to the objects in two places - the list of encountered obstacles, and the list of event subscribers. They were being removed from the list of encountered obstacles, but not being unsubscribed from the event.
How do you think event subscription works? Something has to hold a reference to the objects that are subscribed to the event! That thing is going to hold a reference until you unsubscribe the object - it neither knows nor cares about any other list of references you may be maintaining separately, how could it?
This is a coding error. A subtle, non-obvious one perhaps, but a bug nevertheless. It is not an error in the CLR, and in fact the article never paints it as such. That particular bit of spin is wholly down to the submitter.
It's official. Most of you are morons.
As is the case with Microsoft's GC, Java's won't delete things that are still being referenced by other things, because it quite reasonably assumes that an object which is referenced by another object that hasn't itself been marked for collection isn't garbage.
The main problem with garbage collectors (I like GCs, so this isn't a diatribe against them) is that far too many mediocre programmers assume they have a magical ability to know precisely what they want their code to do. The reality of course is that they use algorithms to decide what should be collected, when it should be collected, and how it should be collected, and those who are unfamiliar with the particular strategies that their GC uses can therefore not only write code with more than a few memory leaks, but also code that results in the GC being used so inefficiently that it does vastly more work than would be necessary if the same functionality was implemented in a slightly different way.
There are plenty of articles about Java memory leaks that can be found by Googling "java memory leaks". Googling "java GC tuning" will produce some useful links to articles containing tips on ensuring that it's not used inefficiently.
I'm not going to change your sheets again, Mr. Hastings.
Well the Event Subscribed 'problem' is well known and makes sense if you think about it. I mean subscribing to an Event means placing a pointer to a delegate of a method in a event subscriber list.. when someone raises that event then each delegate in the list is invoked... so basically it is an implicit reference and hence can prevent the it from being marked for garbage collection.
;) :)
However, i had another memory 'leak' problem where the Garbage Collector simply didn't collect in time which caused my application to use more and more memory until it reached the system limit and crashed... i found that simply calling
GC.Collect();
GC.GetTotalMemory(true);// (the true 'forces' collection
once would fix this problem... i though i needed to call it every minute or so... but when calling just once it did SOMETHING that prevented this problem from occurring again.. no idea exactly what.. but it works
They can't have had anyone on the team with experience of coding for Swing in Java then - you get these all the time, sometimes hanging tens of megabytes of unwanted GUI objects off a single listener registration, and learn how to spot and fix them.
They also didn't pick a very good hack because it didn't leave the car in a safe state when the software broke.
Lack of practical experience I'd say. A few more events like that and they'll make decent devs one day.
It's not the garbage collector's fault. If an object is still in use, it can't be collected and destroyed. Managed memory only prevents the kind of memory leak where the programmer "loses" all references to the memory and thus never frees it. It also prevents the kind of bug where memory which is still in use is freed. Programs usually crash when that happens (either the OS terminates them due to a memory protection violation or they overwrite their own data and crash later on). That is also what would likely have happened in this case if it weren't for managed memory, because obviously the programmers mistakenly thought that these objects were no longer in use, so they would have freed them when they were still handling events.
just to be clear, THE BUG WAS NOT IN THE RUNTIME, not by any stretch.
there are very clear constructs in place in the language/runtime to allow any object to unregister itself from event registrations it initiated.
this was VERY MUCH a bug in the end-user software, not the runtime (i've written code almost IDENTICALLY to this and blew lots of time having made this same mistake).
the only thing the runtime could do to protect the idiot developer (myself included) is automagically make all event references WEAK references, but that has plenty of undesirable side-effects too... in clr, you can do this yourself if you're so inclined... (just like in a JVM)
cheers.
Peter
these don't have any problems with circular reference structures - if it can't be reached from a root and marked, it'll get collected.
still just a blunder, as you say.
this article should be binned - misleading title and nothing but a puff-piece for a profiler. i much prefer YourKit, incidentally:-)
Actually, C# doesn't reference count at all, it 'Reference Traces' :)
Please, let me explain; it's quite sad how often people don't get this ...
.Net has its block of managed memory, called the Managed heap. It's separated into 3 'generations'. This heap has 2 areas, free space and reserved space, from top to bottom.
When you allocate and object to the heap, by using the new command (object o = new object();) there is a set of rules? that have to be enforced:
The GC manages Reference tracing, and this doesn't occur when the object goes out of scope, it actually happens when the Heap is full and you attempt to allocate a new object.
In something called 'the sweep', the GC goes through each object in the heap to see if it's reachable. To do this it starts with so-called 'roots'. It then traces to see which objects are referenced by these roots.
A root identifys a storage location, which referes to objects on the managed heap, or objects that are set to null. For example, all of an applications global and static objects are considered to be it's roots. (hence the reason that all C# apps have a static void main).
When the sweep starts, it assumes that all objects are garbage. So for each root object, it builds up a graph of the objects that root references, and marks them as being live.
However, if it finds an object that's already in the graph, it stops traversing that path. This is two (massively) increase performance by not scanning the same object twice, and more importantly, it stops you getting into an infinite loop by scanning a circular list.
The pinch is, it prevent the circumstance that you mentioned! :)
Because the strong reference to a linked circular list is gone, the circular list isn't attached to a root object, so it gets disposed. If you don't want it to get dropped, unless it theres a memory shortage, the C# GC also supports something called Weak References, but I'm not going to go into those here as it's headhurting
So once all the roots have been checked and we've got a nice graph of all the objects that are referenced by the live parts of the application somehow, the second stage of GC happens.
Any objects that haven't been touched by the walk are of course still marked as Garbage. The GC now walks up the heap linearly, looking for contigious groups of garbage which are now considered to be free space. The GC looks for the next live object and moves it to the start of this free space with a good old memcpy :)
This ofcourse invalidates all the root pointers, so the GC then updates the points in the root objects.
:)
So now, we've got rid of all the garbage and our heap is pleasantly compacted; Take that Heap Fragmentation, Kerpow!!
But, that's not all she wrote of course
Now we're free'd and compacted, the 'nextObjPtr' is moved to the top of the heap. At this point the new object creation that triggered the collection is performed and the new object appears at the top of the heap.
This is a dramatic over-simplification and I've not attempted to explain finalization or weak references, but it's still good to know this stuff, it helps us as .Net programmers to consider how to write our code properly :)
The other thing I've not explained is how the Generations work:
When we remember we are all mad, the mysteries disappear and life stands explained.
Free Software games list and commentary
There's also the issue where you need to explicitly remove your event listeners when you no longer need the object. The listener keeps a reference to the object (via the interface) so even if it goes out of scope or what-have-you, YOU may think you don't have any references to the object but it implicitly does, through the listener you handed to the system. So... if you're using event listeners, make sure you explicitly remove them in your object's destructure... or else you'll end up with a memory 'leak'.
Its quite clear in the article that they forgot to unregister their "deleted" objects from events. Since they were still registered, they weren't garbage collected. And rightly so. This was THEIR programming mistake, and has nothing to do with a GC bug in C# or any such thing. Fuck slashdot is pissing me off these days... as soon as they see a story that could be spun as "Microsoft screwed up!", they publish it withouth any fact-checking (or even reading the goddamn article!).
I wonder is MS could sue Slashdot for slander?
Jeremy
"NO! It is not a good thing, if a program slowly leaks memory then it just makes it harder to find the bug. If you have to reboot the app every week because it has a little leak, no-one's going to be bothered (except the users who see it slowly getting slower). If it has to be restarted daily then you're going to be looking to fix the bug."
Actually the good companies do debug the slow memory leaks, and the bad ones don't debug the slow ones. Besides, any memory leak in a Java program is possible in a C app, so you are eliminating a class of leaks, not replacing them with harder to find leaks. Thus your entire argument is moot. Furthermore, where are you getting the idea that Java memory leaks are going to be slow while C memory leaks are going to be fast? I've seen slow C memory leaks and fast Java ones. I can think of nothing regarding the nature of garbage collection that would effect the speed of the leak.
"I have a good analogy - Firefox. I use FF a lot, I like it, but it does tend to increase its memory usage over time, and has been rightly criticised for it. Now, I'm sure the 'bug' is an aspect of its design and not a programming bug (and I don't want to start a FF memory discussion - I'm only using it as a real-world example) but just imagine if *every* program was like FF - slowly using more and more RAM over time until you restarted it."
First, thats not an analogy, thats an example. Second, Firefox is not an application, at least not in this day and age. Today its a platform for web applications which are just vulnerable to memory leaks as any other. If that cool new javascript app that is running on the page you are loading leaks memory, there really isn't a whole lot Firefox can do.
Third, I fail to see your point. Do memory leaks suck? Of course. Is it best to get rid of them? Of course. Will garbage collection get rid of memory leaks? Of course not. Will they make the problem any worse? No, any code that leaks in Java will also leak in C. Will it make it better? Of course, there are types of leaks which simply are not possible in Java. Those will be eliminated resulting in fewer leaks (though it is of course impossible to eliminate the completely).
Mathematics is made of 50 percent formulas, 50 percent proofs, and 50 percent imagination.