What Debugger Is Best For Multithreaded Apps?

pollyp asks: "I'm on a team that's working on a multi-threaded (pthreads) C++ application running on GNU/Debian 2.2. One of the challenges of this project is trying to debug the app, or making sense of core files; the feeling is that gdb is just too difficult to work with in a heavily threaded app. We're looking for better tools, and if that means changing languages or going to another UN*X, we're ready to do it. So what is the best, most mature OS/language/tool UN*X environment for doing multi-threaded application development on Intel?"

"One part of the team wants to stay on GNU/Linux but use Java. Java's performance is becoming less and less of an issue, and the GNU/Linux development tools are reasonably attractive (e.g., Visual Age's debugger). The other part of the team wants to stick with C++, but move over to Solaris. But none of us have any practical experience with Solaris, nor do any of us have a sense of how fully baked their Intel development environment is."

Comments

The best tool

[Bongo]

So what is the best ... tool ... for doing ... application development?

16 cups of this.

If you can afford it, move to Java

[Kieckerjan]

Given the fact that 90% of all C / C++ bugs are caused by memory related issues (buffer overruns and memleaks being the most notorious ones) it would probably be a wise choice to move to Java, if you don't mind the (minor) performance hit. My own experience is that when writing Java code you spend a lot less time in the debugger, because (a) the garbage collector prevents memleaks (b) buffer overflows generate very clear error messages instead of a stupid core dump. So the choice of debugger automatically becomes much less of a issue.

As to performance: if you're not writing a nuclear explosion simulator, Java's performance is probably adequate. (Think StarOffice here.)
--

Re:If you can afford it, move to Java

[japer]

Actually, it is a misconception that java doesn't suffer from memory leaks. Just because it is garbage collected doesn't mean that you don't have huge trees of objects that will never get gc'd. If you don't explicitly design in creation/destruction of your objects you will leak memory. For the average sized application it propably isn't noticeable, but if your app needs to run for a long time, you will eventually run out of heap.

Re:If you can afford it, move to Java

[dubl-u]

Yes and no.

Actually, it is a misconception that java doesn't suffer from memory leaks.

A classic memory leak is one where you have memory allocated but no longer have any way to make use of it. These kinds of errors aren't possible in a garbage collected language. Once you let go of your last pointer to something in Java (or, say, Perl) then the GC takes care of everything automatically.

When debugging other people's C code, the most common memory leak I see is the temporary allocation of scratch space in some low-level routine without a corresponding release of the memory. This does not happen in Java.

If you don't explicitly design in creation/destruction of your objects you will leak memory.

Agreed. This is especially a problem with people new to OO design, where they haven't yet developed a good understanding of modular integrity. In practice, though, I find these kinds of errors much less frequent and much easier to find than classic memory leaks.

Re:If you can afford it, move to Java

[japer]

Actualy, I don't mean that the VM can't figure out if an object is dead or not. The problem arises as designers make design decisions without understnding the consequences.

The problem arises when, from the designers point of view, the object will no longer be used, but from the VMs point of view, the object is still alive. An object that has a root parent in the reference hierarchy will not be garbage collected.

Things I have seen that hold onto large object trees are things like, anonymous innner classes (implcit "this" reference), listeners on gui components not being removed, and static class variables holding onto objects.

Java is a great programming language because it is easy to use thanks to all the built in stuff, but nothing is free. You generally pay for it with performance and memory usage.

Here's an article discussing "memory leaks" in java

http://www.devx.com/upload/free/features/javapro/1 999/06jun99/tl0699/tl0699.asp

Don't change horses in the middle of the stream

[SLOGEN]

Are you really changing Language and OS for better multi-thread debug?

This is one of the symptomathics of IT today. You start out analyzing the problem, then you choose an aproprite tool. Some month's later you hit trouble and you consider moving to new language, OS, architecture, whatever, totally discarding the initial analysis that led to the choises you made!

On topic: You should not expect multi-threaded code to be easily debugged, afterall it is hard to write!

You should instead put effort into the really core components that are multi-threading sensitive (synchronization when referencing datastructures and stuff like that, btw. C++ is excellent for that, using Monitors), and make all other code REQUIRE to use the methodology you develop in these components.

Try to collect the multi-thread sensitivity in small, well-defined areas of your program, and try to prove, or at least reason, about the correctness of the code there. One thing you may wan't to do is define (in C++) Locking Iterators, which Lock datastructures they work on, and (using reference counts) automatically unlocks the structure when Iterators fall out of scope.

You may also have some luck in using some of the design patterns that deal with multi-threading.

Morale:You cannot debug away every race-condition, that's bad practice you HAVE to think.

Re:Don't change horses in the middle of the stream

[the_quark]

What you say is true, but it's not the issue. You can design the multi-threaded sensitivity properly, you can implement to the best of your ability, but, eventually, someone IS going to do the equivilant of *0 = 1, and your program is going to dump core. Even if it's just a typo. Believe me, I know, I founded the Freeamp project, and it is multithreaded to hell and back, and it has been a nightmare to get all the crash bugs out of it.

The problem is, that, under Linux, if you load the resultant core file (from any multithreaded program) into GDB, it often dumps core. Which isn't very useful.

The end result of this is that you can have a dual PIII-800 workstation, running the very latest in tools, with a highly trained developer, doing a friggin' binary search through the code with printfs trying to figure out where the stomped pointer is coming from.

Believe me, you do this a few time and you're gonna start thinking that a language that does memory management for you probably wouldn't be such a bad idea, either.

My OT rant - RedHat/Cygnus (leader of the gdb effort) needs to get this fixed. Quite frankly, it's hard to take Linux as a serious development platform when multithreaded debugging is twenty years back. Support the developers, first, and everything else will follow.

Fun programming

[Far�]

Functional programming languages will provide you with great tools that will avoid most sources of core-dumping:

• OCaml is multithreaded (linuxthreads was born as a way to get it to work on linux), although a single program won't take advantage of a multiprocessor. Core dump is impossible using its static strong typing, and the optimizing native-code compiler provides with great performance.
• JOCaml is an extension of OCaml based on the Join-Calculus, the latest and greatest paradigm for distributed programming (can be seen as actor-based programming done right, as based on a well-understood algebra). Can take advantage not only of multiprocessors, but of processor farms, or any distributed architecture, even heterogeneous architectures. Only bytecode can migrate, native code modules and primitives must be compiled into the servers.
• Erlang is based on a paradigm quite similar to jocaml, but is designed for industrial applicability rather than hacker coolness; it has dynamic typing, a pure functional core (despite the logic programming syntax, it only has matching, not unification), enriched with explicit asynchronous communication primitives, and an implementation that every phone call you make depends on, if you use British Telecom. If you need tens of thousands of threads and/or lots of nodes, this is what you need.
• There is also Mozart, an distributed implementation of Oz, a real logic programming language (has unification).
• Haskell and Mercury also have extensions for concurrent and distributed programming in the works, but I admit I don't know how usable they are for real programming.

Use real languages with a nicely designed concurrent programming algebra, not fake languages that were never designed to begin with, and had a kluge of concurrent primitives added by drunk people without a clue. Forget core dumps. Catch exceptions and receive exit messages instead.

-- Faré @ TUNES.org

debugger and threads

[Anonymous Coward]

As far as debugging is considered I would recommend DDD. It is a visual frontend to various language debuggers including C++. And it does provide access to threads of a process. So although it does not replace gdb, it may make using it easier for you.

I would also suggest using a thread class that encapsulates the pthreads interface. I have a system in production that uses the omniThreads thread library. omniThreads is part of the omniORB library, but it compiles stand alone without any ORB related stuff. (FYI: I compiled it as a shared library ) It is very portable across machines (may help when switching to another machine)

Writing threaded programs is very easy in Java. However, my apps have been simple enough that I have not had to debug the threaded application. If your application is a server app, with not so realtime requirements, you may want to try java. If it is a GUI app, you probably should not !!

Totalview

[gormanly]

Well, I do parallel F90 development, and the only debugger worth the name that I've come across is Totalview from Etnus. Supports C++ on Linux and the tech support's pretty good. Free trial version too.

Solaris

[MichaelJ]

I used Solaris (on Sparc, though) quite extensively for multithreaded development using their SunPro Workshop (4.x) environment. A very well-done IDE that integrates directly with Emacs, your own Makefiles, and overall can be as much or as little visual vs. command-line as you'd like.

The debugger is dbx, fully supporting multithreaded debugging, and has a full GUI on top of it, as well as full visual integration into Emacs (better than you've ever seen gdb do).

Some people think that Solaris' thread libraries are less than adequate. Personally, I've never used a better multithreading system. The entire "kernel-scheduled LWP managing user-scheduled threads" many-many concept is more efficient and works far better than Linux's one-one kernel-scheduled threads. And all your threads run with the same PID - what a concept!

The obvious disclaimers are that I haven't used the newest Solaris tools, nor was this done on x86, but definitely consider not only the tool quality but also the OS' attitude towards threads. Solaris is better than Linux in that regard, IMO.

Michael J.

Don't change

[heikkile]

Firstly, multithreaded stuff is hard to write, and harder to debug. No fancy tool will get you around that. Changing to new languages, tool, and environments usually screws up the first project or two, and it sounds like your project does not need another reason for things to go wrong.

I can not know much of your situation, but on the surface it looks like a perfect example of the need of a clean rewrite. Scrap all the code you have, and the whole design, and consider all of it at most a preliminary study. But for crying alound, keep the same people and the same tools, unless it is blindingly obvious you are trying to use a totally wrong tool for the job - and C++ for multithreaded programming is not such a bad choice. If need be you may have to add one or two good, experienced people in your team.

Then, start from the beginning. Design the system all again. You know you have problems with thread, so do not overuse them. Consider each thread a black box, and specify its interfaces, just like you are supposed to do with single-threaded systems. Specify the lockings and controls you need for things to work. Then implement bottom up, and test everything as you go.

Spend more time in designing end implementing test suites than on the project itself - this will cost more at the onset, but increases your chances of getting the job done, probably faster than without the investment.Build a good debugging log that can be written from any thread, and where you control bufering. Build test routines to delay things, and pepper your code with them. Routines to exercise each module independently, and various combinations. And what ever else you can think of, that might come in handy. Make most tests automatic, so you can leave them running every night if need be. But most of all, make a clean and understandable design, and do not introduce new languages, tools, methods, or any other reasons for confusion!

Thinking and discipline

[wocky]

In my experience using various debuggers over a number of different projects, none of them are particularly useful with multithreaded programs. Almost any of them are good enough to get stack dumps of the various threads and to print data values when an assertion fails, but they don't help much beyond that.

The best way to get a parallel program working is discipline and understanding. Think about how the threads will interface with each other. Design those interfaces using appropriate known and proven synchronization methodologies. Adhere rigorously to those designs. If you're tempted to leave out a lock in some place, you'd better be able to prove that it can be done safely.

If necessary, write wrappers around your synchronization primitives for better observability and controllability. For example, when the program crashes, you want to be able to see which threads are holding which locks, which are waiting for what conditions, etc. Being able to log the sequence of synchronization actions to a file and to force a replay of those actions in the same order will help in reproducing bugs that otherwise would manifest themselves only once every hundred runs.

Have manual inspections of the code with a coworker. Play a game where either of you can propose a failing scenario and the other must explain using the code why that can't happen. If neither of you can explain why it can't happen, it probably can.

Your basic philosophy should be that it's impossible to convince yourself of the correctness of the synchronization through testing. If you can't prove that your synchronization is right, it's most likely wrong. And if it's not wrong, it's so complicated that you'll break it later because you don't understand it.

The best debugger is 'printf'

[Reality Master 101]

Personally, I despise IDEs because they always slow me down. I've noticed that the best programmers never seem to use them. I think putting well-chosen printf's in the code forces you to think about what you're actually writing, rather than writing a bunch of garbage and then using the debugger to bail you out.

A lot of people think I'm crazy when I say this, but both Linus Torvalds and Gostling have said similar things.

I've developed both ways, and there is no doubt in my mind that it takes me less time to develop when I just use printf's. But -- it does take more discipline, and that's probably why you see so many mediocre programmers slaves to the debugger and IDE.

vi, make and printf (with the occasional grep and find) -- that's my debugging environment, and I'm known for my development speed and code quality.

--

Re:The best debugger is 'printf'

[Switchback]

This is the type of comment I hear from novice and inexperienced developers. Most new hires right out of college seem to have this fear of debuggers and their debugging skills consist of printf's everywhere. Now don't get me wrong, printf's or some other diagnostic output (such as logging) is very handy to hopefully gain a handle on what is going on, but it's not the solution you want to use if after looking at the output you still have no clue what's happening.

Debuggers (both command line and GUI) are all essentially the same. Learning one isn't doesn't take very long (on the order of a few minutes) to do the basic things like set breakpoints, step, and watch variables.

Here are the problems with printf (and other diagnostic output):

• They don't give you the whole picture and very rarely point you to the exact location of your problem.
• They have to be maintained. I don't know how many times people have cut & pasted a printf but failed to change its output causing utter confusion. Or the code changes and the printf didn't. They can become a maintenance headache which can actually hinder the troubleshooting process. The should be controlled somehow (preprocessor..some logging level varible etc.) but getting everyone to do that everywhere consistently is also a problem.
• They can be misleading especially if the output has not been flushed yet.
• They can get in the way of the actual running of the program and mask bugs (especially race conditions).
• They can't tell you when a pointer or variable gets stomped on.

Debuggers on the other hand let you see your entire application state and you can see in an instant if everything is right.

• If the program crashes you immediately see what went wrong and you can view the call stack to see how it got there.
• You can easily debug multi-threaded programs because you can force potential race conditions and see if your code works properly or not.
• You can dynamically alter the flow of your program yourself and force potential problems to check your code behaves properly. Using a debugger is a great way of testing all conditions a piece of code might face: valid input, edge conditions, and invalid input.

'Debugging' with printf is an iterative and lengthy process of looking at the output, adding more printf's (adding to the maintenance problem), compiling, and looking at the output again. With a debugger, I almost always find the problem on the first or second pass through. You also invariably find other potential problems lurking in the code that just haven't shown up yet. Tracking down a problem with a debugger usually takes me just a few minutes, but going through the printf process takes much longer and may never show you the real problem.

Think of it this way: You go to a doctor saying you dont' feel well. The printf doctor pokes and prods you until you yell 'ow!'. The debugging doctor uses the intelligent approach where he looks at your symptoms, runs some tests and makes a much more accurate diagnosis of your problem.

- Rick Alther

score -1, offtopic

[mkettler]

I agree, it's quite ridiculous that this post has been moderated up. It's purely off-topic.

I've actually USED paradigm C++, along with paradigm link and locate tools. It's for embedded application development. (ie: raw rom hanging off an x86 processor bus with no bios (you actually run where a BIOS would be), no OS, etc). The dev tools themselves run on ms-windows.

This tool has little to do with multithreading, in fact I don't belive it contains a thread library at all. (I could be wrong, but I never saw one and it's not a listed feature) It supports real mode, and if you're x86 can do it, protected. It will target as low as 80186's and NEC v## processors.

It also has nothing to do with UNIX. It's for people building non-pc x86 based systems that have no BIOS and no OS. It's for bare-metal embedded c++. You can't even run the IDE on UNIX, much less the target.

They have a site at devtools.com, visit if you're bored, see how much Multithreaded Unix support they have...

My personal opinions of this software aren't very high, but mostly because my opinion of x86 for raw-soldered embedded is pretty low. The product itself is relatively well written.

Shouldn't be the major decision factor

[photon317]

It worries me that you're willing to change platforms/languages/etc just to find the right debugger. If you stick to good programming practices, coredumps should happen rarely to begin with, and therefore the tradeoff is there to have a better language/os and a really hard time dealing with corefiles. Also, most good programmers, even when dealing with a serious bug causing a coredump, only use a debugger to get a backtrace... they don't really _use_ the debugger. And in many circumstances, marker statements in the source code are even simpler than using a debugger for a backtrace. In light of all these things, I could never justify changing my whole coding world arounda decision about a better debugger... something else is wrong here.

Re:Gnu Visual Debugger

[segoave]

Or you could try ddd
If you need a visual dbugger. It acts like gdb, which I already love, but adds a visual edge that gdb alone doesn't have. When I am really stumped with gdb, I go to ddd.

CodeWarrior

[kriegsman]

When writing threaded code the developers must take the responsibility for understanding the threading issues and creating thread-safe code. But the right tools can help you get there more quickly and sanely.

We've used Metrowerks CodeWarrior to develop and debug very heavily multithreaded applications on MacOS, NT, Solaris, and Linux for about five years now; the CodeWarrior tools seem very much 'up to the job' and thread-aware. As a nice plus, CodeWarrior runs on multiple platforms (including Mac OS X now), which is a nice plus for our development, which is all multiplatform.

We now use a diversity of compilers/IDEs/debuggers, but CodeWarrior is still a favorite, even if it's just because of the "Blood, Sweat, and Code" T-shirts.

Comments from Pro with Lots of Experience

[Acceleration]

Lot of good points already made.

• Have small set of proven core threading routines.
• Don't use threading unless it's really justified. At times, it seems right, but you can lose lots to context switches. So reducing threading can help too.
• Can't debug this stuff to demonstrate correctness. Must THINK.
  
  To all this I'll add:
• While staying in mainstream language land (which is worth a LOT), Java has great thread support. It works well. It works well on all platforms (excepting W95). You can use your OS of choice and switch at will for the most part. Linux/NT/Solaris/whatever. I use NT.
• I like being able to use truss on Unix. It's *very* helpful in unraveling knotty problems. I have the most experience with this on Solaris. Solaris threading is good stuff.
  
  Java is a lot more productive than C++.
  C/C++ can be lots more runtime efficient.
  
• In lots of circumstances, these efficiency gains are unimportant.
  
• Usually more important to efficiency is algorithmic choices and quality of program implementation -- and few developers are worth a damn in this arena.
  
• So C++/Java is usually moot point regards efficiency trade-offs, because programmers product is such slop anyway.
  
• When programmer's code is good, then a small subset of the time, you're better off in C++ for runtime efficiency.
  Aside from that, you're better off in Java.
  
  I miss C++ (and even C) sometimes, but Java is usually a better language overall. Not hard to migrate.
  
  If you go with Java, take a strong look at Doug Lea's stuff.

TotalView

[Sax Maniac]

TotalView probably does what you need. Other people have said it here, but I can add a little more info.

It's a parallel debugger, which means that it has built-in support for manipulating groups of objects, as if they were one: threads in a proceses, processes in a group, groups in a cluster, and so on. This means you don't need 20 windows to control a 20-thread app; we roll up an aggregated view into one window, commands work on the entire batch.

And yes, we support Linux, as well as almost every other Unix out there. You can snag a free demo license and download the bits from our website. And for those of you who like printf(), you can add them on-the-fly without recompiling. That saves time.

Disclosure: I am a developer on TotalView, but I do "eat my own dog food"- we use it every day on itself.

Re:I was expecting someone

[dubl-u]

Wow! Who peed in your cheerios this morning?

I was expecting someone to make this stupid suggestion. [...] What the hell has this got to do with his suggestion - he's asking about debugging threads. [...] This is a fact that you pulled out of your ass.

When a developer (or team of developers) is spending a lot of time debugging and still not solving the problems, I ususally take this as a sign that there are more fundamental problems than the quality of the debugger. I've been developing serious software for 15 years in Pascal, C, Perl, C++, Objective-C, and Java, and I rarely need to us anything more than the occasional printf.

Admittedly, developing threaded code is hard stuff, and I don't know enough about the original poster's problems to say what is going on. But it is possible that they are in over their heads, in which case it may behoove them to a) admit that they don't know enough and start again from basics, and b) they may wish to choose a language that is more forgiving of inexperienced developers.

In that case, suggesting Java is a good idea. That's not to say that C is bad; if I am doing something speed-critical and am working with a team of crackerjack developers, then C would be my first choice.

But if I'm doing something where maximizing CPU efficiency isn't our #1 issue, or if I'm working with developers who are less than stellar, then I lean towards Java. Why?

• No pointer errors - Experts may not make pointer errors, but average developers sure do. With Java, there are no buffer overruns, no broken pointer arithmetic, no SEGVs. And even better nobody else making some stupid pointer error that hoses one of your data structures, making you spend days looking for a bug that isn't there.
• No malloc errors - Java allows you to pay less attention to memory allocation. This makes classic memory leaks impossible, and subtle leaks harder. You're right that novice Java developers take longer to learn the value of reference handling than C developers, but this is mainly because Java extracts a much smaller penalty for those errors. By your logic, presumably, C++ would be even better if each time the developer left a dangling reference they received a high-voltage shock to the nipples.
• Exception handling - It's been several years since I've used C++, so maybe the exception handling there has improved. But Java's exception handling is a good thing, making it much easier to track down errors when they do happen.
• Multiple VMs - When I am getting some weird-ass error, it's wonderful to be able to try several different VMs on different hardware platforms. That removes all question of OS issues, endian problems, or bugs in the run-time.

So the "maybe you should use Java" is a reasonable answer to the question, especially since the original poster specifically mentioned that they were looking at using java.

Ergo, there was no need for you to be an asshole about it.

Re:fair enough

[dubl-u]

I over-reacted, but whenever someone mentions C++ someone says, hey - why not use java and all your problems will go away, I get a little tired of it.

Me too. As far as I can tell, these are people who have never done any serious work in Java. Or if they have, then Java is the only language they've used on a serious project.

Luckily, these people get what they deserve. Eventually they will be dumb enough to say things like this to a boss, who will be dumb enough believe them. And then when their optimistically-scheduled, poorly-scoped, under-budgeted project goes up in flames, they will get fucked with the sandpaper condoms. The smart ones learn after the first time that no tool is perfect for every job. The dumb ones, of course, talk trash about last tool and find the next perfect tool to talk about.

Having the program die horribly would probably suffice, but the high-voltage shock to the nipples would be even better. Immediate catastophic failure is a far more useful reaction to bugs during development than limping along, papering over the cracks.

And people call Java a bondage-and-discipline language! Heh. Maybe we should dress this idea up in a lot of fancy talk about neuropsychology and maximizing feedback loop efficiency and see if we can get VCs to cough up a few million dollars to get us going.

Are you kidding?

[Carnage4Life]

In the true open source mind, it would be be better to say:
You are changing a debugger because it misses a feature? THEN IMPLEMENT IT

Comments like this make me wonder exactly whether Slashdot is read by programmers or simply people who have heard about programming and think it's cool. Multithreaded applications are hard to design correctly and difficult to debug from an application writer's point of view. Your simplistic statements belie the fact that you must be an inexperienced programmer because in the real world people don't have time to start om mammoth projects simply to help with a medium sized one. Adding threading support to gdb is more difficult than writing a multithreaded application that uses the pthread library unless the application is very complex like a compiler, relational database management system or a web browser.

Your comment is like telling someone to hack garbage collection into C when they complain about memory leaks instead of simply pointing the person to Purify or BoundsChecker.


Grabel's Law

Use processes instead of threads

[johnnyb]

Use processes instead of threads in as many places as make sense. This makes many of your problems simply vanish. It makes you rethink your design in a much more thorough and thoughtful fashion, choosing distinct parts with obvious interfaces.

Re:The common JAVA chant

[dubl-u]

Really, I agree with many of your points in the abstract. As a language for top-tier, ass-kicking developers who are wise, subtle, and wily, Java has a lot of annoying constraints.

They did this for a reason though. For a lot of real-world software development, you have to do the work with painfully small amounts of time, money, and talent. So they banned a number of things that it takes an expert to use wisely. E.g., pointers, multiple inheritance, allowing unreachable code, preprocessor macros, raw memory allocation, random memory access, self-modifying code, and so on. As cool as those features are in the hands of a genius, they are plain dangerous in the hands of a mediocre developer. And 99% of the time, the genius will be doing what Java would be doing anyhow; it's only the 1% of the time that it sucks.

That's why I'd much rather inherit a bunch of J. Random Programmer's code in Java than almost any other language. There will be little impressive wizardry in it, but there are also unlikely to be many sections that will make me bleed through my eye sockets.

And you're also right about some of the other feature lacks; the whole primitive type thing is just ugly, and it's clear that they haven't heard about the whole mutable/immutable thing yet. Really, it saddens me that they are just now catching up with a lot of the things that NeXT was doing right with Objective C 5-10 years ago.

But as far as getting things done in the real world for server side stuff goes, it seems perfectly adequate for all the OO work I do. And to be fair to them, they're making a fair bit of progress; the java.lang.ref package, for example, answered a lot of my gripes about pointers and garbage collection.

One thing I didn't understand in your post was the section "lack of parametrised types"; could you talk more about that?

Re:The common JAVA chant -- parameterized types

[HarpMan]

Parameterized types are templates in C++. They allow you to do generic programming.

With templates, you can write a function like this:

template
DataType max(DataType a, DataType b)
{ return (a>b ? a : b); }

This function returns an object by value, and is type safe. By type-safe, I mean that it require s that the two arguments and the return type all be the same (or compatible) types. For example, this would be a compiler error:

int x = 3;
complex c(3,4);

int y = max(x, c); // whoops -- different types!

This is actually impossible to do Java. You can't make it return a by-value object, and it's awkward to enforce the type constraints.

Templates allow generic programming, where you write functions and classes (think abstract data types) that are agnostic as to the exact type of some of their parameters. But, you still get compile-time type safety. Templates are great for container classes and algorithms. Container classes in Java are workable, but a bit awkward. You don't have compile-time type safety. The container's values get cast to Object going in, and have to be manually cast back to the desired type when getting an element from a collection. With C++ templates, you can declare containers of specific types, like 'vector' or 'vector'.

For example:

vector v(3); // three element vector of ints

v[0] = 45;
v[1] = 3;
v[2] = 4;

sort(v.begin(), v.end()); // sort the whole vector

int first = v[0]; // no need to cast

All of this is type-safe, and about as efficient as hand-written code.

Also note that the sort function is generic too -- it works for lots of different container types (like list), including container types you write, that the author of sort did not anticipate. Templates are very 'pluggable', both horizontally and vertically. Generic programming has a very different feel from OO programming. It's less hierarchical, easier to plug and play things piecemeal. Of course, you can still use a vector of MyPolyMorphicObject references,('vector to combine OO and generic programming.

There's a lot more to templates then what I just described. The STL is a great example of a powerful, flexible and efficient generic library. Some people say that ML's generic facilities are even better than C++'s. This may be true; I've never used ML. Other people say that you don't need templates at all, that the real problem is strong typing. In languages like Smalltalk, Python and Perl, you don't have to declare the types of arguments, so every thing is generic. Of course, you can get run-time errors if you pass in the wrong type. But you can also get run-time errors in Java if you try to cast to the wrong type.

Strong typing without templates is the worst of both worlds, in my opinion. If you're going to have strong typing (which I personally like), then you really must have support for templates or something like them. Otherwise you get the huge proliferation of complicated interface hierarchies that you see in Java, just to make sure that classes are useable various contexts. Templates decrease the need for multiple inheritence (or interfaces) somewhat.

minimize debugging; use checks and abstractions

[q000921]

I've written a number of pretty substantial multithreaded programs, and I have never used a debugger for fixing them. I don't even know what I'd be looking for with a debugger: serious bugs in multithreaded programs tend to be subtle race conditions, and that's not the kind of thing that's easy to track down with a debugger.

A better way is to avoid bugs in the first place. And for multithreaded programs, a message passing paradigm is generally the best approach in my experience.

I also use assertions and consistency check liberally, so that problems are caught close to where they occur. As a result, the few serious bugs that I have had were usually solvable by inspection of the source code alone.

If you are looking for languages that make avoiding bugs easier, look beyond C++. Java is slightly better than C++ for multithreaded programs because errors tend to be more localized, but Java doesn't have much in the way of useful higher-level abstractions for building multithreaded programs. The currently conceptually best languages for multithreaded programs, in my opinion, are SML/NJ, OCAML, and Erlang.

Use a CVS snapshot of gdb.

[cduffy]

The CVS snapshots of gdb have (for the last several months) had vastly higher quality multithreading support than the released versions. Indeed, I think threading only firmed up properly within the first month after gdb 5.0 was released! So grab the repository from :pserver:anoncvs@anoncvs.cygnus.com:/cvs/src, compile it and have fun!

It depends on app type/completeness

[JohnZed]

If you guys are far along in the development process with c++ already, or if you have limited experience with Java and Java threading, you'll just run into worse threading problems there by trying to fit the pthread paradigm into the Java platform.
That said, I do a lot of multithreaded programming in both Java and C++, and I would give damn near anything to switch completely over to Java. The threading design makes it FAR easier to build correct programs from the ground up without using a debugger. We've known for 15 years that monitors (Java's synchronized blocks, basically) were a better way to write multithreaded programs, and it's embarassing that it took this long to get the idea into a mainstream language.
Also, you should check out jlint (http://www.ispras.ru/~knizhnik/jlint/ReadMe.htm), which checks java programs for basic errors including many types of thread deadlocks. A lot of commercial Unix vendors have similar tools for C/pthread programs, but I haven't found one for Linux.
As for actual debuggers, JBuilder 4 is quite good. I've only used the foundation (free) version, but the enterprise edition can also dynamically detect stalls, deadlocks, and race conditions (see http://www.borland.com/jbuilder/jb4/feamatrix/debu gging.html) and do remote, multi-platform debugging. Also, MetaMata (http://www.metamata.com) has a very good debugger that includes some advanced threading features even in their low-end versions.
On the down side, you'll have to pick a good implementation of the JDK on Linux and stick by it, as the different JVMs tend to have threading incompatabilities. IBM's are quite good, and they support native threads very well. In Sun's 1.2 implementation, at least, native threads were an undocumented feature (with green threads only officially supported).
Hope this helps!
--JRZ