Slashdot Mirror
What Debugger Is Best For Multithreaded Apps?
pollyp asks: "I'm on a team that's working on a multi-threaded (pthreads) C++ application running on GNU/Debian 2.2. One of the challenges of this project is trying to debug the app, or making sense of core files; the feeling is that gdb is just too difficult to work with in a heavily threaded app. We're looking for better tools, and if that means changing languages or going to another UN*X, we're ready to do it. So what is the best, most mature OS/language/tool UN*X environment for doing multi-threaded application development on Intel?"
"One part of the team wants to stay on GNU/Linux but use Java. Java's performance is becoming less and less of an issue, and the GNU/Linux development tools are reasonably attractive (e.g., Visual Age's debugger). The other part of the team wants to stick with C++, but move over to Solaris. But none of us have any practical experience with Solaris, nor do any of us have a sense of how fully baked their Intel development environment is."
Why change views when you change from editor to debugger and back like older tool chains force you to do? In Paradigm C++, the editor and the debugger are the same view. A click lets you edit code, another click lets you rebuild your application and you have never left the position of your code to make the change and confirmed the correction. This is the most time-efficient method of debugging.
.AXE node in the Project window. Once in a debugging session, the Edit window local menu (right-click in window) and Speedbuttons under the Menu bar display debugging commands. The Load command under the Debug menu will load a .AXE file on the target and start a debugging session if you don't have a project file.
.AXE node in the Project window, selecting TargetExpert and choosing a target type. You can even run a protected mode application without a remote target system and test your code right on the PC. This can be done by choosing Win32 emulation as your Target Model in TargetExpert. Memory models can also be changed in this dialog.
You can start a debugging session by double clicking the
You can build your application in real or protected mode. Choosing a Target platform is as easy as right-clicking the
Paradigm C++ even has the smarts to inspect an object by simply passing the mouse cursor over it in the Edit Window. Since it also knows the debug symbols in the current scope, you always have access to the most frequently used debug information without any additional effort.
During debugging, the editor view includes local menus containing debugging options such as Inspect or Watch. Source lines with debugging information are marked with red dots and the current execution point and enabled/disabled breakpoints are all marked.
You can open a CPU view to see the actual code created by the compiler or assembler or see register contents. This window handles viewing of memory, registers, processor flags, and stack and includes options to help you checkout low-level code. You can bring up the CPU view at the current CS:IP by using the View menu command. Or to bring up the CPU view corresponding to any source line, use theView CPU command from the Edit view local menu.
The gnu visual debugger is based on gdb, and claims to have much-improved support for multithreaded code. I haven't used it myself, though.
If you're considering a different language, Ada is worth a look. Gnat is a complete and stable compiler in a way that g++ is not, and the language was designed for multithreading from the start.
...and be aware that there are many, many bugs in Solaris' threading libraries.
Vintage computer games and RPG books available. Email me if you're interested.
So what is the best ... tool ... for doing ... application development?
16 cups of this.
As to performance: if you're not writing a nuclear explosion simulator, Java's performance is probably adequate. (Think StarOffice here.)
--
Being well balanced is overrated. -- John Carmack
Are you really changing Language and OS for better multi-thread debug?
This is one of the symptomathics of IT today. You start out analyzing the problem, then you choose an aproprite tool. Some month's later you hit trouble and you consider moving to new language, OS, architecture, whatever, totally discarding the initial analysis that led to the choises you made!
On topic: You should not expect multi-threaded code to be easily debugged, afterall it is hard to write!
You should instead put effort into the really core components that are multi-threading sensitive (synchronization when referencing datastructures and stuff like that, btw. C++ is excellent for that, using Monitors), and make all other code REQUIRE to use the methodology you develop in these components.
Try to collect the multi-thread sensitivity in small, well-defined areas of your program, and try to prove, or at least reason, about the correctness of the code there. One thing you may wan't to do is define (in C++) Locking Iterators, which Lock datastructures they work on, and (using reference counts) automatically unlocks the structure when Iterators fall out of scope.
You may also have some luck in using some of the design patterns that deal with multi-threading.
Morale:You cannot debug away every race-condition, that's bad practice you HAVE to think.
SLOGEN [ http://ungdomshus.nu : Sebastian cover music]
- OCaml is multithreaded (linuxthreads was born as a way to get it to work on linux), although
a single program won't take advantage of a multiprocessor. Core dump is impossible using
its static strong typing, and the optimizing
native-code compiler provides with great performance.
- JOCaml is an extension of OCaml based on the
Join-Calculus, the latest and greatest paradigm
for distributed programming (can be seen as actor-based programming done right, as based on
a well-understood algebra). Can take advantage not only of multiprocessors, but of processor farms, or any distributed architecture, even heterogeneous architectures. Only bytecode can migrate, native code modules and primitives must be compiled into the servers.
- Erlang is based on a paradigm quite similar to jocaml, but is designed for industrial applicability rather than hacker coolness; it has dynamic typing, a pure functional core (despite the logic programming syntax, it only has matching, not unification), enriched with explicit asynchronous communication primitives, and an implementation
that every phone call you make depends on, if you
use British Telecom. If you need tens of thousands of threads and/or lots of nodes, this is what you need.
- There is also Mozart,
an distributed implementation of Oz, a real logic programming language (has unification).
- Haskell and Mercury also have extensions for concurrent and distributed programming in the works, but I admit I don't know how usable they are for real programming.
Use real languages with a nicely designed concurrent programming algebra, not fake languages that were never designed to begin with, and had a kluge of concurrent primitives added by drunk people without a clue. Forget core dumps. Catch exceptions and receive exit messages instead.-- Faré @ TUNES.org
-- Faré @ TUNES.org
Reflection & Cybernet
As for debuggers, I've used DDD (a gdb frontend) but never with threads. I tend to just use the old trick of sticking a few printf statements in key places.
As far as debugging is considered I would recommend DDD. It is a visual frontend to various language debuggers including C++. And it does provide access to threads of a process. So although it does not replace gdb, it may make using it easier for you.
I would also suggest using a thread class that encapsulates the pthreads interface. I have a system in production that uses the omniThreads thread library. omniThreads is part of the omniORB library, but it compiles stand alone without any ORB related stuff. (FYI: I compiled it as a shared library ) It is very portable across machines (may help when switching to another machine)
Writing threaded programs is very easy in Java. However, my apps have been simple enough that I have not had to debug the threaded application. If your application is a server app, with not so realtime requirements, you may want to try java. If it is a GUI app, you probably should not !!
Well, I do parallel F90 development, and the only debugger worth the name that I've come across is Totalview from Etnus. Supports C++ on Linux and the tech support's pretty good. Free trial version too.
The debugger is dbx, fully supporting multithreaded debugging, and has a full GUI on top of it, as well as full visual integration into Emacs (better than you've ever seen gdb do).
Some people think that Solaris' thread libraries are less than adequate. Personally, I've never used a better multithreading system. The entire "kernel-scheduled LWP managing user-scheduled threads" many-many concept is more efficient and works far better than Linux's one-one kernel-scheduled threads. And all your threads run with the same PID - what a concept!
The obvious disclaimers are that I haven't used the newest Solaris tools, nor was this done on x86, but definitely consider not only the tool quality but also the OS' attitude towards threads. Solaris is better than Linux in that regard, IMO.
Michael J.
Michael J.
Root, God, what is difference?
On the other hand, I've experienced that LinuxThreads (the thread implementation provided by GNU libc) and GDB don't mix too well. Maybe you can try another pthreads implementation, for example the one from Florida State University (FSU Threads).
Threads are notariously hard to debug. Linux threads have been in my experience especially hard. Make sure your using the latest kernel as there are some problems with the If you think about it, how would one debug a highly threaded application? What would you do with something that had 20+ threads? I can just imagine a programmer sitting there switching between terminal...
int func(int a);
func((b += 3, b));
I've been usign GVD for a couple of weeks and works well with multithreaded programs:
http://www.act-europe.fr/gvd"
And it also supports native and cross debuggers (VxWorks, LynxOS, JVM, ...), remote launching of debuggers, different languages, etc.
--
To visit or not to visit: findusclub.com
The opinions in this comment are subject to GPL, you can copy, modify and redistribute freely (as in speech).
The best debugger is your brain analyzing the observable behaviour of the code. Read it, study it. Fix the bugs based on that. With a debugger, you tend to fix symptoms, not the disease. Anything else is second best at best.
Almost all of the Linux kernel was debugged that way.
That you feel it's ok to change the language, or even the OS? I mean if you are at the debugging stage, then unless you have a really simple application (which wouldn't be heavily threaded then), you have all the original conceptual coding done. I agree with the other posters that there's something fishy about this article.
Anyway, I've done some concurrent programming, and in my experience Java is a great language for concurrency. The fact that threads are a built in feature of Java (unlike C++ which has add-ons i.e. p-threads) provides a very powerful tool for concurrent programming.
Java Threads are easy to use, and to learn. I'm assuming you already know Java, but if you don't, you know C++, and if you can afford to rewrite your whole application you can spend the half a day it'll take you to learn Java.
If you are in unix you can look at m_fork from SGI's libraries, but I still say to stick with Java.
-"Those who fought today will die tommorow."-
but Irix has a couple of really handy debugging utilities, Purify and TotalView Debugger. I haven't used Purify much, but TotalView is incredibly handy.
-fnord
Go on, mod me down. There aren't enough trolls on this discussion.
Personally I don't debug code. If it's borken I just give out the source to someone esle to fix. I only gpl buggy stuff, then I ungpl it when it works and claim all the credir. Now I'm not offtopic.
Hope you all get first posts and trolls for Christmas
I can not know much of your situation, but on the surface it looks like a perfect example of the need of a clean rewrite. Scrap all the code you have, and the whole design, and consider all of it at most a preliminary study. But for crying alound, keep the same people and the same tools, unless it is blindingly obvious you are trying to use a totally wrong tool for the job - and C++ for multithreaded programming is not such a bad choice. If need be you may have to add one or two good, experienced people in your team.
Then, start from the beginning. Design the system all again. You know you have problems with thread, so do not overuse them. Consider each thread a black box, and specify its interfaces, just like you are supposed to do with single-threaded systems. Specify the lockings and controls you need for things to work. Then implement bottom up, and test everything as you go.
Spend more time in designing end implementing test suites than on the project itself - this will cost more at the onset, but increases your chances of getting the job done, probably faster than without the investment.Build a good debugging log that can be written from any thread, and where you control bufering. Build test routines to delay things, and pepper your code with them. Routines to exercise each module independently, and various combinations. And what ever else you can think of, that might come in handy. Make most tests automatic, so you can leave them running every night if need be. But most of all, make a clean and understandable design, and do not introduce new languages, tools, methods, or any other reasons for confusion!
In Murphy We Turst
Your options (?) essentially consist of two tools: DDD (http://www.gnu.org/software/ddd/) and GDB 5.0 (http://sources.redhat.com/gdb/)
I wouldn't go with the latest beta version of DDD, but use the latest release version. You *must* use GBD 5.0 in order to get a hope to work with threads.
-- Anne Marie
Do i have the reocrd for most stupid posys without being modded down beyond the first?
Hope you all get first posts and trolls for Christmas
Don't get me wrong. I love Debian/GNU. But let's face it, it is more difficult for them. In this particular case Sun has significant advantages to deliver products adapted to their stuff. A debugger is tighted to the OS and the compiler. Sun has put features in its debugger (thread support) before it appeared in gdb. Concerning C++, gcc is behind others (see advogado for a recent discussion), it is hoped that this will change with gcc 3.0, but it is still not there. It would not be surprising that some of your problems come from here (and gdb 5). You may want to consider to make your development on a platform (Sun sparc, Solaris, Forte ex WorkShop, ex SunPro) with portability in mind and then make the port to Intel. This particular combination is good, but other should be as well. You may also consider to compare differences between a sun sparc platform and a sun X86 platform to see if they are of the same quality. Understandably, quality should be lower for X86, but how much? Now or afterwards, you may want to explain your difficulties with GNU development tools and to pinpoint where they are behind of others for C++ multithread development as objectively and precisely as possible to avoid flaming. Regards, --
to make this stupid suggestion..
1. What the hell has this got to do with his suggestion - he's asking about debugging threads. It would have made some sense if you had talked about threads in Java and the tools provided to debug them (for instance JbuilderX allows thread debugging, but it's far from ideal). Java threads are notoriously error prone same as threads everywhere.
2. "Given the fact that 90% of all C / C++ bugs.."
This is a fact that you pulled out of your ass. It may be roughly true for novices or students, but is nonsense for experienced developers. With tools like purify, these problems are very easy to track down in C/C++, and 2nd nature to avoid.
Java encourages people to forget about these problems and rely on gc, then they leave dangling references dotted around and wonder why their disk is grinding.
3. The performance hit is not minor. Sometimes, in simple cases, Java works great, but it depends what you're doing. An uncontrived comparison is at http://sprout.stanford.edu/uli/java_cpp.html
In summary, with HotSpot 1.3 beta, (a JIT that Sun touted as being faster than C++ !!) Java was roughly 5 times slower than C++.
4. Is StarOffice supposed to be an example of a good Java application with acceptable performance ? Heaven help us, it makes Word2000 seem slick and efficient.
Are there *any* examples of serious Java applications with acceptable performance ?
http://rareformnewmedia.com/
Here's a better place: www.adapower.com/gvd
I have authored some pretty intense multi-threaded network servers and find that the tools and environment offered by Sun are far and above the best. The Sun Workshop 5.0 C++ compiler and debuggers are amazing. The OS is rock solid, and the envionment is a genuine joy to work in. You can outfit your developers with Ultra-5 workstations including the compiler all for just $3000 through the end of December too. Solaris 2.6, 7, or 8 are all a good choice.
KK4SFV
A doctor "fixes" patients by examining their symptoms, determining what their disease is, and then curing it.
Programmers do the same thing. Merely thinking about a problem and mentally stepping through code is a piss-poor way to figure out what's going wrong. Stepping through code with a debugger is a hundred times more effective, especially in a team environment where you are debugging code you didn't write, or when a junior programmer is misusing a library routine he thinks he understands.
While it is true that some programmers take stupid actions based on what they see from the debugger, that is because their brains are stupid -- not because they are using a debugger. The debugger is merely another tool for diagnosing a problem, and provides food for thought. Nobody has ever claimed that is a replacement for a good brain. Yeesh.
--
Do daemons dream of electric sleep()?
Note that might want to have a look at the Ada 95 language, it has pretty good support for portable multi-threading (aka tasking, builtin into the language) and a very good compiler: GNAT.
Laurent
In my experience using various debuggers over a number of different projects, none of them are particularly useful with multithreaded programs. Almost any of them are good enough to get stack dumps of the various threads and to print data values when an assertion fails, but they don't help much beyond that.
The best way to get a parallel program working is discipline and understanding. Think about how the threads will interface with each other. Design those interfaces using appropriate known and proven synchronization methodologies. Adhere rigorously to those designs. If you're tempted to leave out a lock in some place, you'd better be able to prove that it can be done safely.
If necessary, write wrappers around your synchronization primitives for better observability and controllability. For example, when the program crashes, you want to be able to see which threads are holding which locks, which are waiting for what conditions, etc. Being able to log the sequence of synchronization actions to a file and to force a replay of those actions in the same order will help in reproducing bugs that otherwise would manifest themselves only once every hundred runs.
Have manual inspections of the code with a coworker. Play a game where either of you can propose a failing scenario and the other must explain using the code why that can't happen. If neither of you can explain why it can't happen, it probably can.
Your basic philosophy should be that it's impossible to convince yourself of the correctness of the synchronization through testing. If you can't prove that your synchronization is right, it's most likely wrong. And if it's not wrong, it's so complicated that you'll break it later because you don't understand it.
David
There's been a lot of discussion so far about moving to Ada and other languages, and other OSes. I have a similar, but not identical problem: a program written in pure C on Linux (and Win32, via cygwin) that uses threads extensively. So far, I haven't found a debugger that'll do worth a darn at following threads. Rewriting is not an option. Got any suggestions here?
--
Disinfect the GNU General Public Virus!
We have a heavily threaded C++ program that we develop at my company. It compiles under Sun Workshop 5.0, and Visual Studio. I would recommend getting workshop or the new Forte 6.0 C++ tools for Solaris on Sparc. They are great tools. The Front End for DBX makes debugging multiple threads very easy and when you dump a core, you get a call stack for all of your threads. That is much more easier in my opinion then when I was doing pthreads under xxgdb.
Another nice part about doing it on Solaris is that after you get all the bugs worked out of it, you can bring it back to a linux platform and run it there. Our Application compiles and runs under Solaris Sparc, Solaris x86, Linux x86, and Windows, so if you design it well you can get the portability with C++.
maybe you should consider moving to NT? MSVC has the coolest debugger in the biz imho...
I bet I've written more java than you.
I spent about two years doing full time java
development (full fledged scientific java application for Alcan). I first learned java five years ago, and have tried it on and off since then. I recommended using it in that project because the superficial impression it gives is pretty positive, and *anything* is better than using MFC. (I wish I had known about FLTK back then)
I would not recommend Java for client side application development. If you really care I could drone on for days about why it's a bad idea. Personally I wouldn't use it on server side either, but I agree that it is pretty good for that. Sun has spent many millions of dollars putting together suitable libraries and tools to make it a good tool for server development.
Actually, I had thought soffice was largely java because it's a misconception that's quite common and also because soffice is slow and bloated, but not as bad as I expected for a java application. I think it's largely C++ though. Please give me an example of a good, substantial Java client-side application if you believe I'm just spreading FUD.
http://rareformnewmedia.com/
As for Java vs. C++, well Java isn't a bad language, and having threading built in rather then slapped on as a bag on the side makes it nicer (pop quiz: what C++ facet operations are thread safe?). On the other hand I feel a bit limited in Java, and I miss the STL. As far as I know SPIN will help Java coders as well.
I don't think Solaris x86 is a good idea. If you want Solaris, get a SPARC. Otherwise you are always behind the support curve, and the hardware support is far more limited then Linuxes. Besides the SPARC hardware is extreamly reliable (if not all that fast), and works great for lights-out operations.
In "The Practice of Programming", Kerighan and Pike specifically say that other approaches besides debuggers are necessary for multi-threaded programs. Debuggers encourage laziness anyways. The less you rely on them, the better code you will produce, in my opinion. Code reviews, debugging print statements inside the code, etc. will make you look at the code more, think about it more, and be more defensive in your style. This will always produce more robust code.
I'm not a programmer and I have never seen it in action, but SGI has an OSS product named Jessie that's written in Java. I don't know what the performance is like.
Personally, I despise IDEs because they always slow me down. I've noticed that the best programmers never seem to use them. I think putting well-chosen printf's in the code forces you to think about what you're actually writing, rather than writing a bunch of garbage and then using the debugger to bail you out.
A lot of people think I'm crazy when I say this, but both Linus Torvalds and Gostling have said similar things.
I've developed both ways, and there is no doubt in my mind that it takes me less time to develop when I just use printf's. But -- it does take more discipline, and that's probably why you see so many mediocre programmers slaves to the debugger and IDE.
vi, make and printf (with the occasional grep and find) -- that's my debugging environment, and I'm known for my development speed and code quality.
--
Sometimes it's best to just let stupid people be stupid.
Yes, the best way to debug programs is to not ENBUG them. Lo-tech: Read the code. Have each thread print trace info to a log so you can see the interaction and interleaving of the execution. And if you really want a great system for doing multithreading and are willing to change your entire universe to get there just use OS/2. :-)
The revolution will NOT be televised.
Please give a reference for where 'toy' languages have been used and have failed.
Use printf()!
I agree, it's quite ridiculous that this post has been moderated up. It's purely off-topic.
I've actually USED paradigm C++, along with paradigm link and locate tools. It's for embedded application development. (ie: raw rom hanging off an x86 processor bus with no bios (you actually run where a BIOS would be), no OS, etc). The dev tools themselves run on ms-windows.
This tool has little to do with multithreading, in fact I don't belive it contains a thread library at all. (I could be wrong, but I never saw one and it's not a listed feature) It supports real mode, and if you're x86 can do it, protected. It will target as low as 80186's and NEC v## processors.
It also has nothing to do with UNIX. It's for people building non-pc x86 based systems that have no BIOS and no OS. It's for bare-metal embedded c++. You can't even run the IDE on UNIX, much less the target.
They have a site at devtools.com, visit if you're bored, see how much Multithreaded Unix support they have...
My personal opinions of this software aren't very high, but mostly because my opinion of x86 for raw-soldered embedded is pretty low. The product itself is relatively well written.
-Matt
First off, I'll say I recommend a combination of C, Unix (any flavor will do), any compiler that is ANSI standard, your favorite editor, make, debugging statements and light gdb use. These get me farther than any IDE I've had to work with or tried to work with. Incidentally, every developer I can think of who I admire for producing absolutely brilliant code, especially quickly, tends to use the above. Just a favorite editor, whatever their language choice is, and simple tools. Remember the KISS principle.
#1: Can you really switch your language or OS?
At best I'm dubious as far as your suggestion of switching language or operating system. Unless your project is only at the very first stages, you have almost undoubtedly invested far too much effort to reasonably consider backing out. Why did you choose C++ and GNU/Linux in the first place? Surely more thought went into it than "we like C++" and "Linux is nifty", so go back to those decisions and sift for clues.
#2: Why are you spending so much time in a debugger?
You should be spending more time with your concepts, flow charts, and whatever else you came up with before you laid it all into code. Spend some time pondering what exactly you're trying to do, then go look at your code and make sure it's really doing what you want. A few well-placed debugging output statements (printf/cout/whatever), some good comments, and a clear head will get you much farther than a debugger, even the best. Debuggers absolutely cannot tell you what you're thinking.
#3: Do you really need threads?
Is your application specifically designed to be run on parallel systems? Is it designed to take the majority of a system's resources? Chances are you don't need threads, most programs don't. There are usually better ways to do things than multithreading. Also, writing threaded code is considerably more time-consuming than non-threaded code. Are you writing something so complex you need threads? If you haven't really evaluated this, now is the time.
-wd
--
chip norkus(rl); white_dragon('net'); wd@anduril.org
mercenary albino programmer for hire
"question = (to) ? be : !be;" --Shakespeare
11*43+456^2
And your already unit testing your app, you are a little late to be considering a platform/language switch.
We didn't have any debuggers. And we only got 5 compiles to get it right. Of course we are talking about Cobol here. MUHAHAHAHA
I rely on debuggers for a traceback after crash. However, that rarely tells what the program was doing before it got into trouble. For that you need not only the current state, but the previous states looking back in time. Debugging a transport protocol under multi-client load is a good example. I'd suggest creating a tracebuffer fifo, simple means for annotating the threaded code with calls that log state in the buffer, and means for extracting the contents and displaying in various interesting ways. It's easy to do this. It's portable. And it will yield information far beyond what is possible with language/OS-centric tools that know nothing about the theory of operation of your app.
I see people talking about Solaris debugger, dbx and praising it. I can not see how anyone can like that program. Its the worst debugger I have ever used.
I worked for 3 years at a large aerospace company and we did some truly large C++ projects. They were all at least nominaly threaded because they used a CORBA implementation (which was threaded itself). I can not count the number of times that dbx core dumped on me because I tried to look at the wrong symbol name.
Granted, it had nothing to do with threads, but the linker and symbol demangling. But given the fact that it was almost impossible to use on any project of any good size, I do not count that as a good debugger. Also, about a 1/3rd of the time the debugger failed to actualy stop a progam at break points. In my experience, the MSVC++ debugger is much more stable and usable. Dont get me wrong though, I am a Linux person and cant stand Windows. I just have to port Linux apps to Windows for my current job.
Just give me linux and I'll be a happy person. (that includes gdb, with its couple of flaws) Flaws in an Open Source product dont bother me at all. They will be fixed. Flaws in a coporate product will only be fixed if the company can make more money because of it. Guarenteed.
RedShodan --------- Never underestimate the bandwidth of a station wagon full of tapes.
the power of printf. Create a trace mode that spits out a rediculous amount of information about what is happening. I have found this to be much more useful than a debugger in multithreaded apps.
--
47% of all statistics are made up on the spot.
If you have problems with debugging deadlocks and other thread-related problems, I strongly recommend state-threads. If you are struggling with the traditional thread libraries way of dealing with shared address space, State Threads could be the right choice for a threads library. State Threads were designed for server/client I/O intensive applications. The library is portable, scales extremely well on multi-processor machines, and locking is not necessary since it allows for private thread data natively (no mutexes and locks needed, everything is user-space also). The server.c that's shipped with this library is not your common multi-threaded application, very easy to understand and doesn't suffer from locks. It's GPLd. http://oss.sgi.com/projects/state-threads/
We've used Metrowerks CodeWarrior to develop and debug very heavily multithreaded applications on MacOS, NT, Solaris, and Linux for about five years now; the CodeWarrior tools seem very much 'up to the job' and thread-aware. As a nice plus, CodeWarrior runs on multiple platforms (including Mac OS X now), which is a nice plus for our development, which is all multiplatform.
We now use a diversity of compilers/IDEs/debuggers, but CodeWarrior is still a favorite, even if it's just because of the "Blood, Sweat, and Code" T-shirts.
To all this I'll add:
Java is a lot more productive than C++.
C/C++ can be lots more runtime efficient.
Aside from that, you're better off in Java.
I miss C++ (and even C) sometimes, but Java is usually a better language overall. Not hard to migrate.
If you go with Java, take a strong look at Doug Lea's stuff.
Hahaha, that's subtle, but I do it all the time too!
;)
Yeah, for debugging, I just use fprintf(stderr,..,...)! Save my time a lot, bro!
It's a parallel debugger, which means that it has built-in support for manipulating groups of objects, as if they were one: threads in a proceses, processes in a group, groups in a cluster, and so on. This means you don't need 20 windows to control a 20-thread app; we roll up an aggregated view into one window, commands work on the entire batch.
And yes, we support Linux, as well as almost every other Unix out there. You can snag a free demo license and download the bits from our website. And for those of you who like printf(), you can add them on-the-fly without recompiling. That saves time.
Disclosure: I am a developer on TotalView, but I do "eat my own dog food"- we use it every day on itself.
I can explanate how to administrate your network. You must configurate and segmentate it, so it can computate.
If you are doing hard real time where timing tolerances are really critical, neither printf or a software debugger are of any use. The printf will alter the timing the very thing that you are trying to observe. Although there are some fairly expensive solutions, I like using the parallel port and an oscilloscope. Instead of printf's, insert direct io writes to the parallel port into your program. View the output on a multichannel oscilloscope or logic analyzer. Some people find this approach incongruous with an operating system like Linux where such bare metal tactics seem at odds with the prevailing higher levels of abstraction. But timing tolerances can only be observed by direct viewing. If this is a project with actual funding, there are solutions that are less improvisational. Many logic analyzers have bus preprocessor cards that can be inserted into an empty card slot of almost every conceivable chassis type. The logic analyzer also come with software that will display the traces in nice formats for viewing. You can then see the timing relationships of both your inserted I/O writes as well as any other activity. On a modern computer, often all the other activity is too confusing so the logic analyzer can be set up to filter it out. There are also complete logic analyzers that can be inserted into a card slot. In spite of all the modern gadgets now available for real time debugging, I still occasionally find use for some of the improvised tricks I learned before such devices were for sale. Larry
> Here's the true version: 90% of programming bugs come from not reading Dijkstra.
It is one thing to read Dijkstra and know what programming is about, and another thing to build a hundred thousand line , multithreaded application. If you are right (and I wish you were) someone who has reached the stage of programming Enlightenment can type in a application like that and have it work from scratch, in any given decent language. Someone who is able to do that needs no debugger at all. I personally know of no-one like that, and even if these people exist, the vast majority of the programmer-population (inluding you and certainly me) is confronted with his own mistakes on a regular basis. It is irritating and humbling in a way, but a problem we have to deal with anyway. And debuggers can be a great help.
--
Being well balanced is overrated. -- John Carmack
OO programming is great.
Data hiding and abstraction is great.
Implementing the wheel again in every piece of code is not great.
Perhaps you should learn the PHILOSOPHY behind OO programming at some point, rather than the syntax words behind its C++ implementaiton.
By far, the most easy-to-use debugger I've seen
:) I havn't used the thread support yet, but judging by the quality of the rest of the program, thread support should be very good.
for the Unices is kdbg:
http://members.nextra.at/johsixt/kdbg.html
The latest development versions have thread support. The latest development versions are more stable enough, btw
<soapbox>
About 30 years ago an upstart operating system called UNIX came up with a solution to all of the problems with two threads of control that are running simultaneously trying to get access to the same resources. Threads have race conditions, require synchronization primitives to avoid stepping on eachother's toes or access inconsistent data structures, and when out of control can modify data owned by other threads bringing the whole system down in a heap.
The brilliant innovation of UNIX was to seperate threads into seperate processes, each with its own stack and data and memory barriers between processes so that no process should have to be completely dependent on the good behavior of another process.
Since that innovation limited the interoperability of processes, UNIX also invented a series of interprocess communication tools including pipes, sockets, and explicitly shared memory. Together these tools have greatly simplified and are probably even responsible for popularizing multi-processing.
So why are we ignoring these fine inventions?
Because Microsoft is unable to implement an efficient multi-processing environment, threading is the only way to efficiently handle multiple simultaneous requests under Windows NT. I think that this limitation has made threading using no memory protection, and no IPC, into a fashionable approach for 'efficiency' and 'performance'.
But is that really a good idea? Computers these days are extremely fast; so that even VMs like Java are often considered sufficiently fast for use in real projects. Under those circumstances, wouldn't it be better to avoid multi-threading in favor of multi-processing? Isn't it really debugging that takes the most time, so that losing a bit of performance for a massive increase in reliability should be an acceptable tradeoff?
</soapbox>
Submitted for your consideration. There are obviously technologies that don't work well in a multi-processing environment; CORBA for example depends heavily on threads, as does Java (running a single JVM is bad enough; can you imagine running a whole array of them one for each thread?) But I do think that people underestimate the value of Processes and IPC and often choose Threads when they are highly inappropriate.
LibBT: BitTorrent for C - small - fast - clean (Now Versio
As many other posters have said wisely, switching tools midstream is a sure route to pain. Trust me, once you've made core tech decisions, stick with them! At least until you get version 1.0 out the door...
I too don't like using gdb on the command line, but combined with DDD, the Data Display Debugger (works natively on pretty much any Unix, linux strain, I've seen ports to win32, etc) debugging is a much nicer experience. I don't write multitrheaded stuff being a pretty neophyte C++ programmer (eh, we all have to start somewhere :-) ), but I did notice in the online manual that it does have at least rudimentary support for threads.
Plus it's free and doesn't require any changes to your environoment or toolchain. (Basically just install it, it defaults to using gdb as the "inferior" debugger (as opposed to dbx or any of the perl/python/java/whatever debuggers), and you type "ddd progname" instead of "gdb progname". It even has a command window to interact with gdb directly if you so wish...) Last but not least it has a cute logo. ;-)
--
News for Geeks in Austin, TX
I'm currently building some server-side Java web applications. Since the Java Servlet stuff is inherently multi-threaded, I have indeed occasionally been reminded that multi-threaded applications are tricky.
A few months back, we purchased a suite of tools from an outfit called Sitraka (nee KLGroup). They have three products that are part of the JProbe Suite, a CPU and memory profiler, a thread analysis program, and a code coverage tool.
These are all good tools; when I was having some problems that I suspected were due to my less-than-perfect understanding of threading, I used their thread analysis tool under simulated load, and it immediately identfied my race condition, plus a couple more potential races I hadn't noticed yet.
(This is a little off-topic, but I have to mention that their memory/CPU profiler is, pardon my french, fucking awesome; it is the best thing I have ever seen for visualizing the interior structure of a running program. After a day with the profiler, my Java code was substantially faster than the C it was replacing, despite having more features and being more secure.)
Another tool I'm very pleased with is JUnit, a unit-testing framework. If you're interested in trying out the Extreme Programming-style approach to testing (wherein you make automated, integrated unit tests that are run more or less continuously) then this is for you. And if you are having so many problems with bugs that you are considering changing languages, then I would strongly recommend that you do this. Good unit tests slow initial writing down a little, and save you extraordinary amounts of time and agony later.
Oh, and run out right now and buy several copies of Code Complete and Rapid Development for the team. If you are having such large problems on the project, the problem is probably not with your choice of debugger. These books will help you figure out what the problem actually is, and give you all sorts of solutions.
--
For the record, and I don't have any financial interest in any of the things I've mentioned here; I just use 'em and like 'em.
You make some good points, and I over-reacted, but whenever someone mentions C++ someone says, hey - why not use java and all your problems will go away, I get a little tired of it.
One thing though:
"You're right that novice Java developers take longer to learn the value of reference handling than C developers, but this is mainly because Java extracts a much smaller penalty for those errors. By your logic, presumably, C++ would be even better if each time the developer left a dangling reference they received a high-voltage shock to the nipples."
Exactly ! yes, that is an excellent idea. Having the program die horribly would probably suffice, but the high-voltage shock to the nipples would be even better. Immediate catastophic failure is a far more useful reaction to bugs during development than limping along, papering over the cracks.
http://rareformnewmedia.com/
I've tried Solaris on x86 just for this reason. I've found that it's hard to install, and has limited hardware support. But once it gets working, it's very nice for multi-threaded programming. The license is free, so you might want to get one copy to try. We just used it to develop, remove the problems from the threaded areas, then run the code on Linux, in essense it's just another debugging tool for our Linux app.
What you say is true, but it's not the issue.
I realize that, but I simply could not resist commenting on the change of devel tools after the sentece:
We're looking for better tools, and if that means changing languages or going to another UN*X, we're ready to do it
In my mind, this statement shows a common flaw in reasoning of IT project: changing tools based on a single premise (in this case, MT-debug support, normally the premise would be something like "easier GUI", "faster filesystem", "prettier icons",...), instead of making decisions based on the entire needs of the entire project.
It seems to me, that you may actually share this wiew with me:
you're gonna start thinking that a language that does memory management for you probably wouldn't be such a bad idea
I agree with that phrasing. I would like to see a subset of C++, that allows real garbage-collection (not the ad-hoc analysis in currently availible gabage-collectors). Even though that would mean dumping C compatability.
A story I hear particularly often is: you should use Java, it's better to debug than C++, and it's got Garbage Collection.
Well, that are definatly pro's for Java, but what about the con's:
- Java has NO! parametrised types This means that generic programming is impossible. One specific consequence of that is, that collections cannot be type-safe, and you loose a lot of the help that type-errors can buy you.
- Java doesn't have "const" (and friends) which means that when you give someone an object reference you have no control over what they do with that. When returning read-only object, the only solution is to clone() them. Very expensive!
- Java cannot reference simple types In Java, you can only reference Object's not the simple types. Even the integerObject (and friends) are immutable, so that doesn't help you pass reference parameters which have simple types (I realize, that this is normally a sign of OO-design flaw, but I really want that flexibility, especially for JNI code!)
- There are more, but these should make my point.
So, before switching some project to Java, do a cost-benefit analysis.Last but not least, I would like to second your statement, which gives me a chance to repeat something very wise you said:
Support the developers, first, and everything else will follow.
SLOGEN [ http://ungdomshus.nu : Sebastian cover music]
First of all, if you're going to do threads on linux, you're going to wind up debugging the race conditions in the standard library calls if you use C++. The GNU libstdc++ is not yet fully MT-safe. So, if your requirement is C++ and threads, you pretty much need to pick another platform other than linux.
Second, there's threads on linux. Linus spent a lot of effort to make sure that fork()+exec() work well under linux. There's three reasons to use multithreaded code: faster creation, faster context switches, and easier/faster message passing through shared memory in threads vs. e.g. SysV IPC. Under linux you don't get any gains for the context switches. Context switches for processes are only about 10% slower than context switches for threads under linux. You also don't win a whole lot in process creation which is faster under linux than thread creation is under solaris (plus pthread_create() is broken and slow under linux). You should consider looking at your programming model and seeing if good old fork() will work for you. If you don't believe me, check out this and the entire thread here. Also read this whole thread.
Then there's the fact that you've run into problems with the tools that you've got on linux not be adequate to handle threads. Not only is gdb broken, but strace isn't going to help you with threads either. And in disagreement to those who claim that printf() is good enough for everyone (e.g. Linus), I think that its good to have many different debugging tools in your toolbox. Strace and reading corefiles in gdb usually are my first line of getting a handle on what the problem is, then I usually read the code and use printf()s. So this is another strike against threads under linux.
I can't speak on the virtues of C++ vs. Java. I'm merely a lowly sysadmin / system programmer who predominantely uses C. I do know though that lots of very good programmers make mistakes because they know how to program but don't understand the architecture of the systems that they're writing code for. They believe things like "threads are fast" without either hitting the linux-kernel mailing list or the linux sources to find out how threads and processes are actually implimented under linux.
So, if you're going to stick with threads then stay the hell away from Linux and use some other OS like Solaris. If you can avoid using threads, though, Linux may be a perfectly sufficient platform for you to use. Once you've made that choice, then you need to decide if you're going to use Java or C++ and I'm out of my league there.
And when you're considering if you should throw out threads, remember:
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." - Donald Knuth
That suggests to me you shouldn't use pthreads unless you need to.
Speaking of Python, I know that ddd supports debugging of Python programs, but I've never been able to make it work. Any ideas?
I've run production environments with Solaris on both x86 and SPARC. I'm primarily a SysAdmin/Architect, not a developer, so take your coding cues from someone else (I program, but not heavy-duty professionally).
Previous to Solaris 8, there were A LOT of serious bugs in Solaris x86 that caused it to be unusable for serious work. For web serving and simiar crap it was fine, but any complex app tended to expose bugs in the OS. Solaris for SPARC doesn't have these problems.
Solaris 8 for x86 seems to have fixed most of the egregarious problems, and I would trust it to run your app. HOWEVER, x86 still seems to have random low-level (ie driver and the like) problems with certain hardware. Thus, I would seriously recommend that you DEVELOP on a SPARC box, which doesn't have these problems. That way, you can be sure that the bug you've just found is really in YOUR code, and not some obscure problem in the OS. One you have stable code, Solaris 8 for x86 is stable enough to run as your production OS.
Both x86 and SPARC use the same codebase - thus, the system libraries and similar stuff all work equally well (or at least, have identical bugs...). However, the Hardware Abstraction Layer implementaion for x86 is of poorer quality than the one for SPARC (which makes sense, given Sun's priorities).
Best of Luck.
-Erik
There are always four sides to every story: your side, their side, the truth, and what really happened.
Obviously you have no idea what the poster of the "Ask Slashdot" is talking about.
He is asking for a good debugger for multi-threaded programming such as dbx on Solaris because gdb cannot handle threaded debugging. Instead you respond with an uninformative post that describes the worst way to debug multithreaded programs. printf isn't thread safe so it's use in multithreaded programming as a debugging tool is frowned upon unless you are using additional mutexes for logging (adding complexity) or you are using a thread safe print function such as OutputDebugString in the Win32 API.
Grabel's Law
I'm all for open source advocacy and gdb, etc...but telling someone who is trying to debug their huge MT app to "write a patch!" is ridiculous.
Slashdot 's editors are dickheads
That C++ programmers should be using cout and the string class instead of char[]
Trolls throughout history:
Jonathan Swift
C is great, so is C++
C++ has some great features which can
solve hours of work. It may get bloated
sometime if you don't have a good
optimizer but it is nonetheless a very good
language.
Use what you like and leave the rest of
the world who love C++ alone.
I've used their V25 debugger before but
where the heck is a Linux version of any
of their debuggers?
I've never seen any Linux or Unix version
of their debuggers.
As for the V25 debugger I've used, it seemed
a derivative of Borland turbo debugger and
only runs on dos.
I haven't had time to look into this, but there is a gnu compiler which allows compilation of Java into native code for your platform (thus eliminating the need for a VM and the performance drop). It also (I believe) allows for the use of byte-code classes along with the compiled code, if you wish. It is called gcj (GNU Compiler for Java) -- see http://sources.redhat.com/java/ for more info
In the true open source mind, it would be be better to say:
You are changing a debugger because it misses a feature? THEN IMPLEMENT IT
Comments like this make me wonder exactly whether Slashdot is read by programmers or simply people who have heard about programming and think it's cool. Multithreaded applications are hard to design correctly and difficult to debug from an application writer's point of view. Your simplistic statements belie the fact that you must be an inexperienced programmer because in the real world people don't have time to start om mammoth projects simply to help with a medium sized one. Adding threading support to gdb is more difficult than writing a multithreaded application that uses the pthread library unless the application is very complex like a compiler, relational database management system or a web browser.
Your comment is like telling someone to hack garbage collection into C when they complain about memory leaks instead of simply pointing the person to Purify or BoundsChecker.
Grabel's Law
Solaris 8 on x86 (and even more so sparc) is almost certanily what you are looking for. Their threading libraries are excellent (barring the very rare weird problems that were in Sol 7, but even then there were certainly less than the heap of dung which is Linux's "threading" situation, and then I have found any of these bugs, only through the grapevine... and i've coded a whole ton of MT code). Even better than their great threads implementation is their compiler and debugger. their debugger, dbx, is absolutely fantastic. I cannot emphasize enough how good this debugger is. gdb doesn't even warrant being called a debugger compared to this amazing bit of work. important for you, it's threading support is fantastic, especially in versions 5.0 and 6.0. It is possible to get the Solaris 8 source code (have to fax them something) and of course the OS itself is really cheap (just media costs or something, like $10). You can also get demo versions of all their compilers and tools, so you can see for yourself how amazing they are. I strongly recommend you do, you will certainly not be disappointed.
JAVA has (according to you)
Java has other serious problems:
Basically, JAVA lack many of the advances in OO-technolgy that it should have embraced. The JAVA language is a very BAD OO-language, it just has a lot of nice standard libraries, and an emulator for the JVM is available for may machines.
This just goes to show, that there are NO really good OO-languages out there (that I have had my hands on anyway)
SLOGEN [ http://ungdomshus.nu : Sebastian cover music]
Of course. If you are writing... say... a high-level sysadmin tool, or a mail client, or any other sort of application that is far, far from the machine, printfs are perfect.
But if you start doing anything low level at all, printfs are *completely* inadequate. Writing a user-land threading library, for example. If you're handling context switching, or stack size, or anything even remotely close to the hardware, it becomes fantastically easy to... break printf. In which case it's nice to have a debugger handy.
Of course, for most of the higher level stuff, printfs should be all you need. What our VB coding friends would call "business logic," rarely escapes beyond the scope of printf. But everything underneath that really requires more powerful tools. In fact, debuggers are certainly the best way to tell if 'n' where you're overruning stack size. And they're even more necesary when you're working on machines without preemptive multitasking or protected memory.
--
There are no trails. There are no trees out here.
This was the same app written in java that I thought of as well... I've used Togethersoft for about a year. It never crashes on me, but is only decently responsive to input on my system (a 600Mhz P3 with 128M ram). It's worthless or won't even work on machines with under 128 ram and a cpu less than P2/P-pro.
"My mother works for Microsoft now. A whole other cult."
You're making some assumptions about the character of C++ programmers (like myself) that may not lead to the intended results. Remember, we chose to become C++ programmers... what does this say about our (possibly masochistic) personality? Maybe those electric shocks would increase dangling references :-)
"My mother works for Microsoft now. A whole other cult."
Use processes instead of threads in as many places as make sense. This makes many of your problems simply vanish. It makes you rethink your design in a much more thorough and thoughtful fashion, choosing distinct parts with obvious interfaces.
Engineering and the Ultimate
GVD, the GNU Visual Debugger, incorporates all the features of regular debugger GUIs and is complemented with data display and visualization capabilities more powerful than those offered by DDD (the Data Display Debugger). GVD provides the ability to debug multi-process/multi-threaded applications in the same debugging session.
Links for those interested in Ada and GVD:
I spent several years fixing other programmer's problems in a multi-threaded webserver in a very high traffic shop. We supported all sorts of unixes (HP-UX, Irix, Linux, Solaris), and by far my favorite platform for diagnosing and fixing problems was Digital Unix (I don't know what Compaq calls it these days) and their tool ladebug. They also have a tool for monitoring threaded control structures (mutexes et al) and could sometimes detect race conditions and potential race conditions. very cool.
People everywhere are moving to not-so-performance-efficient but easier to work with platforms (Java is a prime example here, Mozilla another)
No IT-Manager with a half a brain would frown upon buying new machines every year (or even every few months) for budget reasons, while losing thousands of $ because the programmers aren't working at top-efficiency on computers which don't support the best development tools.
(And yes, sometimes vi _is_ the best tool. But for most of the applications it has ceased to be that at the start of the nineties...)
--------------------------------------
Ack! the end of my sig is missing (its a TSE fragment) and I'm too lazy to fix it now.
What's a sig?
Parameterized types are templates in C++. They allow you to do generic programming.
// whoops -- different types!
// three element vector of ints
// sort the whole vector
// no need to cast
With templates, you can write a function like this:
template
DataType max(DataType a, DataType b)
{ return (a>b ? a : b); }
This function returns an object by value, and is type safe. By type-safe, I mean that it require s that the two arguments and the return type all be the same (or compatible) types. For example, this would be a compiler error:
int x = 3;
complex c(3,4);
int y = max(x, c);
This is actually impossible to do Java. You can't make it return a by-value object, and it's awkward to enforce the type constraints.
Templates allow generic programming, where you write functions and classes (think abstract data types) that are agnostic as to the exact type of some of their parameters. But, you still get compile-time type safety. Templates are great for container classes and algorithms. Container classes in Java are workable, but a bit awkward. You don't have compile-time type safety. The container's values get cast to Object going in, and have to be manually cast back to the desired type when getting an element from a collection. With C++ templates, you can declare containers of specific types, like 'vector' or 'vector'.
For example:
vector v(3);
v[0] = 45;
v[1] = 3;
v[2] = 4;
sort(v.begin(), v.end());
int first = v[0];
All of this is type-safe, and about as efficient as hand-written code.
Also note that the sort function is generic too -- it works for lots of different container types (like list), including container types you write, that the author of sort did not anticipate. Templates are very 'pluggable', both horizontally and vertically. Generic programming has a very different feel from OO programming. It's less hierarchical, easier to plug and play things piecemeal. Of course, you can still use a vector of MyPolyMorphicObject references,('vector to combine OO and generic programming.
There's a lot more to templates then what I just described. The STL is a great example of a powerful, flexible and efficient generic library. Some people say that ML's generic facilities are even better than C++'s. This may be true; I've never used ML. Other people say that you don't need templates at all, that the real problem is strong typing. In languages like Smalltalk, Python and Perl, you don't have to declare the types of arguments, so every thing is generic. Of course, you can get run-time errors if you pass in the wrong type. But you can also get run-time errors in Java if you try to cast to the wrong type.
Strong typing without templates is the worst of both worlds, in my opinion. If you're going to have strong typing (which I personally like), then you really must have support for templates or something like them. Otherwise you get the huge proliferation of complicated interface hierarchies that you see in Java, just to make sure that classes are useable various contexts. Templates decrease the need for multiple inheritence (or interfaces) somewhat.
Stephen Molitor steve_molitor@yahoo.com
Some of my examples didn't get printed properly, leaving of the data type of the instantiations.
// three element vector of ints
vector v(3);
should have been:
vector<int> v(3);
Hopefully that makes things clearer -- the point is you declare the exact type of a container when you instantiate (use) it.
Should have used preview!
Stephen Molitor steve_molitor@yahoo.com
Just because Microsoft "invented" it, it's not nessesarily bad. It's bad if Microsoft can use it to control a market which they already dominate completely.
One thing we should learn from Microsoft is: embrace and extend.
SLOGEN [ http://ungdomshus.nu : Sebastian cover music]
If this is at all possible, try developing the initial framework under NT (yes, NT). Let me explain.
If most of your code is platform-independent, and, in my experience, this is often the case, you will be able to develop on any platform that supports threads if you are willing to put simple wrapper classes around the platform-specific bits. This gives you the double advantage of being able to hide ugly or repetitive API details, as well as letting you port to another platform by reimplementing a simple inner class. I've put together 200-line or less wrappers for threads & mutexes, file IO, ODBC, etc., that work under BSD, Solaris and NT. This makes writing cross-platform code almost trivial.
Now, about that NT business. NT supports Berkeley sockets, stdio, and almost-POSIX threads, so if you're writing daemons and such, you can write 90% of your code without ever cracking the Win32 SDK documentation. Just use main() and forget all the heinous Windows I/O stuff.
As to the original question of the post, the development tools on NT are pretty damn robust. If you're willing to use commercial (i.e. pay-for, non-open source) tools, the VC++ compiler and Dev Studio debugger are mature and top-rate. You don't need to use the IDE except for debugging if you don't want - makefiles and the commandline compiler work 98% the same as Unix. The Dev Studio debugger handles multiple threads as smoothly as anything you can find, it has breakpoints and watchpoints, you can view and change memory easily.
The other commercial tool for Win32 that is invaluable is BoundsChecker from NuMega (now Compuware). The is the most amazing thing for finding both kinds of memory problems - invalid memory reads/wriites and leaks. It runs your app like a debugger would, (optionally) popping up to the exact source line whenever anything tries to write past a buffer (static or alloced). After your app exits, it gives you detailed report of all memory leaks, their sizes, and the lines *and stack traces* where they were allocated. It can run standalone or plug into the Dev Studio environment. This is what I wish Electric Fence could do.
I realize this is a big change to suggest, and may well be impossible depending on your application/financial requirements. However, if you're in the early enough stages that you're able to change languages/platforms, and you have a budget, then this would be my suggestion: develop the basic app core and threading logic under NT, using Boundschecker religiously. Use makefiles. Periodically move the code to the Unix platform and make required changes to compile. Keep all platform-specific code nicely #ifdeffed. Write simple wrapper classes around any system services that take more than one or two lines of code (again, threads).
This is how I've been working for the last year or so. I've been writing utilities and daemons that run under NT and Unix, and I'm still amazed at how easily the ports go. It just works. And I will tell you this: The difference between tracing back in core files with gdb and letting your app run in the IDE until the debugger pops up with the exact line of source code, with stack context, where your bad pointer/divide by 0/whatever happened, with the app still running, is indescribable.
All that said, I really miss Turbo Debugger. Now *that* was a debugger.
What if life is just a side effect of some other process and God has no idea we exist?
Maybe if they signed a name to it.
The VS debugger is nice, yes, and it'll beat GDB or DDD in a feature comparison anyday. However, for 95% of the debugging tasks that I need to do, they're equivalent. Set breakpoints, set watches, get stack traces--the other stuff is nice but hardly necessary.
It is amazing what you can accomplish by including calls to short tasks in a big inner loop. While it's a bit harder to define your own state machine than it is to write a linear algorithm within a thread, it is SO much cleaner to debug and you can ensure the system is always in a known state when you need to do something.
With today's fast machines you can get away with stuff that would never have worked 10 years ago. At most in Windoze you might need ONE (1) extra thread to maintain the user interface, especially if you're using hoggy database tools instead of building your own files in an efficient manner. I have however updated flat sequential files of nearly a megabyte within single-threaded (don't laugh) VB apps (!) without noticeable UI glitches. Modern machines are just amazing. And that's not even to mention hauling an entire 5 or 10 megabyte array into RAM to do sorts and searches.
Whatever environment you are in, you probably don't need more than 2 threads, and if you think about it you can make one of them really, really simple so that you can fully debug it before you get to work on the other one. You can also temporarily haul your interthread comms out into the file system so you can actually run your threads as separate applications, each with its own debugger, before combining them into a single app. (Really, if you need tighter interthread communication than this allows, you probably don't need to be using separate threads.)
Just my $US0.02. But I write a lot of custom software which must run 24/7 and for which I am the only support staff, so I favor reliablility over all other considerations. I have started to write several threaded apps, but have never actually finished one. Unthreading them has always turned out to be easier than debugging the threads.
Brackets contain world's first nanosig, highly magnified:[.]
I'm going to add, as someone who spends their days working on massively parallel (>1000 procs) programs, yet another endorsement for TotalView. It is the only usable, robust, multi-platform parallel debugger that I'm aware of, and this is something I try to keep an eye on.
A better way is to avoid bugs in the first place. And for multithreaded programs, a message passing paradigm is generally the best approach in my experience.
I also use assertions and consistency check liberally, so that problems are caught close to where they occur. As a result, the few serious bugs that I have had were usually solvable by inspection of the source code alone.
If you are looking for languages that make avoiding bugs easier, look beyond C++. Java is slightly better than C++ for multithreaded programs because errors tend to be more localized, but Java doesn't have much in the way of useful higher-level abstractions for building multithreaded programs. The currently conceptually best languages for multithreaded programs, in my opinion, are SML/NJ, OCAML, and Erlang.
Sometimes people amaze me...
the above comment completely misrepresents the opinion of the author and should therefore not be heeded....no matter how accurate it may be. thank you
I've used both the IDE and printfs for debugging. The IDE is great for tracing flow through deeply nested calls. It's also good for pinpointing problems like infinite loops. For example, you keep hitting F-10, and then it goes grey on line 532. Bingo! That's the function that's gone loopy.
But then printf is better for checking the contents of things sometimes. For example, after you read in the data file, you can write a simple "dump" function for the data structure to check and make sure that it has the right contents. In general, printf is great when you know what value a variable should have at a particular point.
As for threads, I can't really say since I haven't done any.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
... you must make your code more simple, as you will never catch all possible troubles with a debugger. If multithreading caused it to be more complex, switch to either single thread with nonblocking operations, multiple processes or any combination of two.
And, of course, Java will ALWAYS remain inadequate by both performance and reliability -- five years is more than enough to fix a bad product even for Sun (ex: Solaris) if it really cared about doing that.
Contrary to the popular belief, there indeed is no God.
As an alternative, the Insight GUI component of GDB 5.0 has been MUCH more stable for me, although I don't know if it is any good at debugging threaded applications.
The wise programmer uses the right tool for the right job, and an interactive debugger is usually the wrong tool. If you have a core dump, it's worthwhile to take a quick look at the stack dump, etc., but beyond that, you should carefully question time spent on further interactive debugging. It's a mental monkey trap--it feels like work and it's the easy thing to do. And five hours later it'll still seem that way.
The disadvantages listed for printf debugging are either incorrect, or also true for interactive debugging.
Likewise, the advantages listed for interactive debugging are incorrect, or do not consider the tremendous timesuck that interactive debugging can be. Consider, for example, the first point: Yes, you may immediately discover that a null pointer was dereferenced, but this may well tell you nothing about what really happened. The true bug could have occurred hours ago in a completely different part of the program.
Stepping through your program is no replacement for really thinking about what's going on.
This of it this way: You go to a doctor saying you don't feel well. The printf doctor eyeballs you, carefully forms theories about what might be wrong with you and takes various measurements to validate/invalidate those theories. The (interactive) debugging doctor says, "Nurse--scalpel!"
"Not an actor, but he plays one on TV."
I develop MT daemon-type software that runs on a vareity of unices and NT and, with the occasionaly exception of dbx on IRIX, we can't debug squat in terms of MT-specific issues except on NT, using VC++.
Look, some projects, esp. ones that work on platforms where threads really do work well, really lend themselves to MT designs, and that takes modern tools. GDB, even 5.0, just ain't all that (to add insult to injury, our project makes heavy use of shared libraries, making for a good ten minute core file load-up time when going over NFS -- not so w/DLLs and SMB).
The VC++ IDE-based debugger is excellent, and will help you get work done. Period. Thread-specific stack-traces, memory editing, watching, tracing, the whole bit, in a fully MT situation. We have a unified, cross-platform, Make-based build system that allows us to bypass the IDE compeltely for coding, but the debugger just can't be beat.
"Professional coder on closed source. Do not attempt."
Check the easy stuff first:
1. Make each c++ object have a validate(), assert_valid(), of similar funcation - through a base class - this function has no body if not in debug mode
All that function does is to
-- 1. verify that the object has not been destructed (use a boolean flag set to true in constructor and false in destructor)
-- 2. whatever validation on the internal data in the object (e.g., verify that pointers are not null, file handles are open, etc.)
2. set each pointer to null after it is free()'d or deleted
3. (more difficult) write a wrapper library for each and every api function called:
-- 1. make each function inline
-- 2. make each function validate all arguments passed into it (e.g., pointers not null, file handles open, etc)
-- 3. debug print before and after the api call
-- 4. check return value - call a function that has a breakpoint in it if an error occurs - this function has no body if not debug compile - assert() kills the process in unix and does not allow you to dump the call stack/continue executing if an error occurs
-- 5. provide you several ways to call the function - see all of the file handling functions in unix that eventually call fcntl(). Usually, you create on function that does everything, and lots of functions that do nothing but provide different interfaces to the do-everything function
4. for device drivers and other things that are difficult to run under a debugger, set a control-c or similar type interrupt or ioctl() call to dump the contents of the data structures - NOTE see 5 below for problems with non-thread safe printf() type functions
5. some os's (e.g., ms windows) behave badly if you call api calls or even malloc/free during an interrupt. It is much, much, much better to send/queue a request so that your application can finish whatever it is doing then process the interrupt request normally.
6. put parameter/data structure validation assert_valid() type macro calls at the beginning of each function to catch bad parameters as soon as possible.
7. write a function to validate each type of data structure - i.e., an overloaded validate() that takes whatever struct you pass it.
8. use counting semaphores to watch for overly long queues waiting on each object.
9. have each thread uses its own private object to store data and not let any thread access shared memory space in the application except through well defined static function calls that use appropriate semaphores/critical sections.
whoops - black magic type coding secrets for the non-development people revealed.
The CVS snapshots of gdb have (for the last several months) had vastly higher quality multithreading support than the released versions. Indeed, I think threading only firmed up properly within the first month after gdb 5.0 was released! So grab the repository from :pserver:anoncvs@anoncvs.cygnus.com:/cvs/src, compile it and have fun!
Or rather, it did three years ago, the last time I used it. I assume this hasn't changed, but you never know (especially since Cygnus dragged their feet quite shamefully when it came to integrating the hurd changes into the mainline gdb).
-Miles
We live, as we dream -- alone....
In the Free world, CMUCL has kernel-level threads support (for FreeBSD and Linux), but only in the current development versions.
In the commercial world, Franz's Allegro CL has OS-native threads on Unix, Linux, and Windows.
One of the advantages of Lisp is that it makes it very easy to get working code written very quickly, giving you time to change your design if need be, and to go back and optimize only the parts of the code that need it (no point optimizing code that gets run .001% of the time). One of the disadvantages is that it makes it easy for poor programmers (or naive Lispers) to write amazingly inefficient code that works correctly. But well written Lisp runs comperable to or faster than C++. Both CMUCL and ACL compile to the native architecture, of course.
...printf, make, gcc, (g)vi(m) are the perfect tools... however, gdb is also a must, use it to get the stack trace so you know where to put those printf's!
Mike
Intel transfer the difficult from Hadware to software, for get more power, programmer need more technology. -- chinaitn
Regarding those kernel patches, though: There are several available on linux-kernel. The simplest, by Patrick Wildi, is a patch for 2.2.x which dumps only the thread which actually failed. Another patch by Terje Malmedal against 2.2.x dumps each thread into an individual core file. I'd post links, but (I just realized) my mouse is unplugged so I can't cut+paste; they shouldn't be too hard to find.
A quick search on Google shows another interesting-looking multithreaded core patch labeled as for 2.4.0t8 by John Jones and Jason Villarreal, apparently funded by Spinway Inc (good 'fer them!). At a quick glance-over, it looks to make a single combined core dump file for all threads. Once again, I can't cut+paste the URL, so you go find it yourself. :)
There's other work ongoing on 2.4test on the topic; see the thread "Anyone working on multi-threaded core files for 2.4 ?" for more information.
So fear not, folks, there's light at the end of the tunnel! :)
I work on multiple platforms at the Sun-Netscape Alliance. We develop the web server on IBM AIX, HP-UX 11.0, DEC OSF1 4.0/5.0, Linux 2.2, IRIX 6.5, Solaris 2.6/2.8, NT 4.0. The web server is inherently a multithreaded application and a good debugger has been a must in the part of my day job. I find the two best platforms for debugging to be Solaris and WINNT. Sun has with its Workshop by far the best multithreaded debugging tools on any Unix I have seen. Microsoft has a very nice debugger as well for NT as part of VC++.
IMHO the very worst Unix platform for doing anything thread-related is Linux. And it's got nothing to do with gdb, but with the Linux kernel. On Solaris, Irix and HP, gdb is available and works fine with threads.
It doesn't work with Linux threads because threads are independent processes in Linux. When you have one applications with 500 threads, it's just not practical to launch 500 copies of gdb on each of the processes in order to set one breakpoint accross all threads. You have to use another platform with decent debugging tools in order to do that.
FYI, at home I use OS/2, and IBM has an excellent multithreaded debugger too, as part of Visual Age C++ 3.0/3.6 for OS/2. IMHO it's the best debugger of all for threads. Unfortunately IBM hasn't got anything remotely close to it in its VisualAge C++ for AIX.
-- Julien Pierre http://www.madbrain.com/blog
TotalView is your answer. It's the only thing I've found that really works under multi-threaded multi=processor environments
Dykstra, I hate you stupid Dykstra's algorithm why are you so stupid why, why! You and your foolish data structures, I'll data structure you mom. Haha woo,
I got a c in cs310 cause of Dykstra
printf owns me. Never used something else. Debuggers are for tarts.
Don't like printf ? #ifdef DEBUG it !
:->
First - a disclaimer - I work for Rogue Wave so feel free to do all your own research et al. Just wanted to make you aware of it.
:)
Rogue Wave (www.roguewave.com) has a library that comes with source code called Threads.h++ that works on Linux (as well as Solaris, Windows, HP, AIX etc) designed for making threading programs easier. I invite you to look at the product but here's how this is related:
First, the API is totally portable and more C++ like. More importantly, there is a really cool little tracing package included with the product (sometimes traces are a better way to debug threads than pure debugger usage.) And finally, I've read people commenting that most C/C++ bugs are memory related (very true) - the Threads.h++ product includes smart pointer classes (reference counting, etc) to help eliminate that when used properly.
I appoligize for the shameless plug but it did seem related especially considering the platform switch as an option. Flame me if you must, just hope I helped.
~Tim
Tim T.
What is necessary? 10 xterms running tail -f, 1 running the editor, 1 the debugger, 1 running xman and 1 running netscape 4.7?
What's the difference between this and running a few command shells, Visual Studio, MSDN, and Internet Explorer?
#ifdef DEBUG
printf("blah %d",var);
#endif
then when you get annoyed with the extra info
#undef DEBUG
The wise programmer uses the right tool for the right job, and an interactive debugger is usually the wrong tool.
Well said. There are definitely times and places for debuggers (particularly assembly language programming). And the other poster had it 180 degrees backward -- it's the newbies who have grown up with debuggers and IDEs. It's the old hands that have learned not to let it suck away endless amounts of time.
And that's the point -- it's not about how easy it is to develop something, it's about how much time does it take? There is no question that debuggers make things easier, but I've found that it makes development far, far slower.
--
Sometimes it's best to just let stupid people be stupid.
If you guys are far along in the development process with c++ already, or if you have limited experience with Java and Java threading, you'll just run into worse threading problems there by trying to fit the pthread paradigm into the Java platform., which checks java programs for basic errors including many types of thread deadlocks. A lot of commercial Unix vendors have similar tools for C/pthread programs, but I haven't found one for Linux.
u gging.html) and do remote, multi-platform debugging. Also, MetaMata (http://www.metamata.com) has a very good debugger that includes some advanced threading features even in their low-end versions.
That said, I do a lot of multithreaded programming in both Java and C++, and I would give damn near anything to switch completely over to Java. The threading design makes it FAR easier to build correct programs from the ground up without using a debugger. We've known for 15 years that monitors (Java's synchronized blocks, basically) were a better way to write multithreaded programs, and it's embarassing that it took this long to get the idea into a mainstream language.
Also, you should check out jlint (http://www.ispras.ru/~knizhnik/jlint/ReadMe.htm)
As for actual debuggers, JBuilder 4 is quite good. I've only used the foundation (free) version, but the enterprise edition can also dynamically detect stalls, deadlocks, and race conditions (see http://www.borland.com/jbuilder/jb4/feamatrix/deb
On the down side, you'll have to pick a good implementation of the JDK on Linux and stick by it, as the different JVMs tend to have threading incompatabilities. IBM's are quite good, and they support native threads very well. In Sun's 1.2 implementation, at least, native threads were an undocumented feature (with green threads only officially supported).
Hope this helps!
--JRZ
Just because someone is using a debugger doesn't mean they don't know what's going on. On the contrary, I would argue that anybody continuously adding printf's everywhere really doesn't know what's going on because they're shooting in the dark hoping for the miracle output that will point them to the error in their code.
Using a debugger requires that you know your code fairly well, although it can also help you learn code you haven't seen before. If you don't know how your code works, how are you going to know where to set your breakpoints and watches appropriately?
Using a debugger to track down a problem should only take a few minutes. Anybody spending hours just isn't using the tool properly. Debugging takes skill and experience. I know newbies who get into the debugger and step through everything. That's nuts and a waste of time. Learn to use the tool. It'll give you much more insight into how your code works and how it interacts with other libraries.
My typical debugging sessions goes like this:
- Something incorrect happens in the program.
- I look at my diagnostic output. Usually the culprit is pretty obious and I fix it.
- If it's not, I take a look at the code itself to see if I missed something. If I notice a problem in the code, I fix it.
- If it's not obvious, I fire up the debugger and set appropriate breakpoints and/or watches based on my knowledge of the problem and the code. Usually I'll see the problem on the first pass and I fix the code within minutes.
Enough of the doctor analogiesDebuggers aren't a a crutch. They are a tool to use...and a productive one at that. You are correct in your statement "The wise programmer uses the right too for the right job." Usually it's a combination of tools.
- Rick Alther
Now that I'm in that habit of using gdb early and often, I reckon that my code has fewer bugs to begin with, although obviously I don't claim to write bug-free code.
Having said that, I use gdb and printf() (actually, it's been syslog() for me lately, as I've been writing some daemon code) about 60%/40% for debugging. It's always good to use whatever tools you have at your disposal.
I'm not going to make any bad analogies, however. Nothing hampers understanding like a bad analogy.
--
You're a suburbanite.
I would suggest you check out Be's platform and GUI debugger. As the OS is natively highly multi-threaded, they must have good tools to do the debugging. They also use gcc, so your stuff should be pretty portable to BeOS.
I think you would benefit by considering how a master programmer (which I don't claim to be) might beneficially use printf's in preference to interactive debugging.
Here's an example that comes to mind. Suppose you have a large software system that depends on some sort of numerical computation library. It may be useful to trace all calls across the library interface, printing the trace in an easy-to-parse manner (perhaps to be checked by another program). It's relatively easy to do this with printf's; it's possible to do with a debugger (by maintaining a debugger script with the required breakpoints, along with breakpoint scripts), but fraught with problems. Maintaining complex debugger scripts is a nightmare.
Also, printf's, or some similar logging mechanism, work just about everywhere. There are plenty of situations where invoking a debugger isn't practical or even possible.
Using a debugger to track down a problem should only take a few minutes. Anybody spending hours just isn't using the tool properly.
Maybe you've never run across a really subtle bug in a large software system before. By subtle, I'm talking about a bug which might occur hours (in execution time) before any externally visible aberrant behavior, during which time it has caused a cascading series of minor errors which finally lead to (say) a crash. You could spend your entire life trying to diagnose a problem like this with a debugger.
I believe that interactive debuggers are very useful in their place, and that a good programmer must master them, but also that they should be used sparingly, and not as a substitute for really thinking about the problem at hand. Personally, I find myself spending less and less time inside a debugger as the years pass, and more and more time making sure my code is constructed so that those kinds of problems don't occur in the first place.
"Not an actor, but he plays one on TV."
Hi, We develop a fairly large multi-threaded application on various Unix platforms and Solaris Forte (formerly Workshop comes out tops). Multi-threaded debugging is very painful on Linux. strace doesn't work on multithreaded programs (this is the single biggest problem for MT debugging on Linux). I found insight, a tck/tk front end to gdb (available at sourceware.cygnus.com) invaluable during some development on Linux/Alpha. I used a cvs version of it and it is coming along pretty well. IMHO a GUI is a must for debugging, especially for MT apps. (What constitutes a GUI is open to debate, but for me emacs with gdb is good enough). Solaris Workshop debugger on the other hand is excellent to use, though it has it's problems. For example, you can control only one thread at a time. This is true for all multi-threaded debuggers I've come across. I've also encountered occasional crashes. But even with these warts it comes out tops. I've worked on MT applications for over 4 years on many platforms (Solaris, Tru64, Linux, Linux Alpha etc) and found Solaris the best supported among these (despite all the thread related bugs that others have mentioned here). If I can get strace working on Linux and have a stable release of insight on Linux, I'll be happy on Linux too.
I'm a member of the customer services department at Etnus, so I'm interested in helping you get TotalView working. You didn't give any information about your system, so it's a little hard to troubleshoot your problem. But I'll take a stab anyway.
Since you are posting to Slashdot I'd say that the likelihood is good that you are on top of the latest releases of all things related to Linux, and that you are comfortable dealing with libraries.
The latest versions of libbfd changed in a way that breaks functionality that TotalView relies on. If this is indeed what is happening in your case then the fix is simple. The fix is just to include the library that TotalView expects to find. Grab libbfd-2.9.5.0.22.so out of binutils-2.9.5.0.22-6.
You can install it in /usr/lib/libbfd-2.9.5.0.22.so or you can place it
anywhere you want and just update the symbolic link libbfd-2.9.1.0.15.so.0 in /usr/toolworks/totalview.4.1.0-2/linux-x86/shlib/
to point at your libbfd-2.9.5.0.22.so.
Our release package will be changed to include a working libbfd sometime in the next week.
Please feel free to contact us at support@etnus.com about this or any other problem.
binutils and libbfd are licensed under the GPL.
Cheers,
Chris Gottbrath, Etnus
The way I approach thread debugging is: 1)Plenty of diagnostics printf's, but via a macro. All diagnostics go via a macro that allows debug levels to be set, and also prepends line,file,time and thread to the diagnostics string. You can then analyse this output on a thread by thread basis, global timeline basis, etc. 2) Memory Tracking. Tracking memory leaks is reasonably easy - use a macro (say d_malloc) that calls a wrapper function to malloc that allocations a structure containing file, line, time & thread info for the point that the memory is allocated. You can then correlate that with you diagnostics to track leaks. 3) Resource Tracking. Use a similar approach to mutex / semaphore locks to spot points where you're not releasing resources. 4) If all that fails... GDB5 supports threads, and a recent version of Kdbg is a reasonable interface to it.
+++ BASELINE REALITY FAILURE+++ +++ PLEASE REBOOT UNIVERSE +++
Do you really honestly need to multithread or can you replace all or most threads by a "select" statement?
Then most of your strangest of bugs will vanish like the mist in the morn, and gdb will be a magic bullet against the rest.
Inferno's Limbo uses just that. And has a nice GUI multithreaded debugger. Runs on all platforms. And the source is publicly available.
Anssi Porttikivi / app@iki.fi