Good Books On Programming With Threads?

uneek writes "I have been programming for several years now in a variety of languages, most recently C#, Java, and Python. I have never had to use threads for a solution in the past. Recently I have been incorporating them more in my solutions for clients. I understand the theory behind them. However I am looking for a good book on programming threads from an applied point of view. I am looking for one or more texts that provide thorough coverage and provide meaningful exercises. Anyone have any ideas?"

Comments

PThreads & Java Threads

[eldavojohn]

However I am looking for a good book on programming threads from an applied point of view. I am looking for one or more texts that provide thorough coverage and provide meaningful exercises. Anyone have any ideas?

I went through grad school not too long ago for Computer Science (disclaimer: it was the kind of computer science degree that doesn't focus on hardware so I might not be the best expert on this). Anyway there were two books for the class.

One dealt with coding regular old C on a plain jain Unix machine and method of (I believe there are others) doing multithreaded in that environment is PThreads (or the super short overview). The book we used is the Addison Wesley book (ISBN 0-201-63392-2). It was informative and comprehensive ... wasn't concentrated specifically on applications like you ask but very good reference. Also, I think there are a lot of good books free online in respect to that topic.

As for Java, there was an O'Reilly book (there's probably a new version out for Java 6) that was pretty good. Not as great of a reference but better on applications of threads in Java. Although, as far as introductory material, I personally learned it all from java.sun.com. Although I can't vouch for whether this is an applied approach or not, I would suggest the concurrency tutorial and a good book on Java Patterns or even a design pattern wiki.

I've never done concurrent programming in C# or Python so I do not know first hand what is best. I do know that erlang has been fun to mess around with in my spare time though!

Recently I have been incorporating them more in my solutions for clients.

Most important rule of thumb of multi-threaded programming is to avoid it if possible. Maybe hardware (multi-core) will change that, maybe you feel the scheduler can't do its job as well as you can and maybe you feel it's more intuitive. But, often is the case, that you're just adding more complexity to your code resulting in more difficult bugs and harder maintenance for others. Keep it simple.

Re:PThreads & Java Threads

[Anonymous Coward]

The Addison-Wesley book mentioned by the parent is "Programming with POSIX Threads" by David R. Butenhof. It's what I used when I needed to get up to speed on p-threads in a hurry - clear and easy to follow. P-threads are what's in Darwin, (and so BSD) Linux, and I'm guessing based on POSIX compliance, just about every commercial flavor of UNIX. (Presmuably, OpenServer uses fraying threads)

Re:PThreads & Java Threads

[Anonymous Coward]

Most important rule of thumb of multi-threaded programming is to avoid it if possible. Maybe hardware (multi-core) will change that, maybe you feel the scheduler can't do its job as well as you can and maybe you feel it's more intuitive. But, often is the case, that you're just adding more complexity to your code resulting in more difficult bugs and harder maintenance for others. Keep it simple.

Man, I have to disagree with you. That kind of dinosaur thinking will hold back progress. Multi-core is the future and multi-threaded apps are exactly what's needed to fully utilize its potential. I'm sorry if its too hard for you to debug but its just the way the cookie crumbles.

Re:PThreads & Java Threads

[ByOhTek]

for the morbidly curious, there's even a pthreads library for windows. LGPLed

http://sourceware.org/pthreads-win32/

Re:PThreads & Java Threads

[zolaar]

Erm, the tenets of programming usually involve the general concept of "Eliminate the unnecessary." Therefore, the parent is correct: if multi-threaded processing is unnecessary, avoid it.

What you meant to add to the dicussion is the corollary: If it is unavoidable, use it wisely.

Re:PThreads & Java Threads

[fitten]

I'm pretty sure that stuff was some rumor that came out before Barcelona was released about how the Barcelona core was going to 'destroy' Core2 (basically a load of wild and crazy speculation).

There's already some parallelisation of sequential code in all modern processors (out of order execution) that works well because it has fairly narrow focus on the instruction stream window. Going out further would be a much, much larger problem. Looking for parallelism in larger windows of the instruction stream, to the point of trying to execute whole 'subroutines' in parallel would require vast analysis (and probably not be very viable anyway as most routines are serial with respect to other subroutines.

So far, such parallelism has only been taken advantage of because the programmers put either explicit threading into the source code (pthreads and other threading APIs) or at least put hints into the code (OpenMP). This isn't a new problem... it's been around for many decades now and, so far, there haven't been really any success in automated tools to thread code. Languages like Erlang and functional languages take advantage of the fact that some language semantics allows parallelism.

Re:PThreads & Java Threads

[discord5]

Multi-core is the future and multi-threaded apps are exactly what's needed to fully utilize its potential.

For each application you name that is benefited by threading, someone else will be able to name one that isn't. Some processes simply are not parallelizable in a meaningful way, where meaningful is defined as in speed of execution not as in the interactive extravaganza of "looky how I can clicky the button while it's still doing hard maths".

There's a good bit of reading about the subject, although much of it is boring and is often difficult to apply to real-world situations. Amdahl's law in many situations can predict if it's worth bothering with multithreading (or other forms of parallelizing) quite easily.

A tool like cat or grep has no benefit of being threaded since it's a simple sequential task. Suppose you were to multithread "cat" into one thread that reads from disk, and another that displays a line of text on the screen. Thread 1 will spend most of its time waiting for I/O, and thread 2 will spend most of its time waiting for thread 1 to pass data. Except now, your multithreaded cat has a somewhat complicated synchronization mechanism on top of it that makes it a bit harder to debug and probably eats some extra cycles as well.

While the previous example is overly simple, there are plenty of tasks that are a lot more complicated but simply have no benefit of being threaded, because they spend more time waiting for I/O than actually calculating or because the algorithm is simply not worth parallelizing because there is no benefit in speed.

Another example would be an application divided in 3 steps. Step A and B can be executed at the same time independently of each other, while step C depends on step A and B. Both step A and B can be written to use two threads, and if they'd use two threads they'd run in half the time of their non-threaded equivalent. On a dual core machine (or 2 CPU machine) running step A multi-threaded and then step B multi-threaded takes 1 hour. In the other case, running step A and at the same time (on the other core/CPU) running step B single threaded also takes 1 hour. At this point you gain nothing by threading. Of course here I assume that I/O by both processes at the same time doesn't create some sort of delay. But if you're working with large enough data sets (more than you can keep in memory) this becomes less and less of an issue since the I/O overhead will already be there anyway.

If you add to that the fact that threading (especially synchronization) is a subject that is not well understood by everyone (in the "find me out of 200 programmers fresh from school, 10 who can write a program that benefits from multi-threading and actually works" sense), threading suddenly becomes less appealing if there aren't any clear benefits for the application you're working on.

The reason I mention that last part is that because so many schools give kids the "make two threads count to 100 then exit" exercise but fail completely at getting the message across of the fact that most of the time the threads actually need to synchronize with each other. They'll give this long lecture about the dining philosophers problem without actually SHOWING them what that means.

In conclusion: it depends on a lot of factors (size of your dataset, how well your algorithm can be split up in parallel tasks, ...) if your process benefits from threading or not, and you should evaluate at design time using Amdahl's law if there's an advantage or not. If your results in a multithreaded environment are only marginally better, the economical factor of cost of development time suddenly weighs in very heavily.

Having said that: if you're a programmer, have fun with threads at least once. Write something silly in your spare time, it can be an amazing amount of fun and often offers an interesting way of approaching future problems.

Re:PThreads & Java Threads

[greenbird]

Erm, the tenets of programming usually involve the general concept of "Eliminate the unnecessary." Therefore, the parent is correct: if multi-threaded processing is unnecessary, avoid it.

Although unnecessary, threading usually simplifies a program rather than adding complexity. The only caveat is that you understand threading. In my experience I've used threading to greatly reduce the size and complexity of solutions that either were or could have been implemented without them.

Re:PThreads & Java Threads

[ELProphet]

Most important rule of thumb of multi-threaded programming is to avoid it if possible. Maybe hardware (multi-core) will change that, maybe you feel the scheduler can't do its job as well as you can and maybe you feel it's more intuitive. But, often is the case, that you're just adding more complexity to your code resulting in more difficult bugs and harder maintenance for others. Keep it simple.

I'm going to have to disagree with you on this one. Especially in Java client side rich GUI apps, background threads are one of the most useful components to ensure a responsive interface when dealing with asynchronous requests. They really only need two and a half pieces to implement them easily and efficiently. The first component is the request itself, either a subclass of java.lang.Runnable or javax.swing.SwingWorker. The second is a callback handler. The half piece is the shared data structure, and it's only a half piece because you'll want to use the synchronized collections wrapper to get a (you guessed it) synchronized collection.

• http://http//www.google.com/search?q=SwingWorker
• http://http//java.sun.com/javase/6/docs/api/java/util/Collections.html#synchronizedList(java.util.List)

Brushing up on those pieces will give you the background you need to not block the UI whenever something needs to happen. Threads aren't hard, they just take a little thought.

Re:PThreads & Java Threads

[ShakaUVM]

My Master's Degree was in High Performance Computing from UC San Diego, and I taught parallel processing.

Yes, you're right that most new programmers out of college will screw up (and screw up badly) if they try to write a multithreaded application. Learning to write parallel applications requires more mind-bending mental gymnastics than, say, when you first learned to write recursive applications. That said, once you get a solid understanding of how safe parallel code should look like, and how it should work, it's fairly trivial to write code that works, and doesn't deadlock. From my experience, it takes about 3 to 6 months of pounding on parallel code to reach that state.

While it's not a trivial amount of time, given the importance parallel code has (and will increasingly have in the future), I don't think it's too great a hurdle to ask for people to learn this stuff. All talk about multi-core programming always boils down to "Well, we'll never find enough programmers who are able to write multi-threaded apps." Well... why?

I think it would be in the best interests of Intel and AMD to sponsor online college classes teaching how to do parallel coding. People aren't buying the new chips since code can't take advantage of it -- if they flip it around and make every program able to multithread (that could benefit from multithreading, as you point out, Amdahl's Law) then demand for their chips would surge, and they'd make the money back in billions.

Nobody mentioned needles

[Corpuscavernosa]

Working with threads! Get it! BA-ZING! Sorry. I'm not a programmer. Clearly.

real world haskell

[j1m 5n0w]

Probably not what you're looking for, but Real World Haskell is soon to be released and has chapters on concurrent and multicore programming and software transactional memory. Even if you're not interested in Haskell per se, STM is kind of an interesting idea.

Language/Environment specific

[MikeRT]

Pthreads, Java threads and .NET threads are implemented differently. If you need a good Java book, just pick up one of the "Core Java" books that covers threading in one of its chapters since Java threads aren't that complicated. That said, with Java applications (the platform I know pretty well), if you're doing "enterprise" development it's best to avoid using them and let the application server do its black magic for you.

Re:Language/Environment specific

[discord5]

if you're doing "enterprise" development it's best to avoid using them and let the application server do its black magic for you

Finally, confirmation!!!! I always suspected all those acronyms to be some form of arcane hex.

Re:Language/Environment specific

[Cyberax]

On the contrary, Pthreads, Java threads and .NET threads are mostly the same thing in different packages.

There are _really_ different ways to implement multithreading: fork-join model, pi-calculus, STM, message-passing model, etc.

Re:Language/Environment specific

[TheRaven64]

There are _really_ different ways to implement multithreading: fork-join model, pi-calculus, STM, message-passing model, etc.

No, there are different ways of implementing concurrency. Threading, in particular, means shared-memory concurrency with a private control stack. Pi-calculus, STM, Linda and CSP are all examples of other models for concurrency, not of multithreading. They differ in many respects (although pi-calculus and CSP have a lot in common), but share one feature - they are all easier to reason about (and therefore to debug) than multithreading. The only valid use for multithreading is to provide an efficient implementation of one of the other models.

Free eBook on Threading in C#

[Deffexor]

I'm still getting the hang of Threading in C# myself, but I found this eBook immensely helpful in getting me understand some of the difficult issues such as Thread Safety, Cross-threading issues, Race Conditions, and Event-Delegate pairs.

http://www.albahari.com/threading/

Concurrent Programming in Java

[progressnerd]

Concurrent Programming in Java is more or less *the* book on good practices for multi-threaded programming for Java, with many lessons that apply to other languages as well.

Re:Concurrent Programming in Java

[K.B.Zod]

I recommend Java Concurrency in Practice as well. It's an updated, in-depth look at Java threads. Doug Lea, author of Concurrent Programming in Java, is a co-author of the newer book. A great read.

Re:Concurrent Programming in Java

[Anonymous Coward]

Don't forget Brian Goetz's "Java Concurrency In Practice", which covers the changes they made to the JVM memory model in Java 5. Also check out the Java Theory and Practice section in IBM's developerworks site.

Obligatory serious needle reply

[davidwr]

The Jacquard Loom involves programming of a sort, albeit without branching or computations. In that sense it's more like a translator, translating punches into patterns. Sort of like printf for the clothing industry.

Java Concurrency in Practice

[mckayc]

I highly recommend this book if you are doing threads or any sort of concurrent programming in Java. It's written by the guys who designed Java's concurrency features.

Two books

[grindcorefan]

First: Programming Erlang: Software for a Concurrent World
by Joe Armstrong
http://www.pragprog.com/titles/jaerlang/programming-erlang

The Erlang programming language is well suited to develop concurrent programs with.

The second book I'd recommend is
Distributed Systems: Principles and Paradigms, 2/E
by Andrew S. Tanenbaum
http://www.pearsonhighered.com/educator/academic/product/0,,0132392275,00%2Ben-USS_01DBC.html

Not specific to any programming language, but a very good introduction to the concepts and methods used developing distributed systems, as all multi-threaded programs are.

not covered in books on threads

[bugi]

The thread model has some fundamental problems, but since they seem here to stay there are some things you should keep in mind, nicely summarized in this article(pdf).

Article also available in html if you click on the first computer.org link from google. Hmm, why does it work from google and not from slashot?

Re:not covered in books on threads

[TheRaven64]

Threads are a very good tool for building tools for building concurrent applications. They are not, themselves, a good tool for building concurrent applications and should not be treated as such. If you are building an application, stay away from using threads directly, and instead use a high-level concurrency API. If you are building a concurrency API, then by all means use threads (I have done, and so did the Erlang guys), but you probably don't want to be doing this just after reading a book on threads. In short, if you are the kind of person who needs a book on threads to understand them, you probably are not the kind of person who can safely use threads. You would be better off picking up a book on operating systems or distributed systems theory and reading the chapters on concurrency. This will give you a deeper understanding of the problems of concurrency and give you a much, much better overview of the tools which can be used to solve it (threads, message passing, transactional memory, and so on), and how they are implemented.

Multithreading Applications in Win32

[GogglesPisano]

Here's one I found useful: Multithreading Applications in Win32 by Jim Beveridge and Robert Wiener. It's a little dated (no coverage of .NET, for example - it's more focused on C/C++), but it still provides a good introduction to threading and synchronization on Windows.

If you can find an inexpensive used copy, it's worth a read.

oldie but a goodie

[fred fleenblat]

Some background in parallelism is helpful for mastering threads.
I learned from this book:

http://www.lindaspaces.com/book/

C-linda never caught on, but it's not hard to read the examples and apply them to pthreads, java, MPI or whatever framework you're using.

I haven't found a decent book, but...

[EWIPlayer]

Herb Sutter has been doing a lot of work on this stuff over the last 10 years and his blog is full of stuff on what you should do... it's not too nitty gritty in terms of languages and stuff, but it's very informative in terms of understanding the issues and what not. Check out http://herbsutter.wordpress.com/.

Some rules of thumb that I've found useful:

• Hide mutexes and locks at (nearly) all costs. If you have a queue class, for example, that has a locking push() function, and someone needs to lock for a series of pushes, don't expose the lock to let them lock things for the series of pushes, but provide a push function that takes a list of items instead. Keep thinking of ways to hide your locking strategies. If your class is deadlock-free then you can be reasonably sure (I've always said "reasonably" but I've never seen it not work either) that you'll never see a deadlock in real life either. Race conditions are a different story, however.
• Trying to figure out a solution where you never have to think about the concurrency of things is a scary place to go... Have a logical concurrent model instead. For example, if you work with user's and user's get events, rather than just letting them process any number of events in parallel, it may be reasonable to sequence events per-user and let the users run in parallel.
• If you do have to expose locks, use a locking hierarchy. Herb shows this here: http://herbsutter.wordpress.com/2007/12/11/effective-concurrency-use-lock-hierarchies-to-avoid-deadlock/
• Avoid any concept of being impolite among your threads (i.e. forced interrupts or kills). Be polite. Herb has this here: http://herbsutter.wordpress.com/2008/04/10/effective-concurrency-interrupt-politely/
• Locking sucks, but it's necessary. If you think you can get away without having to lock in a dubious situation, you're probably wrong.
• Unit test, unit test, unit test. If your classes hide all of your locks, then unit tests cover a ton of cases.

I believe that following strict OO guidelines is even more important when dealing with concurrency than when dealing with general ideas in software... and let's face it, it's extremely important even when not dealing with concurrency :)

Re:I haven't found a decent book, but...

[TheRaven64]

Locking sucks, but it's necessary. If you think you can get away without having to lock in a dubious situation, you're probably wrong.

There are lots of good, reusable, lockless data structures around if you know where to look. Keir Fraser's PhD thesis contains a really nice lockless ring buffer design (which he implemented for Xen) and several other useful things (including a transactional list and some other shiny stuff). If you have implementations of these in a library somewhere, then you can often get away without locks. There is one rule you should always obey when writing parallel code though:

No data may be aliased and mutable.

As long as you remember that, then it's easy to write concurrent code. In Erlang, for example, this is enforced for you, since the only mutable data structure is the process dictionary, which is not ever shared. This rule actually applies in a lot of serial code too, but in parallel code failure to apply it is the cause of a great many bugs.

What the hell?

[QuoteMstr]

You don't need a book about threaded programming in Python.

You need two books: one about Python, one about threads. Concepts are universal and can be applied across as many languages as you want. It's like saying you need to re-take Calculus because you just learned French!

Re:Use processes whenever possible

[QuoteMstr]

No. IPC is dirt cheap. Take X11 for example -- that's perfectly usable using plain old named pipes, even without Xshm.

Most of the time, a threaded GUI program will want to use threads in order to perform some operation or another "in the background" while the UI remains responsive. If this operation has well-defined inputs and outputs, why not write it as a separate program? Communication overhead is going to be low.

Re:Python

[Vornzog]

Howsabout books or sites on Python threaded programming? I'm going to be working on a project in a short while which will require the use of GTK and twisted together in a sort of network scanner system with asynchronous results.

As much as I love Python, it does have some weak points, and threading is one of them. From the python documentation:

The Python interpreter is not fully thread safe. In order to support multi-threaded Python programs, there's a global lock that must be held by the current thread before it can safely access Python objects.

Threading is there, and I'm sure some decent documentation exists somewhere. But the GIL (global interpreter lock) generally means that there are better ways to approach the problem in python, i.e. processes instead of threads.

It's a point of contention in the community, and the GVR-BDFL point of view is that any attempt to remove it makes Python a lot slower, so he won't.

While I don't use twisted, I am given to understand that it does most of its asynchronous stuff using callbacks - you may be able to leave most of the concurrency to it and avoid the process all together...

The answer is obvious

[paulyt10]

Google. Finds answers much faster than reading.

Re:Use processes whenever possible

[PotatoFarmer]

Communication overhead may be low, but it's also more likely to be tied to the underlying platform. Why rely on an external provider when you get it free in the same process space?
There's also the issue of process management. When the other end of that named pipe breaks, what happens to that separate process? Is it really dead? If it's still alive, how do you kill it cleanly?
I'm not saying separate processes are bad, I'm saying that they're appropriate for certain problems, just like threaded applications are appropriate for other problems. Picking your technology and then trying to mold your solution to fit it is backward.

Re:Python

[Anonymous Coward]

There is a new multiprocessing module 2.6 (and eventually 3.) that does allow for true multiprocessing. It uses an interface similar to but not exactly like threads, but actually is forking off new processes and using pipes to communicate in the background. It is possible to use shared memory between processes to give you all the benefits (and pitfalls) of threads.
    On systems where spawning processes is cheap (like Linux) I think it could be pretty useful. While certainly not as low-overhead as pthreads, it should finally allow Python to utilize multicore CPUs if you code your program the right way.

the little book

[scoopr]

I found Little Book of Semaphores a good read.

Avoiding threads.....

[refactored]

1. You can avoid threads all you like... but several libraries / toolkits automagically spin threads for you. eg. If you using java graphical stuff odds on it has it's own thread whirring away doing stuff.
2. Threads have subtle and noxious interactions with processes. Say "man pthread_atfork" sometime to see what I mean.
3. ISRs/Timer/alarm/signal callbacks are effectively another thread context. ie. Most largish systems that claim to be "single threaded" aren't.

I can suggest three books...

[Troposphere]

I can suggest three books... But you've got to be able to read them all at the same time ;-)

Re:Use threads

[goose-incarnated]

Say goodbye to IPC and all its problems, say hello to shared memory space and all its problems ;-)

Here are two good ones:

[Jane Q. Public]

Programming With Card Looms, Jacques de Vaucanson, The King's Press, 1745.

Weaving Technology, Joseph Jacquard, Colonies? What Colonies? Publishing, 1801.

Taming Java Threads

[jed_reynolds]

I took some classes taught by Allen Holub. Very smart guy, and I certainly enjoyed his book.

He provides good solid explanation on functional models for queue design and listener patterns. He also discusses some pitfalls of threads in Java.

http://www.holub.com/training/java.threads.html

http://www.amazon.com/Taming-Java-Threads-Allen-Holub/dp/1893115100

Python doesn't have threads

[Secret Rabbit]

That might seem wrong given that Python lists threading modules, but just look at Python's GIL to know what I mean. As in, no matter what you do, Python will still be running on one core. So, if you just want a performance boost because of a lot of I/O, then threads can get you there. Unfortunately, if you want to take advantage of a multi-core CPU with Python, Python's threads won't get you there. There has actually been a lot of discussion on this topic, but Guido just refuses to do it. The interpreter has no threads and the lib is not thread safe.

If you want to do multi-processing with Python, look at its subprocess module.

Guido's blog post on the GIL:
http://www.artima.com/weblogs/viewpost.jsp?thread=214235

The FAQ entry on a (fallacious) reason why they won't remove it:
http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-interpreter-lock

Re:Python doesn't have threads

[Secret Rabbit]

Actually, the conclusion is not supported by the reasoning. For those that don't like clicking links, Guido's reason is that there exists a patch the removed the GIL and replaced it with fine grain locks. This failed miserably. BUT, when one thinks about it, this implementation would certainly be doomed to fail for obvious reasons.

When one implements fine grain locks, every time something is accessed, it is locked accessed and released. Clearly, this will impact performance on even a single threaded application. Clearly, this impact will be more and more significant with an increasing number of threads. So, the only thing that can be said by Guido's reasoning is that:

That SPECIFIC IMPLEMENTATION failed to remove the GIL.

Now, if one put everything that's currently global into the specific interpreter, there would be no reason for locks and thus the performance wouldn't suffer. Then each thread could run independently without including (many) locks. Lua has the ability to do this. So, don't tell me that this is impossible WHEN ANOTHER LANGUAGE HAS ALREADY DONE IT.