How do Linux Threads Compare to NT Threads?

BurnMage asks: "I run a small network at a company that does heavy web and graphic work, as well as digital video production. The DV stuff is absolutely wonderful, cutting edge really nice hardware that costs thousands of dollars... and is only supported on the Windows platform. My personal experience with the machines has really been a constant nightmare until recently; we have gotten things running relatively smoothly. We use the DVRex and RexFX from Canopus, beautiful stuff and great tech support... but a few small messages asking why not port to Linux had a couple people muttering that threads in Linux 'aren't properly implemented'. I have also heard the same about NT. What's the story? I want a clear answer to post on their site."

Comments

``Not properly implemented.''

[Kaz Kylheku]

Linux threads don't quite conform to the POSIX standard because of the underlying model that is markedly different from systems like Solaris or Digital UNIX.

There are some differences in signal handling, and also notable differences in file locking. According to POSIX, a lock set by fcntl() is owned by a process, so if one thread places a lock, and another thread in the same process places a conflicting lock, the lock region should simply be extended In Linux, the second thread will block because it is de-facto a second process and the flock() code doesn't take into account that the process is actually part of a ``cluster'' of threads.

There also have been issues in the threading library as recently as glibc 2.1.1. I submitted a patch that fixed a serious crash in the case that the process creates too many threads (which could happen even if the number of threads is low, due to ulimit).

Other than some stability issues and incompatibilities with POSIX, I would not conclude that LinuxThreads is improperly implemented. The API is there and works. Things like mutexes, condition variables and thread-specific keys are nicely implemented. I particularly like the wait-free queuing algorithm used to resolve contentions for locks, based on an atomic compare-exchange sequence.

As for NT threads: there are some nasty issues that require painful workarounds. I could write volumes about the deficiencies, but here are some highlights:

1. No cancellation mechanism! In NT, if you want to shut down a thread in an orderly manner, you have to figure out what object(s) it is blocking on and disentangle it. In POSIX threads, you simply cancel the thread and when it hits the next cancellation point, it executes registered cleanup handlers and terminates. In NT, there are no cleanup handlers; a thread killed by TerminateThread just leaks all of its local resources.

2. Thread local storage is a joke! In POSIX, every thread specific key lets you register a cleanup handler. When a thread terminates, it executes cleanup handlers for all thread specific keys that have handlers. The only way I know of to do this in NT is to put your stuff into a DLL, so that you can hook the clean up to the THREAD_DETACH command passed to DllMain()---under this solution you have to write your own thread specific storage subsystem that supports destructors.

3. No condition variables! NT's events are poor substitutes for condition variables. There is no way to create an event that can wake up all threads waiting on it AND which can wake up just a single thread. With POSIX conditions, you have pthread_cond_signal() and pthread_cond_broadcast(). Furthermore, pthread_cond_wait() is a cancellation point (recall that NT has no cancellation). Events are horrible, stateful objects. Conditions are stateless, making it easier to verify code to be free of race conditions and other problems.

4. Poor exclusion mechanisms! Critical sections are the rough equivalent of the POSIX mutex. But they are fraught with problems. For one thing, each critical section has its own internal event that is used to resolve contention. This event is allocated when needed. Thus in low memory conditions, EnterCriticalSection() can throw a win32 exception due to failure to allocate the event handle! InitializeCriticalSection() returns nothing. (But there is a little known hack in NT only: if you pass a spin count with its high order bit set to InitializeCriticalSectionAndSpincount() then it will pre-allocate the event). Preallocated or not, each critical section potentially consumes non-swappable kernel memory!

By contrast, Linux mutexes do not consume any kernel resources. Nothing has to ever be allocated other than the space for the pthread_mutex_t object itself. Contention is resolved by suspending, and each thread comes with a way to suspend and resume so no semaphore-like object needs to be allocated.

Another problem with NT critical sections is that there is no way to atomically release one of these and wait on some object. POSIX has pthread_cond_wait() which releases a mutex and waits on a condition in a way that avoids losing the wakeup signalled from some other thread that subsequently acquires the mutex.

My experiences with Win32 programming under NT have led me to implement everything myself that is missing in the low level API: thread specific storage, mutexes and condition variables, cancellation, etc.

Looks like your co-workers read Russinovich

[tilly]

Linux threads are fine. The following letter may be a good thing to point to. And here is a follow-up.

The basic story is that Linux, unlike most operating systems, does not have a hard distinction between a thread and a process. Instead it has the idea of a context of execution, and with the clone() call a context of execution can reproduce a copy of itself, and decide how much is copied. The last is important, clone() can be anything from spawning another thread to a traditional fork.

This threading model is somewhat different than what most operating systems provide, but it is quite sufficient to provide a full POSIX thread implementation with very good speed on a context-switch between threads or processes. (In fact Linux does a faster context switch between processes than most operating systems do switches between threads on the same hardware.)

By contrast NT has a different problem. NT has tremendous difficulty with processes. The time to create a process is abysmal. Context switches are not cheap. And once you start paging, the paging algorithm has beeen found to literally worse than straight chance!

However NT has a pretty good time on context switches between threads and (on paper) some nice specs for working with them. But NT's threading model is somewhat different from the POSIX model and anyone who is experienced with Microsoft knows that what is on paper and what really happens are not always the same...

Cheers,
Ben Tilly