# DRD: a thread error detector

To use this tool, you must specify `--tool=drd` on the Valgrind command line.

## Background

DRD is a Valgrind tool for detecting errors in multithreaded C and C++ shared-memory programs. The tool works for any program that uses the POSIX threading primitives or that uses threading concepts built on top of the POSIX threading primitives.

### Multithreaded Programming Paradigms

For many applications multithreading is a necessity. There are two reasons why the use of threads may be required:

- To model concurrent activities. Managing the state of one activity per thread can be a great simplification compared to multiplexing the states of multiple activities in a single thread. This is why most server and embedded software is multithreaded.

- To let computations run on multiple CPU cores simultaneously. This is why many High Performance Computing (HPC) applications are multithreaded.

Multithreaded programs can use one or more of the following paradigms. Which paradigm is appropriate a.o. depends on the application type -- modeling concurrent activities versus HPC.

- Locking. Data that is shared between threads may only be accessed after a lock is obtained on the mutex associated with the shared data item. A.o. the POSIX threads library, the Qt library and the Boost.Thread library support this paradigm directly.

- Message passing. No data is shared between threads, but threads exchange data by passing messages to each other. Well known implementations of the message passing paradigm are MPI and CORBA.

- Software Transactional Memory (STM). Data is shared between threads, and shared data is updated via transactions. After each transaction it is verified whether there were conflicting transactions. If there were conflicts, the transaction is aborted, otherwise it is committed. This is a so-called optimistic approach. There is a prototype of the Intel C Compiler (`icc`) available that supports STM. Research is ongoing about the addition of STM support to `gcc`.

- Automatic parallelization. A compiler converts a sequential program into a multithreaded program. The original program may or may not contain parallelization hints. As an example, `gcc` supports OpenMP from version 4.3.0 on. OpenMP is a set of compiler directives which tell a compiler how to parallelize a C, C++ or Fortran program.

DRD supports any combination of multithreaded programming paradigms as long as the implementation of these paradigms is based on the POSIX threads primitives. DRD however does not support programs that use e.g. Linux' futexes directly. Attempts to analyze such programs with DRD will result in false positives.

### POSIX Threads Programming Model

POSIX threads, also known as Pthreads, is the most widely available threading library on Unix systems.

The POSIX threads programming model is based on the following abstractions:

- A shared address space. All threads running within the same process share the same address space. All data, whether shared or not, is identified by its address.

- Regular load and store operations, which allow to read values from or to write values to the memory shared by all threads running in the same process.

- Atomic store and load-modify-store operations. While these are not mentioned in the POSIX threads standard, most microprocessors support atomic memory operations. And some compilers provide direct support for atomic memory operations through built-in functions like e.g. `__sync_fetch_and_add()`.

- Threads. Each thread represents a concurrent activity.

- Synchronization objects and operations on these synchronization objects. The following types of synchronization objects are defined in the POSIX threads standard: mutexes, condition variables, semaphores, reader-writer locks, barriers and spinlocks.

Which source code statements generate which memory accesses depends on the memory model of the programming language being used. There is not yet a definitive memory model for the C and C++ languagues. For a draft memory model, see also document WG21/N2338.

For more information about POSIX threads, see also the Single UNIX Specification version 3, also known as IEEE Std 1003.1.

### Multithreaded Programming Problems

Depending on how multithreading is expressed in a program, one or more of the following problems can be triggered by a multithreaded program:

- Data races. One or more threads access the same memory location without sufficient locking.

- Lock contention. One thread blocks the progress of one or more other threads by holding a lock too long.

- Improper use of the POSIX threads API. The most popular POSIX threads implementation, NPTL, is optimized for speed. The NPTL will not complain on certain errors, e.g. when a mutex is locked in one thread and unlocked in another thread.

- Deadlock. A deadlock occurs when two or more threads wait for each other indefinitely.

- False sharing. If threads that run on different processor cores access different variables located in the same cache line frequently, this will slow down the involved threads a lot due to frequent exchange of cache lines.

Although the likelihood of the occurrence of data races can be reduced by a disciplined programming style, a tool for automatic detection of data races is a necessity when developing multithreaded software. DRD can detect these, as well as lock contention and improper use of the POSIX threads API.

### Data Race Detection by DRD versus Helgrind

Synchronization operations impose an order on interthread memory accesses. This order is also known as the happens-before relationship.

A multithreaded program is data-race free if all interthread memory accesses are ordered by synchronization operations.

A well known way to ensure that a multithreaded program is data-race free is to ensure that a locking discipline is followed. It is e.g. possible to associate a mutex with each shared data item, and to hold a lock on the associated mutex while the shared data is accessed.

All programs that follow a locking discipline are data-race free, but not all data-race free programs follow a locking discipline. There exist multithreaded programs where access to shared data is arbitrated via condition variables, semaphores or barriers. As an example, a certain class of HPC applications consists of a sequence of computation steps separated in time by barriers, and where these barriers are the only means of synchronization.

There exist two different algorithms for verifying the correctness of multithreaded programs at runtime. The so-called Eraser algorithm verifies whether all shared memory accesses follow a consistent locking strategy. And the happens-before data race detectors verify directly whether all interthread memory accesses are ordered by synchronization operations. While the happens-before data race detection algorithm is more complex to implement, and while it is more sensitive to OS scheduling, it is a general approach that works for all classes of multithreaded programs. Furthermore, the happens-before data race detection algorithm does not report any false positives.

DRD is based on the happens-before algorithm, while Helgrind uses a variant of the Eraser algorithm.

## Using DRD

### Command Line Options

The following command-line options are available for controlling the behavior of the DRD tool itself:

`--check-stack-var=<yes|no> [default: no]`

  Controls whether `DRD` reports data races for stack variables. This is disabled by default in order to accelerate data race detection. Most programs do not share stack variables over threads.

`--exclusive-threshold=<n> [default: off]`

  Print an error message if any mutex or writer lock is held longer than the specified time (in milliseconds). This option is intended to allow detection of lock contention.

`--report-signal-unlocked=<yes|no> [default: yes]`

  Whether to report calls to `pthread_cond_signal()` and `pthread_cond_broadcast()`where the mutex associated with the signal via `pthread_cond_wait()` or `pthread_cond_timed_wait()`is not locked at the time the signal is sent. Sending a signal without holding a lock on the associated mutex is a common programming error which can cause subtle race conditions and unpredictable behavior. There exist some uncommon synchronization patterns however where it is safe to send a signal without holding a lock on the associated mutex.

`--segment-merging=<yes|no> [default: yes]`

  Controls segment merging. Segment merging is an algorithm to limit memory usage of the data race detection algorithm. Disabling segment merging may improve the accuracy of the so-called 'other segments' displayed in race reports but can also trigger an out of memory error.

`--shared-threshold=<n> [default: off]`

  Print an error message if a reader lock is held longer than the specified time (in milliseconds). This option is intended to allow detection of lock contention.

`--show-confl-seg=<yes|no> [default: yes]`

  Show conflicting segments in race reports. Since this information can help to find the cause of a data race, this option is enabled by default. Disabling this option makes the output of DRD more compact.

`--show-stack-usage=<yes|no> [default: no]`

  Print stack usage at thread exit time. When there is a large number of threads created in a program it becomes important to limit the amount of virtual memory allocated for thread stacks. This option makes it possible to observe the maximum number of bytes that has been used by the client program for thread stacks. Note: the DRD tool allocates some temporary data on the client thread stack. The space needed for this temporary data is not reported via this option.

`--var-info=<yes|no> [default: no]`

  Display the names of global, static and stack variables when a data race is reported. While this information can be very helpful, by default it is not loaded into memory since for big programs reading in all debug information at once may cause an out of memory error.

The following options are available for monitoring the behavior of the process being analyzed with DRD:

`--trace-addr=<address> [default: none]`

  Trace all load and store activity for the specified address. This option may be specified more than once.

`--trace-barrier=<yes|no> [default: no]`

  Trace all barrier activity.

`--trace-cond=<yes|no> [default: no]`

  Trace all condition variable activity.

`--trace-fork-join=<yes|no> [default: no]`

  Trace all thread creation and all thread termination events.

`--trace-mutex=<yes|no> [default: no]`

  Trace all mutex activity.

`--trace-rwlock=<yes|no> [default: no]`

  Trace all reader-writer lock activity.

`--trace-semaphore=<yes|no> [default: no]`

  Trace all semaphore activity.

### Detected Errors: Data Races

DRD prints a message every time it detects a data race. You should be aware of the following when interpreting DRD's output:

- Every thread is assigned two thread ID's: one thread ID is assigned by the Valgrind core and one thread ID is assigned by DRD. Both thread ID's start at one. Valgrind thread ID's are reused when one thread finishes and another thread is created. DRD does not reuse thread ID's. Thread ID's are displayed e.g. as follows: 2/3, where the first number is Valgrind's thread ID and the second number is the thread ID assigned by DRD.

- The term segment refers to a consecutive sequence of load, store and synchronization operations, all issued by the same thread. A segment always starts and ends at a synchronization operation. Data race analysis is performed between segments instead of between individual load and store operations because of performance reasons.

- There are always at least two memory accesses involved in a data race. Memory accesses involved in a data race are called conflicting memory accesses. DRD prints a report for each memory access that conflicts with a past memory access.

Below you can find an example of a message printed by DRD when it detects a data race:

    $ valgrind --tool=drd --var-info=yes drd/tests/rwlock_race
    ...
    ==9466== Thread 3:
    ==9466== Conflicting load by thread 3/3 at 0x006020b8 size 4
    ==9466==    at 0x400B6C: thread_func (rwlock_race.c:29)
    ==9466==    by 0x4C291DF: vg_thread_wrapper (drd_pthread_intercepts.c:186)
    ==9466==    by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
    ==9466==    by 0x53250CC: clone (in /lib64/libc-2.8.so)
    ==9466== Location 0x6020b8 is 0 bytes inside local var "s_racy"
    ==9466== declared at rwlock_race.c:18, in frame #0 of thread 3
    ==9466== Other segment start (thread 2/2)
    ==9466==    at 0x4C2847D: pthread_rwlock_rdlock* (drd_pthread_intercepts.c:813)
    ==9466==    by 0x400B6B: thread_func (rwlock_race.c:28)
    ==9466==    by 0x4C291DF: vg_thread_wrapper (drd_pthread_intercepts.c:186)
    ==9466==    by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
    ==9466==    by 0x53250CC: clone (in /lib64/libc-2.8.so)
    ==9466== Other segment end (thread 2/2)
    ==9466==    at 0x4C28B54: pthread_rwlock_unlock* (drd_pthread_intercepts.c:912)
    ==9466==    by 0x400B84: thread_func (rwlock_race.c:30)
    ==9466==    by 0x4C291DF: vg_thread_wrapper (drd_pthread_intercepts.c:186)
    ==9466==    by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
    ==9466==    by 0x53250CC: clone (in /lib64/libc-2.8.so)
    ...

The above report has the following meaning:

- The number in the column on the left is the process ID of the process being analyzed by DRD.

- The first line ("Thread 3") tells you Valgrind's thread ID for the thread in which context the data race was detected.

- The next line tells which kind of operation was performed (load or store) and by which thread. Both Valgrind's and DRD's thread ID's are displayed. On the same line the start address and the number of bytes involved in the conflicting access are also displayed.

- Next, the call stack of the conflicting access is displayed. If your program has been compiled with debug information (-g), this call stack will include file names and line numbers. The two bottommost frames in this call stack (`clone` and `start_thread`) show how the NPTL starts a thread. The third frame (`vg_thread_wrapper`) is added by DRD. The fourth frame (`thread_func`) is interesting because it shows the thread entry point, that is the function that has been passed as the third argument to `pthread_create()`.

- Next, the allocation context for the conflicting address is displayed. For static and stack variables the allocation context is only shown when the option `--var-info=yes` has been specified. Otherwise DRD will print `Allocation context: unknown` for such variables.

- A conflicting access involves at least two memory accesses. For one of these accesses an exact call stack is displayed, and for the other accesses an approximate call stack is displayed, namely the start and the end of the segments of the other accesses. This information can be interpreted as follows:

  1. Start at the bottom of both call stacks, and count the number stack frames with identical function name, file name and line number. In the above example the three bottommost frames are identical (`clone`, `start_thread` and `vg_thread_wrapper`).

  2. The next higher stack frame in both call stacks now tells you between in which source code region the other memory access happened. The above output tells that the other memory access involved in the data race happened between source code lines 28 and 30 in file `rwlock_race.c`.

### Detected Errors: Lock Contention

Threads should be able to make progress without being blocked by other threads. Unfortunately this is not always true. Sometimes a thread has to wait until a mutex or reader-writer lock is unlocked by another thread. This is called lock contention. The more granular the locks are, the less likely lock contention will occur. The most unfortunate situation occurs when I/O is performed while a lock is held.

Lock contention causes delays and hence should be avoided. The two command line options `--exclusive-threshold=<n>` and `--shared-threshold=<n>` make it possible to detect lock contention by making DRD report any lock that is held longer than the specified threshold. An example:

    $ valgrind --tool=drd --exclusive-threshold=10 drd/tests/hold_lock -i 500
    ...
    ==10668== Acquired at:
    ==10668==    at 0x4C267C8: pthread_mutex_lock (drd_pthread_intercepts.c:395)
    ==10668==    by 0x400D92: main (hold_lock.c:51)
    ==10668== Lock on mutex 0x7fefffd50 was held during 503 ms (threshold: 10 ms).
    ==10668==    at 0x4C26ADA: pthread_mutex_unlock (drd_pthread_intercepts.c:441)
    ==10668==    by 0x400DB5: main (hold_lock.c:55)
    ...

The `hold_lock` test program holds a lock as long as specified by the `-i` (interval) argument. The DRD output reports that the lock acquired at line 51 in source file `hold_lock.c` and released at line 55 was held during 503 ms, while a threshold of 10 ms was specified to DRD.

### Detected Errors: Misuse of the POSIX threads API

DRD is able to detect and report the following misuses of the POSIX threads API:

- Passing the address of one type of synchronization object (e.g. a mutex) to a POSIX API call that expects a pointer to another type of synchronization object (e.g. a condition variable).

- Attempt to unlock a mutex that has not been locked.

- Attempt to unlock a mutex that was locked by another thread.

- Attempt to lock a mutex of type `PTHREAD_MUTEX_NORMAL` or a spinlock recursively.

- Destruction or deallocation of a locked mutex.

- Sending a signal to a condition variable while no lock is held on the mutex associated with the signal.

- Calling `pthread_cond_wait()` with a mutex that is not locked, that is locked by another thread or that has been locked recursively.

- Associating two different mutexes with a condition variable via `pthread_cond_wait()`.

- Destruction or deallocation of a condition variable that is being waited upon.

- Destruction or deallocation of a locked reader-writer lock.

- Attempt to unlock a reader-writer lock that was not locked by the calling thread.

- Attempt to recursively lock a reader-writer lock exclusively.

- Reinitialization of a mutex, condition variable, reader-writer lock, semaphore or barrier.

- Destruction or deallocation of a semaphore or barrier that is being waited upon.

- Exiting a thread without first unlocking the spinlocks, mutexes or reader-writer locks that were locked by that thread.

Regarding the message DRD prints about sending a signal to a condition variable while no lock is held on the mutex associated with the signal: DRD reports this because some calls of `pthread_cond_signal()` or `pthread_cond_broadcast()` while no lock is held on the mutex associated with the condition variable introduce subtle race conditions.

### Client Requests

Just as for other Valgrind tools it is possible to let a client program interact with the DRD tool.

The interface between client programs and the DRD tool is defined in the header file `<valgrind/drd.h>`. The available client requests are:

- `VG_USERREQ__DRD_GET_VALGRIND_THREAD_ID`. Query the thread ID that was assigned by the Valgrind core to the thread executing this client request. Valgrind's thread ID's start at one and are recycled in case a thread stops.

- `VG_USERREQ__DRD_GET_DRD_THREAD_ID`. Query the thread ID that was assigned by DRD to the thread executing this client request. DRD's thread ID's start at one and are never recycled.

- `VG_USERREQ__DRD_START_SUPPRESSION`. Some applications contain intentional races. There exist e.g. applications where the same value is assigned to a shared variable from two different threads. It may be more convenient to suppress such races than to solve these. This client request allows to suppress such races. See also the macro `DRD_IGNORE_VAR(x)` defined in `<valgrind/drd.h>`.

- `VG_USERREQ__DRD_FINISH_SUPPRESSION`. Tell DRD to no longer ignore data races in the address range that was suppressed via `VG_USERREQ__DRD_START_SUPPRESSION`.

- `VG_USERREQ__DRD_START_TRACE_ADDR`. Trace all load and store activity on the specified address range. When DRD reports a data race on a specified variable, and it's not immediately clear which source code statements triggered the conflicting accesses, it can be helpful to trace all activity on the offending memory location. See also the macro `DRD_TRACE_VAR(x)` defined in `<valgrind/drd.h>`.

- `VG_USERREQ__DRD_STOP_TRACE_ADDR`. Do no longer trace load and store activity for the specified address range. range.

Note: if you compiled Valgrind yourself, the header file `<valgrind/drd.h>` will have been installed in the directory `/usr/include` by the command `make install`. If you obtained Valgrind by installing it as a package however, you will probably have to install another package with a name like `valgrind-devel` before Valgrind's header files are present.

### Debugging GNOME Programs With DRD

GNOME applications use the threading primitives provided by the `glib` and `gthread` libraries. These libraries are built on top of POSIX threads, and hence are directly supported by DRD. Please keep in mind that you have to call `g_thread_init()` before creating any threads, or DRD will report several data races on glib functions. See also the GLib Reference Manual for more information about `g_thread_init()`.

One of the many facilities provided by the `glib` library is a block allocator, called `g_slice`. You have to disable this block allocator when using DRD by adding the following to the shell environment variables: `G_SLICE=always-malloc`. See also the GLib Reference Manual for more information.

### Debugging Qt Programs With DRD

The Qt library is the GUI library used by the KDE project. Currently there are two versions of the Qt library in use: Qt3 by KDE 3 and Qt4 by KDE 4. If possible, use Qt4 instead of Qt3. Qt3 is no longer supported, and there are known problems with multithreading support in Qt3. As an example, using QString objects in more than one thread will trigger race reports (this has been confirmed by Trolltech -- see also Trolltech task #206152).

Qt4 applications are supported by DRD, but only if the `libqt4-debuginfo` package has been installed. Some of the synchronization and threading primitives in Qt4 bypass the POSIX threads library, and DRD can only intercept these if symbol information for the Qt4 library is available. DRD won't tell you if it has not been able to load the Qt4 debug information, but a huge number of data races will be reported on data protected via `QMutex` objects.

### Debugging OpenMP Programs With DRD

OpenMP stands for Open Multi-Processing. The OpenMP standard consists of a set of compiler directives for C, C++ and Fortran programs that allows a compiler to transform a sequential program into a parallel program. OpenMP is well suited for HPC applications and allows to work at a higher level compared to direct use of the POSIX threads API. While OpenMP ensures that the POSIX API is used correctly, OpenMP programs can still contain data races. So it makes sense to verify OpenMP programs with a thread checking tool.

DRD supports OpenMP shared-memory programs generated by gcc. The gcc compiler supports OpenMP since version 4.2.0. Gcc's runtime support for OpenMP programs is provided by a library called `libgomp`. The synchronization primites implemented in this library use Linux' futex system call directly, unless the library has been configured with the `--disable-linux-futex` flag. DRD only supports libgomp libraries that have been configured with this flag and in which symbol information is present. For most Linux distributions this means that you will have to recompile gcc. See also the script `drd/scripts/download-and-build-gcc` in the Valgrind source tree for an example of how to compile gcc. You will also have to make sure that the newly compiled `libgomp.so` library is loaded when OpenMP programs are started. This is possible by adding a line similar to the following to your shell startup script:

    export LD_LIBRARY_PATH=~/gcc-4.3.2/lib64:~/gcc-4.3.2/lib:

As an example, the test OpenMP test program `drd/tests/omp_matinv` triggers a data race when the option -r has been specified on the command line. The data race is triggered by the following code:

    #pragma omp parallel for private(j)
    for (j = 0; j < rows; j++)
    {
      if (i != j)
      {
        const elem_t factor = a[j * cols + i];
        for (k = 0; k < cols; k++)
        {
          a[j * cols + k] -= a[i * cols + k] * factor;
        }
      }
    }

The above code is racy because the variable `k` has not been declared private. DRD will print the following error message for the above code:

    $ valgrind --check-stack-var=yes --var-info=yes --tool=drd drd/tests/omp_matinv 3 -t 2 -r
    ...
    Conflicting store by thread 1/1 at 0x7fefffbc4 size 4
       at 0x4014A0: gj.omp_fn.0 (omp_matinv.c:203)
       by 0x401211: gj (omp_matinv.c:159)
       by 0x40166A: invert_matrix (omp_matinv.c:238)
       by 0x4019B4: main (omp_matinv.c:316)
    Allocation context: unknown.
    ...

In the above output the function name `gj.omp_fn.0` has been generated by gcc from the function name `gj`. Unfortunately the variable name (`k`) is not shown as the allocation context -- it is not clear to me whether this is caused by Valgrind or whether this is caused by gcc. The most usable information in the above output is the source file name and the line number where the data race has been detected (`omp_matinv.c:203`).

Note: DRD reports errors on the `libgomp` library included with gcc 4.2.0 up to and including 4.3.2. This might indicate a race condition in the POSIX version of `libgomp`.

For more information about OpenMP, see also openmp.org.

### DRD and Custom Memory Allocators

DRD tracks all memory allocation events that happen via either the standard memory allocation and deallocation functions (`malloc`, `free`, `new` and `delete`) or via entry and exit of stack frames. DRD uses memory allocation and deallocation information for two purposes:

- To know where the scope ends of POSIX objects that have not been destroyed explicitly. It is e.g. not required by the POSIX threads standard to call `pthread_mutex_destroy()` before freeing the memory in which a mutex object resides.

- To know where the scope of variables ends. If e.g. heap memory has been used by one thread, that thread frees that memory, and another thread allocates and starts using that memory, no data races must be reported for that memory.

It is essential for correct operation of DRD that the tool knows about memory allocation and deallocation events. DRD does not yet support custom memory allocators, so you will have to make sure that any program which runs under DRD uses the standard memory allocation functions. As an example, the GNU libstdc++ library can be configured to use standard memory allocation functions instead of memory pools by setting the environment variable `GLIBCXX_FORCE_NEW`. For more information, see also the libstdc++ manual.

### DRD Versus Memcheck

It is essential for correct operation of DRD that there are no memory errors like dangling pointers in the client program. Which means that it is a good idea to make sure that your program is memcheck-clean before you analyze it with DRD. It is possible however that some of the memcheck reports are caused by data races. In this case it makes sense to run DRD before memcheck.

So which tool should be run first ? In case both DRD and memcheck complain about a program, a possible approach is to run both tools alternatingly and to fix as many errors as possible after each run of each tool until none of the two tools prints any more error messages.

### Resource Requirements

The requirements of DRD with regard to heap and stack memory and the effect on the execution time of client programs are as follows:

- When running a program under DRD with default DRD options, between 1.1 and 3.6 times more memory will be needed compared to a native run of the client program. More memory will be needed if loading debug information has been enabled (`--var-info=yes`).

- DRD allocates some of its temporary data structures on the stack of the client program threads. This amount of data is limited to 1 - 2 KB. Make sure that thread stacks are sufficiently large.

- Most applications will run between 20 and 50 times slower under DRD than a native single-threaded run. Applications such as Firefox which perform very much mutex lock / unlock operations however will run too slow to be usable under DRD. This issue will be addressed in a future DRD version.

### Hints and Tips for Effective Use of DRD

The following information may be helpful when using DRD:

- Make sure that debug information is present in the executable being analysed, such that DRD can print function name and line number information in stack traces. Most compilers can be told to include debug information via compiler option `-g`.

- Compile with flag `-O1` instead of `-O0`. This will reduce the amount of generated code, may reduce the amount of debug info and will speed up DRD's processing of the client program. For more information, see also .

- If DRD reports any errors on libraries that are part of your Linux distribution like e.g. `libc.so` or `libstdc++.so`, installing the debug packages for these libraries will make the output of DRD a lot more detailed.

- When using C++, do not send output from more than one thread to `std::cout`. Doing so would not only generate multiple data race reports, it could also result in output from several threads getting mixed up. Either use `printf()` or do the following:

  1. Derive a class from `std::ostreambuf` and let that class send output line by line to `stdout`. This will avoid that individual lines of text produced by different threads get mixed up.

  2. Create one instance of `std::ostream` for each thread. This makes stream formatting settings thread-local. Pass a per-thread instance of the class derived from `std::ostreambuf` to the constructor of each instance.

  3. Let each thread send its output to its own instance of `std::ostream` instead of `std::cout`.

## Using the POSIX Threads API Effectively

### Mutex types

The Single UNIX Specification version two defines the following four mutex types (see also the documentation of `pthread_mutexattr_settype()`):

- normal, which means that no error checking is performed, and that the mutex is non-recursive.

- error checking, which means that the mutex is non-recursive and that error checking is performed.

- recursive, which means that a mutex may be locked recursively.

- default, which means that error checking behavior is undefined, and that the behavior for recursive locking is also undefined. Or: portable code must neither trigger error conditions through the Pthreads API nor attempt to lock a mutex of default type recursively.

In complex applications it is not always clear from beforehand which mutex will be locked recursively and which mutex will not be locked recursively. Attempts lock a non-recursive mutex recursively will result in race conditions that are very hard to find without a thread checking tool. So either use the error checking mutex type and consistently check the return value of Pthread API mutex calls, or use the recursive mutex type.

### Condition variables

A condition variable allows one thread to wake up one or more other threads. Condition variables are typically used to notify one or more threads about state changes of shared data. Unfortunately it is very easy to introduce race conditions by using condition variables as the only means of state information propagation. A better approach is to let threads poll for changes of a state variable that is protected by a mutex, and to use condition variables only as a thread wakeup mechanism. See also the source file `drd/tests/monitor_example.cpp` for an example of how to implement this concept in C++. The monitor concept used in this example is a well known concept in computer science -- see also Wikipedia for more information about the monitor concept.

### pthread_cond_timedwait() and timeouts

Historically the function `pthread_cond_timedwait()` only allowed the specification of an absolute timeout, that is a timeout independent of the time when this function was called. However, almost every call to this function expresses a relative timeout. This typically happens by passing the sum of `clock_gettime(CLOCK_REALTIME)` and a relative timeout as the third argument. This approach is incorrect since forward or backward clock adjustments by e.g. ntpd will affect the timeout. A more reliable approach is as follows:

- When initializing a condition variable through pthread_cond_init(), specify that the timeout of pthread_cond_timedwait() will use the clock `CLOCK_MONOTONIC` instead of `CLOCK_REALTIME`. You can do this via `pthread_condattr_setclock(..., CLOCK_MONOTONIC)`. See also `drd/tests/monitor_example.cpp` for an example.

- When calling `pthread_cond_timedwait()`, pass the sum of `clock_gettime(CLOCK_MONOTONIC)` and a relative timeout as the third argument.

### Assigning names to threads

Many applications log information about changes in internal or external state to a file. When analyzing log files of a multithreaded application it can be very convenient to know which thread logged which information. One possible approach is to identify threads in logging output by including the result of `pthread_self()` in every log line. However, this approach has two disadvantages: there is no direct relationship between these values and the source code and these values can be different in each run. A better approach is to assign a brief name to each thread and to include the assigned thread name in each log line. One possible approach for managing thread names is as follows:

- Allocate a key for the pointer to the thread name through `pthread_key_create()`.

- Just after thread creation, set the thread name through `pthread_setspecific()`.

- In the code that generates the logging information, query the thread name by calling `pthread_getspecific()`.

## Limitations

DRD currently has the following limitations:

- DRD has only been tested on the Linux operating system, and not on any of the other operating systems supported by Valgrind.

- Of the two POSIX threads implementations for Linux, only the NPTL (Native POSIX Thread Library) is supported. The older LinuxThreads library is not supported.

- DRD, just like memcheck, will refuse to start on Linux distributions where all symbol information has been removed from ld.so. This is a.o. the case for the PPC editions of openSUSE and Gentoo. You will have to install the glibc debuginfo package on these platforms before you can use DRD. See also openSUSE bug 396197 and Gentoo bug 214065.

- When DRD prints a report about a data race detected on a stack variable in a parallel section of an OpenMP program, the report will contain no information about the context of the data race location (`Allocation context: unknown`). It's not yet clear whether this behavior is caused by Valgrind or by gcc.

- When address tracing is enabled, no information on atomic stores will be displayed. This functionality is easy to add however. Please contact the Valgrind authors if you would like to see this functionality enabled.

- If you compile the DRD source code yourself, you need gcc 3.0 or later. Gcc 2.95 is not supported.

## Feedback

If you have any comments, suggestions, feedback or bug reports about DRD, feel free to either post a message on the Valgrind users mailing list or to file a bug report. See also &vg-url; for more information about the Valgrind mailing lists or about how to file a bug report.