Thread transitions and scheduler locking

Switching away from a thread

mi_switch() in kern_synch.c performs the machine-independent parts of a context switch. Its entry contract requires the thread lock; the normal return path no longer holds that lock. It records voluntary or involuntary switching and runtime accounting, then calls the selected sched_switch() implementation.

The implementation decides what to do with the outgoing thread, chooses the next thread, and enters the machine-dependent switch machinery. The reason and flags passed to mi_switch() distinguish operations such as voluntary blocking from preemption. A caller must establish a valid thread state before entering that path.

Waking and becoming runnable

Sleep queues and turnstiles manage different kinds of waiting. Their wakeup paths eventually coordinate with scheduler operations to make a thread eligible for execution. Wakeup does not guarantee that the awakened thread runs immediately or that it will resume on its previous CPU.

setrunnable() checks the thread’s state and calls sched_wakeup() when it is ready to transition from TDS_CAN_RUN. Its normal behavior releases the thread lock, with SRQ_HOLD and SRQ_HOLDTD selecting different lock-retention behavior. A thread still inhibited by another condition is not made runnable merely by clearing one reason to wait.

The implementation then applies placement, priority, and preemption policy. For a remote CPU, notification and a later scheduling point can separate enqueueing from actual execution.

The thread lock can change

The lock protecting scheduler-related thread state is reached through the thread’s lock association. Scheduler operations and migration can transfer that association between locks. Use thread_lock() and the established scheduler helpers rather than assuming a thread permanently belongs to one particular run-queue lock.

ULE’s queue locks protect much of each tdq. 4BSD has its own scheduler locking protocol. Their td_sched structures also have different field layouts and protection rules. Code shared between implementations must respect the interface instead of inspecting an implementation’s private extension as though it were universal thread state.

Preemption, pinning, and affinity

A critical section and a thread’s pin count serve different purposes. Preventing an immediate scheduling action is distinct from restricting CPU migration. Likewise, an allowed cpuset is a set of eligible CPUs, while binding selects a particular CPU through the scheduler’s binding interface.

Use the interfaces for the actual invariant needed by the caller. Do not substitute a priority boost for a lock or treat affinity as protection for data that another CPU can still access.

Reviewing a change

For each transition, identify the incoming state, held lock, outgoing state, and outgoing lock ownership. Check the wakeup, timeout, signal, and error paths as well as the successful wait path. Priority lending through a turnstile adds another state transition that must be reconciled when a lock owner stops needing the borrowed priority.

The source assertions and WITNESS/INVARIANTS configurations help detect violations. Useful tracing distinguishes enqueue, dequeue, priority change, off-CPU, and on-CPU events; observing only a switch can hide the earlier event that caused the scheduling decision.