SO_REUSEPORT_LB / receive-CPU affinity / the deal in detail
A receiving CPU spends its connections on the workers nearest to it, and "nearest" is not a number this design invents. It is read from the tree the scheduler already built to describe the machine. Ring 0 is the CPU itself; each ring after that is one step up that tree; and one final ring catches whatever is left so the deal always closes.
The drawings use the two-socket machine the deal was validated on, where each socket is its own NUMA domain with its own L3. There that comes to four rings: the CPU, its SMT sibling, the rest of its socket, and a last ring, the catch-all, that admits anyone still owed, which here means the other socket, plus any member that named no CPU. On a different machine it is a different number: the rings are however deep that machine's tree is, and the code asks rather than assumes.
cg_parent, and the n-th group below the root is ring n; a last pass then admits anyone still owed. On this machine the NUMA domain and the L3 are the same group, so ring 2 is the socket and the last pass is the other socket. A machine that reports more levels gets more rings; one that reports none gets two.| machine | chain from a leaf | rings that mean something | what the deal does |
|---|---|---|---|
| 2x Xeon E5-2470 v2 (two sockets, two NUMA domains) | SMT pair, socket (its NUMA domain and L3), root | 0 exact, 1 sibling, 2 same socket, 3 last pass: the other socket (QPI) | the worked example above; validated 2026-09-26: receive on both sockets, deal -11.3% cyc/req vs hash, no worker starved |
| this host, 5950X (two CCDs, one NUMA domain) | SMT pair, L3 group, root | 0 exact, 1 sibling, 2 same L3, 3 last pass: the other CCD | the same four rings, with a CCD in place of a socket |
| one L3, no SMT | root only (a one-CPU core is not a group) | 0 exact, 1 same L3 | exact first, then anyone in the group |
flat topology (smp_topo_none) | root only | 0 exact, 1 everyone | degrades to exact-or-share, which is the honest answer for a machine that reports no structure |
| two sockets, 8 CCDs | SMT pair, L3, socket, root | 0 to 4, five rings | fills its own CCD, then its own socket, then crosses |
| chains of unequal depth | varies by CPU | as many as that CPU has | a CPU whose chain is shorter than the current level sits that pass out and waits for the catch-all, rather than reaching further than its neighbours |
| kernel without SMP | none | 0 exact, 1 everyone | one CPU, so the question does not arise |
kern.smp.topology override | whatever was asked for | follows it | nearness follows the administrator's description, right or wrong |
in_pcblbgroup_remap() counts the chain for each receiving CPU and the deal runs one pass per level plus the catch-all, so a machine the scheduler describes in more detail gets a finer-grained deal without any change here.Widening rings over a machine's own topology are established practice in schedulers and memory allocators, where they place a thread or a page near the work that needs it. Equal-share slot tables are established practice in load balancers, where they divide traffic evenly and survive a membership change without reshuffling everything.
Neither has been brought to the socket layer's choice of which listener receives a connection. That choice has until now been an exact CPU match or a hash: the first starves every worker not sitting on a receive queue, the second throws locality away. Putting the two together there is new. The rings decide who is nearest, the slot table guarantees the share, and between them a machine can use every worker it has without giving up the locality the hardware set up.
This host (Ryzen 9 5950X, two CCDs on one package - the same four rings as the drawings above, with the other CCD in place of the other socket), 2026-08. Here ring 3 is the cross-CCD hop over Infinity Fabric, and its cost is measured, not assumed: against a same-CCD wakeup it is +19% cycles per request. The exact-claim predecessor that the ring deal generalizes was run across three worker layouts - workers all on the receiving CCD gained nothing (the wakeup already shares L3), workers spanning both CCDs gained -15% (affinity vs hash), and forcing every worker to the far CCD cost -19% (affinity vs anti, the pure ring-3 penalty). The ring deal itself was not exercised here; it needs the two-socket receive topology below. On this host they make the cross-CCD hop the one expensive step.
Two-socket 2x Xeon E5-2470 v2 (Ivy Bridge-EP), two sockets over QPI, 2026-09-26. This is the machine drawn above: each socket is one NUMA domain with one L3, the chain from any leaf CPU is the SMT pair, then its socket, then the root, and ring 3, the last pass, is the cross-socket step. Booted the affinity kernel with direct netisr dispatch; cycles per request from a process-scoped PMC over the worker threads only. The -18.1% in the drawings is the exact-claim predecessor on this machine, with eight workers and receive on socket 0: workers co-located with their receive CPU against the same workers pinned to the far socket.
The deal delivers when receive spans both sockets. One receive-queue set on each socket, one worker per receive CPU across both (16 workers): plain hash scatters about half the connections to the far socket, while the ring deal keeps each socket's connections on its own socket and still uses every worker. Cross-CPU wakeups fell from ~292k to ~15k and cycles per request dropped 11.3% against the hash - matching the exact-claim design's 12.1% while starving no worker and needing no application change (the learning is on by default; the workers only pin themselves).
Same-socket, the mechanism co-locates ~96% (workers one per receive CPU on one socket): cross-CPU wakeups ~266k to ~8k, learned automatically at accept(2). Cycles per request are flat there, because same-socket wakeups already share the L3 - the ring-3 hop is the only expensive step, exactly as drawn.
One caveat the runs surfaced: supply is learned in two ten-second windows, so a benchmark shorter than that under-converges and the deal stays partly on the hash. Sustained traffic converges it; short bursty runs show variance. Worth weighing where the window length meets real workloads.