From a packet on the wire to the worker that accepts it

A TCP/IPv4 connection to port 443 through aq(4), iflib, netisr, TCP and SO_REUSEPORT_LB listener selection, on either code version (Code, top left). Current FreeBSD picks the listener by hash alone; the proposed change adds the receive-CPU locality map and accept feedback. Each step has the controls that change it, and the left panel follows every change. Hashes, queue placement, scheduler groups, the map builder and the accept hook are ports of the source; topology, traffic and worker setup are synthetic.

Settings to try, the ones deployments commonly tune: RX queues, core_spread, NICs,net.isr.dispatch, worker pinning and reuseport_lb_balance. Tags show where a setting lives, highlighted where it is commonly tuned; untagged controls describe the hardware, the workload or the model.

1Packet on the wire

The final handshake ACK of a new connection from a client to 198.51.100.20:443.

2NIC: hash, indirection table, RX queue

The NIC computes a 32-bit Toeplitz hash H with its 40-byte key, uses H to pick one of 64 table entries, and DMAs the packet into the RX queue that entry names.

Click an entry to change its queue on every NIC (the SIOCSIFRSSTABLE ioctl; ifconfig aq0 rsstable repeats a queue list).

Which bits of H index the table is hardware behavior, not visible in the driver. This page uses the low 6 bits, the usual RSS convention. Changing the key moves flows between queues. It does not change the listener hash in step 5. The global software key only enters when the driver reports no hash (step 6).

3Interrupt and RX task CPU

Each RX queue's interrupt and iflib task were bound to one CPU at attach. That CPU drains the queue's descriptors, builds the mbuf with the NIC's hash and hash type, and runs the stack.

RX queueTable entriesRX task CPUConnections in run

4IP and TCP input CPU

Ethernet input is always direct. IP input runs on the RX task CPU unless netisr queues it to a netisr thread (deferred, or hybrid when that thread is on another CPU).

5Socket lookup and listener selection

No connected socket matches yet, so TCP looks up the SO_REUSEPORT_LB group for :443. A Jenkins hash of the client IP, XORed with the ports and finalized, picks a slot in the row that belongs to the CPU running TCP. Each slot names a listener.

This CPU's row

All rows in the current map ()

6Handshake completes, worker accepts

syncache_socket() creates the child on the selected listener and stamps it with the current CPU. When the worker's accept(2) returns, the TCP hook may count that CPU and may learn the worker's CPU.

7Feedback over many connections

The map is rebuilt from what accept observes. This run sends connections from a pool of client IPs through steps 2-6 and replays the hook, starting from freshly bound listeners. Drag the slider to see the map, and the trace above, at any point in the run.

Connections per listener

Distance from TCP CPU to worker CPU

Map rebuilds (click to jump)

Accept counts the map is built from

8After accept

Later segments find the connected socket by exact lookup on whatever CPU receives them, append to its socket buffer and wake the worker. The worker copies data out on its own CPU. The map only chose which listener, and so which worker, gets the connection; it does not move TCP input, queues or packet memory. Stacks that use HPTS (RACK, BBR) run pacing and timers on the HPTS CPU from step 6.

Scope and model notes
  • Proposed change as of 2026-11-17, compared with FreeBSD main of 2026-11-11. GENERIC-like amd64 kernel without options RSS, one or more identical aq NICs, IPv4. Not measured on hardware.
  • Scheduler groups are built by a port of x86topo_add_sched_group() over a synthetic APIC topology; CPU ids follow APIC order, so SMT siblings are adjacent. The Ryzen 9 5950X preset matches that machine's kern.sched.topology_spec; the other presets follow each CPU's published cache layout and were not checked on the hardware. Hybrid P-core/E-core parts cannot be drawn.
  • Hashes, scheduler groups, the map builder, the selector and the accept hook were checked against the kernel's own functions compiled natively, including the page's runs replayed connection by connection.
  • Not modeled: other iflib queue tunables, MSI-X limits, a busy netisr thread under hybrid, NUMA turned off. With no NIC hash the queue still comes from the aq table here.
  • An unbound netisr thread is shown as a random CPU per packet; the real CPU is whatever the scheduler picked.
  • Arrival times are evenly spaced; every connection is accepted at once by the selected listener's worker.
  • TCP_REUSPORT_LB_NUMA group affinity (already in FreeBSD) is not modeled: all listeners are in one M_NODOM group. The NUMA controls shape the scheduler groups and the NIC's interrupt CPU set.