people.freebsd.org / ~nprice / diagrams

Diagrams

Design and review drawings for FreeBSD networking and storage work. The design proposals below are prototyped and measured on a two-socket Xeon E5-2470 v2 (Ivy Bridge, QPI) and a Ryzen 9 5950X (two CCDs); the numbers appear on each page.

Proposed designs

SO_REUSEPORT_LB receive-CPU affinity

End-to-End Receive Localityoverview

One flow from wire to worker, the measured cost ladder, and how the three commits connect RSS steering to the accepting worker. Start here.

Receive-Queue Spreadcommit 1

iflib optionally spreads a NIC's queue CPUs across cache groups, so the deal has a local worker to hand out on each.

Receive-CPU Dealingcommit 2

Each receiving CPU deals its share of connections to the members nearest it; every member is owed an equal share, so no worker starves.

The Ring Modelcommit 2 detail

Where "nearest" comes from — the scheduler's own group chain — and a deal traced ring by ring.

per-NIC RSS

Per-NIC RSS Controls

Read and program a card's RSS hash key and indirection table through iflib, with aq(4) as the reference driver and ifconfig as the consumer. Get and set proven on aq A1 + A2.

PCI passthrough

Passthrough Reset Lifecycle

The assign/reset guardrails - IOMMU-map, quiesce, wait-for-FLR, restore config - and the FLR to bus to fabric reset escalation ladder for bhyve PCI passthrough.

Reference

how existing mechanisms work

iflib & RSS: From Packet to Processorprimer

How iflib and RSS steer inbound traffic by default, and how the socket-layer affinity work connects that steering to the application.

Magicsock Path States

Tailscale's path selection between direct and relayed connections.

The Flow-Steering Landscape

Where RSS and hardware flow steering can and cannot be programmed across the FreeBSD stack - the read-only, dead, and vendor-private pieces, and the hole in the middle.

Memory Ordering: x86-64 & arm64

How x86-64 and arm64 order memory, how Linux and FreeBSD ask for order on each (with the per-call SPL mapping), and where the ZIO batch race fits. Companion to the batch page.

umtx Chain-Hash Distribution

How umtxq_hash spreads wait words across chains, why a sparse multiplier collapses at power-of-two strides, and the fair-constant swap that fixes it.

Bug fixes

a defect, and how it was fixed

ZFS Error Log Routing

How ZFS files, keeps, and reads a permanent error, and the routing bug that made scrub entries permanent.

ZIO Batch Arrival Race

An arm64 txg wedge: a ZIO batch publish/release with no barrier pairing on relaxed SPL atomics.

Quadratic vm_map Check After fork()

How a DIAGNOSTIC kernel made every forked smbd do over a billion node visits in its first page fault, and the linear map check that fixes it.

vm_map Check Walkthrough

The vm_map consistency check walked on a seven-entry example: the bounds at every node, each assertion with its values, the pop rule, and why 64 pending slots always suffice.

Archive

superseded designs, kept for the record

Exact Claim to Ring Dealsuperseded designs

Why the earlier exact-claim designs starved workers, and how the ring deal answers all of it. Consolidates three retired pages.