people.freebsd.org / ~nprice / diagrams

Diagrams

Design and review drawings for FreeBSD networking and storage work. The design proposals below are prototyped and measured on a two-socket Xeon E5-2470 v2 (Ivy Bridge, QPI) and a Ryzen 9 5950X (two CCDs); the numbers appear on each page.

Proposed designs

SO_REUSEPORT_LB receive-CPU affinity

End-to-End Receive Localityoverview

One flow from wire to worker, the measured cost ladder, and how the three commits connect RSS steering to the accepting worker. Start here.

Receive-Queue Spreadcommit 1

iflib optionally spreads a NIC's queue CPUs across cache groups, so the deal has a local worker to hand out on each.

Receive-CPU Dealingcommit 2

Each receiving CPU deals its share of connections to the members nearest it; every member is owed an equal share, so no worker starves.

The Ring Modelcommit 2 detail

Where "nearest" comes from — the scheduler's own group chain — and a deal traced ring by ring.

per-NIC RSS

Per-NIC RSS Controls

Read and program a card's RSS hash key and indirection table through iflib, with aq(4) as the reference driver and ifconfig as the consumer. Get and set proven on aq A1 + A2.

PCI passthrough

Passthrough Reset Lifecycle

The assign/reset guardrails - IOMMU-map, quiesce, wait-for-FLR, restore config - and the FLR to bus to fabric reset escalation ladder for bhyve PCI passthrough.

Reference

how existing mechanisms work

iflib & RSS: From Packet to Processorprimer

How iflib and RSS steer inbound traffic by default, and how the socket-layer affinity work connects that steering to the application.

Magicsock Path States

Tailscale's path selection between direct and relayed connections.

The Flow-Steering Landscape

Where RSS and hardware flow steering can and cannot be programmed across the FreeBSD stack - the read-only, dead, and vendor-private pieces, and the hole in the middle.

Memory Ordering: x86-64 & arm64

How x86-64 and arm64 order memory, how Linux and FreeBSD ask for order on each (with the per-call SPL mapping), and where the ZIO batch race fits. Companion to the batch page.

umtx Chain-Hash Distribution

How umtxq_hash spreads wait words across chains, why a sparse multiplier collapses at power-of-two strides, and the fair-constant swap that fixes it.

Bug fixes

a defect, and how it was fixed

ZFS Error Log Routing

How ZFS files, keeps, and reads a permanent error, and the routing bug that made scrub entries permanent.

ZIO Batch Arrival Race

An arm64 txg wedge: a ZIO batch publish/release with no barrier pairing on relaxed SPL atomics.

Archive

superseded designs, kept for the record

Exact Claim to Ring Dealsuperseded designs

Why the earlier exact-claim designs starved workers, and how the ring deal answers all of it. Consolidates three retired pages.