FreeBSD · sys/net · six commits

Per-NIC RSS Controls

A generic way to read and program a network card's receive-side scaling hash key and indirection table, wired through iflib, with aq(4) as the reference driver and ifconfig as the consumer.

PROVEN

Validated on hardware

Hardware-validated 2026-09-26 on bare metal, both silicon generations. Get and set of the hash key and the indirection table are proven on aq(4) A1 (AQC107) and A2 (AQC113): ifconfig rsstable (single, list, and range) and rsskey read back correctly and live-reprogram the card - with the default 0-7 table sixteen flows spread across all eight rx_queues, and with rsstable N every received packet landed on rx_queue N. ix(4)/ixgbe correctly returns EOPNOTSUPP (not converted). A2 additionally reports the ipv6ex/tcp6ex hash types. Proven on a real two-socket machine.

01

What the two controls actually do

A card with more than one receive queue has to decide which queue each frame lands in. It hashes a tuple from the headers with a Toeplitz key, masks the result into an index, and looks the queue up in an indirection table. The key decides which bucket a flow falls in; the table decides which queue that bucket feeds, and therefore which CPU does the receive work.

frame 4-tuple Toeplitz hash 40-byte key hash & mask index indirection table [0] 0 [1] 1 [2] 2 [3] 3 [4] 4 ... queue RX queue 3 ithread bound at attach CPU 4 SIOCSIFRSSKEY ifconfig aq0 rsskey <hex> SIOCSIFRSSTABLE ifconfig aq0 rsstable 3 Pointing every entry at one queue sends every flow there, which is what makes steering observable in a per-queue counter.
The receive path the two setters reach into. The key selects the bucket, the table selects the queue. Everything downstream of the queue, including which CPU runs the interrupt thread, is fixed when the driver attaches.
02

How a request reaches the hardware

All interface ioctls enter through ifhwioctl(), so privilege is settled before anything looks at the request. The setters join the group that already carries SIOCSIFMEDIA, which also means a successful set stamps if_lastchange for free. Validation then happens once, in iflib, so no driver repeats it.

An interface that cannot answer gives one of two different errors, and the difference is diagnostic rather than cosmetic: an iflib driver that simply has not implemented the methods answers from the default stub, while a driver outside iflib never had a case at all.

ifconfig aq0 rsstable 3 ioctl(SIOCSIFRSSTABLE) ifhwioctl() priv_check(PRIV_NET_HWIOCTL) on success: if_lastchange EPERM unprivileged caller if_ioctl iflib_if_ioctl() nentries == isc_rss_table_size every entry < isc_nrxqsets key is Toeplitz, length fits CTX_LOCK held across the call EINVAL malformed request IFDI_SET_RSS_TABLE aq(4) method hardware, then softc shadow default stub EOPNOTSUPP em0: iflib, no method ether_ioctl() EINVAL vtnet0, lo0: no case at all All four answers are exercised by the regression test; the three on real interfaces were confirmed on a kernel built from this stack.
One entry point, three answers. The dashed edges are the paths that refuse the request. Keeping EOPNOTSUPP distinct from EINVAL is what lets a caller tell “this driver has not implemented it” from “this interface has no such concept”.
03

Meeting the methods that landed first

The upstream get_rss_key and get_rss_hash methods went in on 2026-09-18 so that hn(4) could read a passed-through card's settings. This work carried a parallel five-method family of its own, which after the rebase left two case SIOCGIFRSSKEY labels in the same switch.

The resolution keeps the upstream methods untouched and adds ours beside them in the same naming, sharing one default stub per signature the way the file's own queue-setup and VLAN methods already do.

ifdi methodOriginAnswersDefault
get_rss_keyupstreamSIOCGIFRSSKEYnull_rss_key_op
set_rss_keythis stackSIOCSIFRSSKEYnull_rss_key_op
get_rss_hashupstreamSIOCGIFRSSHASHnull_get_rss_hash
get_rss_tablethis stackSIOCGIFRSSTABLEnull_rss_table_op
set_rss_tablethis stackSIOCSIFRSSTABLEnull_rss_table_op

The table the ABI has to carry

The structure is fixed-size, so its length is encoded in the ioctl number and can never grow later. ice(4) selects its lookup table size from a firmware capability and its admin queue defines a two-thousand-entry table for physical functions, which is the largest any in-tree driver can ask for; the structure is sized for that and still lands well inside the kernel's ioctl argument limit.

QuantityValueWhy
Table entries2048ice's 2K lookup table; a fixed struct cannot grow past it
Struct size4116 Bmalloc'd by the ioctl path, never on a stack, nothing per interface
Argument limit8192 Bthe ceiling the ioctl command word can express
Entry widthuint16ice reports up to 256 queues, which a byte cannot count
04

Writing to a device that can fail mid-write

aq programs its key ten registers at a time and its table in register-sized pieces, each behind a handshake that can time out. A write that stops half way leaves a mixture of old and new entries in hardware while the driver's own copy still describes the old state.

The answer adds no new state. The driver keeps its copy unchanged, reports the error, and asks iflib for a reinit, because aq_if_init() already reprograms the key and the table from that copy on every path. The hardware is pulled back to the thing the getters have been reporting all along.

set arrives validated by iflib interface running and init did not fail? no update the softc copy only applied at the next up yes program hardware register pieces, handshake each ok commit the copy getters now agree timeout: hardware half written leave the copy alone print once, request a reset, defer the admin task aq_if_init() reprograms key and table from the unchanged copy caller still sees the error; hardware returns to the reported state
Hardware first, copy second, reinit on failure. Committing the copy first would leave the getters describing a key the device never took; rolling back through the same handshake could fail the same way.
05

What a per-NIC key means for the common key

This was asked twice in review: every card already programs the same key, exported through a sysctl, so why add anything. The answer is that nothing changes until somebody sets something.

rss_config.c rss_getkey() net.inet.rss.key (read only) at init every driver aq, ix, ice, bnxt, mlx5en, ... unchanged by this work default: everyone shares one key, as today after an explicit, privileged set on one interface rss_config.c key and rss_table[] still untouched at init ix, ice, ... common key aq0 only operator's key, kept across reinit PRIV_NET_HWIOCTL under options RSS the software hash no longer predicts aq0's queue Undoing it needs no new interface: read net.inet.rss.key and program that value back with ifconfig. A jail's root can set a key but cannot read the common one, which is the only gap left open on purpose.
Divergence is opt-in and reversible. No code path and no allocation changes for an interface nobody configures, which is what keeps a hundred thousand interfaces and Netflix's pipeline out of it.
06

The stack

Six commits, each building on its own, cut against an unmodified upstream so every diff applies to head one at a time. The order is a real dependency chain, not a filing convention: the ABI needs no driver, iflib needs the ABI, and both driver commits rely on the attach gate iflib adds, the read side because it dropped its own readiness flag.

  1. net: add ioctls for the RSS key and indirection tableThree new commands, the table structure, privilege placement, and the helper that maps the two constant families, which are a permutation of one another rather than a shift.
  2. iflib: add the RSS key and indirection table settersThree methods beside the two that landed, central validation, a single attach gate that answers ENXIO for all five until the driver has finished attaching, and the first documentation any of the five have had.
  3. aq: report the programmed RSS key and hash typesThe two query methods in the existing per-driver style, reporting what the driver actually programs, with no readiness flag of its own because iflib gates the calls. Atlantic 2 selects hash types individually in a register; Atlantic 1 has no such register, so it does not claim the IPv6 extension-header tuples.
  4. aq: implement the RSS key and indirection table settersThe write path, the live-device check, and the reinit that recovers a half-written register sequence.
  5. ifconfig: report and set the RSS configurationVerbose status plus two verbs, arguments parsed before the kernel is asked anything, with manual text and shell tests.
  6. tests: cover the RSS ioctl dispatch and validationStructure sizes, the commands that encode them, privilege, and all three refusal paths; the hardware round trip is skipped unless an interface is named.
07

What was run

CheckScopeResult
Per-commit compilekernel objects and the aq module at each of the sixclean
Kernel buildsGENERIC, an options RSS config, and the host's own config with debuggingno warnings
Userlandifconfig with and without netlink, tests, the placement toolclean
Manual pageslint on three pages; no new findings in ifconfig'sclean
Kernel regression testsfive runnable cases on a guest booted from this stack; the hardware round trip skips without an interface name5 of 5
ifconfig testsstatus, both parsers, refusal, privilege5 of 5
Dispatch on real interfacesem0, vtnet0 and lo0 on the new kernelas designed
Placement tool fixturesre-run against the larger table11 of 11
Code reviewfifteen verified findings; eleven fixed in place, three decided, one pre-existing aq gap left for its own commitdone
Card steering and key A/Bneeds the Atlantic card handed to the guestnot yet run
Still gated

One check waits on a decision rather than on work: the card steering and key A/B needs the Atlantic card handed through to the test guest, and PCI passthrough teardown has panicked the machine once before.

08

Deliberately absent

The scope was set by the person who has to review it, and the answer to “why not also” is written into the commit messages rather than left for a reviewer to ask.

  • A hash type setter and a capability query. Nothing consumes one, and the tuple selection is tied to the chip generation, so a portable setter needs discovery first.
  • Hash function or symmetry selection. No driver in the tree offers a second function to select.
  • Drivers outside iflib, and lagg or vlan proxying. Those forward only media ioctls today.
  • Queue placement under options RSS. iflib's own clamps there are dead code for an unrelated reason; reviving them is its own change with its own testing.
  • Anything touching receive-CPU affinity. Explicitly excluded by the people who asked for this.