SO_REUSEPORT_LB accept-learned receive locality A connection arrives and lands on a CPU. The kernel stores which CPU it landed on. A worker calls accept() and the kernel: - Increments the landing CPU's connection count in that listener group - Learns which CPU that socket is drained on, if the worker is pinned to a single CPU Now the kernel knows where traffic lands, where workers run, and how much traffic is landing on each CPU, and can tie it together. Once a listening socket is associated with a CPU (at accept()), the kernel builds a table for the group: - Each CPU gets slots in proportion to the amount of traffic landing on it - Each listening socket gets the same number of slots. - Each CPU fills its slots with the closest sockets first, based on scheduler topology, then farther ones as those fill up The next connection that lands on a CPU is sent to one of that CPU's slots and goes to that slot's socket. The table is rebuilt when the sockets or receiving CPUs change, and every 10 seconds while connections keep arriving. If no worker is pinned or if the CPU has no slots assigned, existing behavior (hash selection) is retained. I have been running a test implementation of this for ~2 months without issue. From synthetic testing (64-byte request/response from a client) on a two-socket (SMT + NUMA) machine (against current implementation): - With 8 workers on the 8 receiving CPUs of one socket: - ~96% of requests were handled by the worker on the CPU they landed on, - cross-CPU wakeups fell from about 266k to about 8k per ~300k requests - (Cycles per request were unchanged, since all of it had one L3) - With receive traffic on both sockets and 16 workers (one NIC per socket): - cross-CPU wakeups fell from about 292k to about 15k per ~300k requests. - cycles per request fell about 11%