A question about RSS bytes and Toeplitz hash

From: Marek Zarychta <zarychtam_at_plan-b.pwste.edu.pl>
Date: Sat, 04 Mar 2023 21:40:53 UTC
Dear subscribers,

I have recently enabled and turned on Receive Side Scaling/Steering. The 
main reason was to use different hashing for the distribution of UDP 
datagrams over multiple paths. RSS uses 2-tuple hashing for UDP, thus 
UDP transmission from one host to another takes always the same path. 
Without RSS bits enabled hashes computed for choosing the path in 
MULTIPATH are done for both UDP and TCP on the same basis of srcip:port 
dstip:port.
At a glance "options RSS" seems to be a hidden feature. There is no 
RSS(9), though PCBGROUP(9) refers to that page, neither RSS is present 
in LINT.  Anyway, it can be added to the custom kernel and turned on.
 From time to time I am profiling the system with pmcstat(8). It seems 
that when IPv6 traffic is processed, the computations of the Toeplitz 
hashes (with RSS enabled) take a higher share of CPU cycles, which is 
expected, but the overall share of CPU cycles spent on the computations 
of these hashes is unexpectedly high (when compared for example with 
computation of different hashes, fib lookups etc.) and seems to be the 
cost (drawback ?) of enabling RSS bytes. I can share the samples and 
flame graphs. If anyone is concerned, please PM me.
Digging a bit into the code I found out that our implementation is 
relatively simple[1] when compared with for example approach taken by 
DragonFlyBSD[2].  On the other hand, there is a notion of the importance 
of caching in our code[3]. I am not the programmer, perhaps refactoring 
won't help much in this case.
Concluding, recently there is more cleanup of RSS code in the source 
repository than any progress or refactoring. Please let me ask then if 
these RSS bytes are still relevant and worth enabling or should be 
considered abandoned and going to be purged with time.

[1] https://cgit.freebsd.org/src/tree/sys/net/toeplitz.c
[2] 
https://github.com/DragonFlyBSD/DragonFlyBSD/blob/master/sys/net/toeplitz.c
[3] https://cgit.freebsd.org/src/tree/sys/net/toeplitz.h

Yours sincerely

-- 
Marek Zarychta