RSS causing bad forwarding performance?

Hello Group,

I'm using Cisco TRex to evaluate forwarding perfomance of my FreeBSD
routers. I wanted to establish a baseline of what FreeBSD 15 can forward
without pf and compliated routing. The DUT is using a 6-core Intel
E-2146G CPU with disabled HT, Intel x520 for management and Mellanox
ConnectX-5 for forwarding. The mce interfaces use a separate fib, there
are just a few static routes to make TRex work, they are configured as
they should be for a router: -lro -mediaopt rxpause,txpause. The tests
have been performed without any NIC sysctl tuning.

Testing is done with
single udp stream:
  start -f stl/udp_1pkt_simple.py -m 50% --port 0
multiple udp streams:
  start -f stl/udp_1pkt_repeat_random.py -m 50% --port 0

Links are at 25Gb/s so at 50% TRex pushes around 18 Mpps to the DUT.

NetISR is configured to make use of all CPU cores:
net.isr.bindthreads=1
net.isr.maxthreads=-1

On the GENERIC kernel I'm getting:
dispatch=deferred single stream    :  5.2  Mpps
dispatch=deferred multiple streams :  4.2  Mpps
dispatch=direct   single stream    :  3.2  Mpps
dispatch=direct   multiple streams : 10.7  Mpps

GENERIC + option RSS:
dispatch=deferred single stream    :  0.4  Mpps
dispatch=deferred multiple streams : 11.0  Mpps
dispatch=direct   single stream    :  0.4  Mpps
dispatch=direct   multiple streams : 11.0  Mpps

GENERIC + option RSS + forwarding over Intel x520 NICs just to be sure
that it's not Mellanox's fault:
dispatch=deferred single stream    :  between 1.9 and 0.1 Mpps
dispatch=deferred multiple streams :  4.5  Mpps
dispatch=direct   single stream    :  between 1.9 and 0.1 Mpps
dispatch=direct   multiple streams :  4.5  Mpps

As you can see with option RSS and a single UDP stream the router
totally clogs, dropping forwarding performance as low as 100kps. Without
option RSS it works just fine.

Please note, that this test is not about forwarding "real" traffic, like
an iperf TCP stream, which would adjust the packet sending rate to
capacity of the DUT, but flooding it with more traffic than it can not
forward. Sadly the later is often the case for devices exposed to the
Internet.

-- 
| pozdrawiam / regards | Powered by Debian and FreeBSD  |
| Kajetan Staszkiewicz |   www: http://tuxpowered.net   |
|                      | matrix: @vegeta:tuxpowered.net |
`----------------------^--------------------------------'