Re: RSS causing bad forwarding performance?

On Sun, Dec 07, 2025 at 11:04:57PM +0100, Kajetan Staszkiewicz wrote:
> Hello Group,
> 
> I'm using Cisco TRex to evaluate forwarding perfomance of my FreeBSD
> routers. I wanted to establish a baseline of what FreeBSD 15 can forward
> without pf and compliated routing. The DUT is using a 6-core Intel
> E-2146G CPU with disabled HT, Intel x520 for management and Mellanox
> ConnectX-5 for forwarding. The mce interfaces use a separate fib, there
> are just a few static routes to make TRex work, they are configured as
> they should be for a router: -lro -mediaopt rxpause,txpause. The tests
> have been performed without any NIC sysctl tuning.
> 
> Testing is done with
> single udp stream:
>   start -f stl/udp_1pkt_simple.py -m 50% --port 0
> multiple udp streams:
>   start -f stl/udp_1pkt_repeat_random.py -m 50% --port 0
> 
> Links are at 25Gb/s so at 50% TRex pushes around 18 Mpps to the DUT.
> 
> NetISR is configured to make use of all CPU cores:
> net.isr.bindthreads=1
> net.isr.maxthreads=-1
> 
> On the GENERIC kernel I'm getting:
> dispatch=deferred single stream    :  5.2  Mpps
> dispatch=deferred multiple streams :  4.2  Mpps
> dispatch=direct   single stream    :  3.2  Mpps
> dispatch=direct   multiple streams : 10.7  Mpps
> 
> GENERIC + option RSS:
> dispatch=deferred single stream    :  0.4  Mpps
> dispatch=deferred multiple streams : 11.0  Mpps
> dispatch=direct   single stream    :  0.4  Mpps
> dispatch=direct   multiple streams : 11.0  Mpps
> 
> GENERIC + option RSS + forwarding over Intel x520 NICs just to be sure
> that it's not Mellanox's fault:
> dispatch=deferred single stream    :  between 1.9 and 0.1 Mpps
> dispatch=deferred multiple streams :  4.5  Mpps
> dispatch=direct   single stream    :  between 1.9 and 0.1 Mpps
> dispatch=direct   multiple streams :  4.5  Mpps
> 
> As you can see with option RSS and a single UDP stream the router
> totally clogs, dropping forwarding performance as low as 100kps. Without
> option RSS it works just fine.
It is somewhat strange that with/without RSS results differ for UDP.
mlx5en driver always enable hashing the packet into rx queue.  And,
with single UDP stream I would expect all packets to hit the same queue.
As consequence, with/without RSS should be same (low).

Would it be UDP which encapsulates some other traffic, e.g. tunnel that
can be further classified by the internal headers, like inner headers
of the vxlan, then more that one receive queue could be used.

BTW, mce cards have huge numbers of supported offloads, but all of them are
host-oriented, they would not help for the forwarding.

> 
> Please note, that this test is not about forwarding "real" traffic, like
> an iperf TCP stream, which would adjust the packet sending rate to
> capacity of the DUT, but flooding it with more traffic than it can not
> forward. Sadly the later is often the case for devices exposed to the
> Internet.

Again, since iperf stream would hit single send/receive queue.
Parallel iperfs between same machines scale.