Re: Chelsio NIC with RSS - Traffic distribution to different Queues

Ok we are getting closer to a solution. But I’m a little bit confused, I’ll try to explain why.

I’ve removed the following sysctl values, as you stated that they are not official FreeBSD values:
- net.inet.rss.bits="2"
- net.inet.rss.enabled="1"
- net.isr.bindthreads="1"
- net.isr.maxthreads="-1"

Without the setting above and without your proposed sysctl values, my system was stable and the traffic distributed over all CPUs (even HT cores). So no more drops on CARP if CPU0 has had high load. It looks like my problem is clearly related to the “Custom” OPNsense RSS values above. As the RSS sysctl values are noted in the OPNsense guide (https://docs.opnsense.org/troubleshooting/performance.html), my intention was, that those values needs to be set anyway for RSS. But it seems that those 4 values are making the thing worse!

Back to your topic. You are right, queue-0 does only control traffic if I’m enabling your sysctl values. Where can I see which protocols are falling into control traffic (queue 0)? So your change works as expected, thanks a lot!

As you stated, queue 0 is never CPU 0, which is important to know.

I’m pretty sure that I needed the RSS values above to get RSS work on intel NICs and with “netstat -Q” I could see the RSS queues. With Chelsio it seems that the FreeBSD process “netisr” doesn’t do the RSS stuff and “netstat -Q” shows nothing about RSS. For me this means that your driver directly does the RSS magic, is this correct? Please help to shed light on this topic.

Cheers Josef