poor performance with Intel X520 card

[Eugene Grosbein]

16.07.2020 14:57, Patrick Lamaiziere wrote:

>> I'm sure pf is the bottle-neck. Try testing such card without any
>> packet filter enabled and you'll see great difference definitely.
> 
> That's not a good news as I don't see how to simplify the ruleset :(
> But thanks anyway :)

First, you need to determine if all your CPU cores loaded evenly or not. Use top -SHPI to check it out.
It's possible that they are not because defaults could be not optimal for your hardware
and you have bottle-neck just because of wrong settings for hardware receiving queues of the NIC:

FreeBSD/SMP: 2 package(s) x 6 core(s)
ix0: Using MSI-X interrupts with 9 vectors

This means 8 RX/TX queues (plus one link queue) for your 12 cores, not good.
8 is driver's default for systems with 8 cores or more unless you override it with
hw.ix.num_queues=12 in /boot/loader.conf, do it.

Also you should monitor interrupt counters for receiving queues for the card with
"systat -vm 3" (in real time) and with vmstat -ai | grep ix0 (cumulative).

If you have interrupts evenly distributed over receiving queues but still uneven CPU cores load,
that would be next problem to deal with. Maybe you'll need to (re-)bind queues to cores with cpuset(1).

Or maybe your received queues are loaded not evenly due to nature of Ethernet frames you work with,
f.e. non-IP ethernet traffic (PPPoE frames) or many non-TCP/UDP IP packets (GRE, ESP etc.)
cpuset will be your friend for that case, too.