Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1

[Rudy]

On 2/12/20 7:21 PM, Rudy wrote:
>
>
> I'm having issues with a box that is acting as a BGP router for my 
> network.  3 Chelsio cards, two T5 and one T6.  It was working great 
> until I turned up our first port on the T6.  It seems like traffic 
> passing in from a T5 card and out the T6 causes a really high load 
> (and high interrupts).


Update, I have found the chelsio_affinity script and run that, doesn't 
seem to help.  I did notice the T6 card is on numa-domain 0 and the T5 
cards on on numa-domain 1.  Does that matter?

Here is the relevant part of dmesg:

t6nex0: <Chelsio T62100-LP-CR> mem 
0xc5300000-0xc537ffff,0xc4000000-0xc4ffffff,0xc5984000-0xc5985fff irq 48 
at device 0.4 numa-domain 0 on pci12
cc0: <port 0> numa-domain 0 on t6nex0
cc1: <port 1> numa-domain 0 on t6nex0
t6nex0: PCIe gen3 x16, 2 ports, 34 MSI-X interrupts, 66 eq, 33 iq
t5nex0: <Chelsio T540-LP-CR> mem 
0xed300000-0xed37ffff,0xec000000-0xecffffff,0xed984000-0xed985fff irq 88 
at device 0.4 numa-domain 1 on pci18
cxl0: <port 0> numa-domain 1 on t5nex0
cxl1: <port 1> numa-domain 1 on t5nex0
cxl2: <port 2> numa-domain 1 on t5nex0
cxl3: <port 3> numa-domain 1 on t5nex0
t5nex0: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq
t5nex1: <Chelsio T540-LP-CR> mem 
0xfb300000-0xfb37ffff,0xfa000000-0xfaffffff,0xfb984000-0xfb985fff irq 98 
at device 0.4 numa-domain 1 on pci20
cxl4: <port 0> numa-domain 1 on t5nex1
cxl5: <port 1> numa-domain 1 on t5nex1
cxl6: <port 2> numa-domain 1 on t5nex1
cxl7: <port 3> numa-domain 1 on t5nex1
t5nex1: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq



Box does have 20 cores and hyper-threading is disabled.
sysctl hw.ncpu
hw.ncpu: 20


I've added updated /boot/loader.conf to the following:

hw.cxgbe.nrxq=16
hw.cxgbe.ntxq=16
hw.cxgbe.qsize_rxq=4096
hw.cxgbe.qsize_txq=4096
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.holdoff_timer_idx=3
machdep.hyperthreading_allowed="0"


Box works fine unless I turn on TX on the cc1 device (I am filtering 
routes learned from that interface via BGP).

Rudy