Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1

[Rudy]

On 2/12/20 7:21 PM, Rudy wrote:
 > I'm having issues with a box that is acting as a BGP router for my 
network.  3 Chelsio cards, two T5 and one T6.  It was working great 
until I turned up our first port on the T6.  It seems like traffic 
passing in from a T5 card and out the T6 causes a really high load (and 
high interrupts).


Looking better!  I made some changes based on BSDRP which I hadn't known 
about -- I think ifqmaxlen was the tunable I overlooked.

# 
https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local
net.link.ifqmaxlen="16384"

Also, I ran chelsio_affinity to bind queues to specific CPU cores.  The 
script only supports a single t5 card, I am revising and will submit a 
patch that will do multiple t5 and t6 cards.


I made both changes at once, and rebooted, so we'll never know which 
fixed it.  ;)



Right now, I have:

#/boot/loader.conf
#
# https://wiki.freebsd.org/10gFreeBSD/Router
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.holdoff_timer_idx=3
# Before FreeBSD 13, threading bad on router: 
https://calomel.org/freebsd_network_tuning.html
machdep.hyperthreading_allowed="0"
hw.cxgbe.nrxq=16
hw.cxgbe.ntxq=16
hw.cxgbe.qsize_rxq=4096
hw.cxgbe.qsize_txq=4096
#hw.cxgbe.pause_settings="0"
# 
https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local

net.link.ifqmaxlen="16384"




#/etc/sysctl.conf
# FRR needs big buffers for OSPF
kern.ipc.maxsockbuf=16777216

# Turn FEC off (doesn't work with Cogent)
dev.cc.0.fec=0
dev.cc.1.fec=0

# Entropy not from LAN ports... slows them down.
kern.random.harvest.mask=65551

net.inet.icmp.icmplim=400
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.blackhole=2          # drop any TCP packets to closed ports
net.inet.tcp.msl=7500             # close lost tcp connections in 7.5 
seconds (default 30)
net.inet.udp.blackhole=1          # drop any UDP packets to closed ports
#
hw.intr_storm_threshold=9000
net.inet.tcp.tso=0