Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
Rudy
crapsh at monkeybrains.net
Thu Feb 13 03:21:34 UTC 2020
I'm having issues with a box that is acting as a BGP router for my
network. 3 Chelsio cards, two T5 and one T6. It was working great
until I turned up our first port on the T6. It seems like traffic
passing in from a T5 card and out the T6 causes a really high load (and
high interrupts).
Attached graphs (will that work on a mailing list?) shows the load going
up.
The work around at this point is that I block inbound routes (out bound
traffic) on cc1.
I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't
seem to lower Interrupts.
# netstat -rn | wc -l
891149
CPU: 0.2% user, 0.0% nice, 0.6% system, 64.4% interrupt, 34.8% idle
Traffic (not that much, right?)
Dev RX bps TX bps RX PPS TX PPS Error
cc0 0 0 0 0 0
cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink,
filtering inbound routes to keep TX low)
cxl0 287 k 2015 M 353 244 k 0 (our network)
cxl1 940 M 3115 M 176 k 360 k 0 (our network)
cxl2 634 M 1014 M 103 k 128 k 0 (our network)
cxl3 1 k 16 M 1 4 k 0
cxl4 0 0 0 0 0
cxl5 0 0 0 0 0
cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0)
cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0)
ixl0 913 k 18 M 0 0 0
ixl1 1 M 30 M 0 0 0
lagg0 4019 M 1554 M 491 k 271 k 0
lagg1 1 M 48 M 0 0 0
FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg
1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827
# /boot/loader.conf
# ZFS!
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
# Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card,
40/100Gbps prots) devices
# built into kernel...
# if_cxgbe_load="YES"
# t5fw_cfg_load="YES"
# t6fw_cfg_load="YES"
# Before FreeBSD 13, threading bad on router:
https://calomel.org/freebsd_network_tuning.html
machdep.hyperthreading_allowed="0"
# https://wiki.freebsd.org/10gFreeBSD/Router
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
# /etc/sysctl.conf
net.inet6.ip6.redirect=0
net.inet.ip.redirect=0
# FRR needs big buffers for OSPF
kern.ipc.maxsockbuf=16777216
# Turn FEC off (no link to Cogent otherwise)
dev.cc.0.fec=0
dev.cc.1.fec=0
# Entropy not from LAN ports... slows them down.
kern.random.harvest.mask=65551
# TODO: try this,
https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html
# (Default was 2)
hw.cxgbe.nm_holdoff_tmr_idx=3
net.inet.icmp.icmplim=400
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports
net.inet.tcp.msl=7500 # close lost tcp connections in 7.5
seconds (default 30)
net.inet.udp.blackhole=1 # drop any UDP packets to closed ports
# vmstat -i
interrupt total rate
cpu0:timer 54113958 1102
cpu1:timer 55286089 1125
cpu2:timer 53479994 1089
cpu3:timer 55210439 1124
cpu4:timer 25989916 529
cpu5:timer 25908493 527
cpu6:timer 55187512 1123
cpu7:timer 54097494 1101
cpu8:timer 54111676 1101
cpu9:timer 54541125 1110
cpu10:timer 55283621 1125
cpu11:timer 55285466 1125
cpu12:timer 55283322 1125
cpu13:timer 55287200 1125
cpu14:timer 55293560 1126
cpu15:timer 55286002 1125
cpu16:timer 55285700 1125
cpu17:timer 55288330 1125
cpu18:timer 55292101 1126
cpu19:timer 55287826 1125
irq265: xhci0 85784 2
irq266: ahci1 720421 15
irq267: ixl0:aq 1 0
irq268: ixl0:rxq0 8568015 174
irq269: ixl0:rxq1 10726322 218
irq270: ixl0:rxq2 7915180 161
irq271: ixl0:rxq3 8527106 174
irq272: ixl0:rxq4 13080418 266
irq273: ixl0:rxq5 7456434 152
irq274: ixl0:rxq6 8012988 163
irq275: ixl0:rxq7 8807168 179
irq276: ixl0:rxq8 9247761 188
irq277: ixl0:rxq9 8439429 172
irq278: ixl1:aq 1 0
irq279: ixl1:rxq0 7750089 158
irq280: ixl1:rxq1 6604876 134
irq281: ixl1:rxq2 7180431 146
irq282: ixl1:rxq3 5990725 122
irq283: ixl1:rxq4 7385876 150
irq284: ixl1:rxq5 10252020 209
irq285: ixl1:rxq6 9155331 186
irq286: ixl1:rxq7 8384309 171
irq287: ixl1:rxq8 6642749 135
irq288: ixl1:rxq9 10019391 204
irq290: t6nex0:evt 4 0
irq299: t6nex0:1a0 488060047 9935
irq300: t6nex0:1a1 379683827 7729
irq301: t6nex0:1a2 369471291 7521
irq302: t6nex0:1a3 393993838 8020
irq303: t6nex0:1a4 354430416 7215
irq304: t6nex0:1a5 599261919 12199
irq305: t6nex0:1a6 337249778 6865
irq306: t6nex0:1a7 511196035 10406
irq308: t5nex0:evt 8 0
irq309: t5nex0:0a0 3117499 63
irq310: t5nex0:0a1 3123964 64
irq311: t5nex0:0a2 3163023 64
irq312: t5nex0:0a3 3163675 64
irq313: t5nex0:0a4 3036876 62
irq314: t5nex0:0a5 3207249 65
irq315: t5nex0:0a6 2997717 61
irq316: t5nex0:0a7 2861439 58
irq317: t5nex0:1a0 670962265 13658
irq318: t5nex0:1a1 666875099 13575
irq319: t5nex0:1a2 679409070 13830
irq320: t5nex0:1a3 700097218 14251
irq321: t5nex0:1a4 672477227 13689
irq322: t5nex0:1a5 674052457 13721
irq323: t5nex0:1a6 660159886 13438
irq324: t5nex0:1a7 797020409 16224
irq325: t5nex0:2a0 412131923 8389
irq326: t5nex0:2a1 561770787 11435
irq327: t5nex0:2a2 461510991 9394
irq328: t5nex0:2a3 420200582 8554
irq329: t5nex0:2a4 392331601 7986
irq330: t5nex0:2a5 410021075 8346
irq331: t5nex0:2a6 467853129 9524
irq332: t5nex0:2a7 423194899 8615
irq333: t5nex0:3a0 719985 15
irq334: t5nex0:3a1 613535 12
irq335: t5nex0:3a2 635461 13
irq336: t5nex0:3a3 613540 12
irq337: t5nex0:3a4 614360 13
irq338: t5nex0:3a5 613547 12
irq339: t5nex0:3a6 613539 12
irq340: t5nex0:3a7 613556 12
irq342: t5nex1:evt 5 0
irq359: t5nex1:2a0 378696408 7709
irq360: t5nex1:2a1 383630463 7809
irq361: t5nex1:2a2 473624262 9641
irq362: t5nex1:2a3 391170574 7963
irq363: t5nex1:2a4 393111484 8002
irq364: t5nex1:2a5 428581285 8724
irq365: t5nex1:2a6 380367906 7743
irq366: t5nex1:2a7 382910534 7794
irq367: t5nex1:3a0 379035068 7716
irq368: t5nex1:3a1 381198007 7760
irq369: t5nex1:3a2 379586223 7727
irq370: t5nex1:3a3 391500901 7969
irq371: t5nex1:3a4 411892536 8384
irq372: t5nex1:3a5 395213950 8045
irq373: t5nex1:3a6 385156810 7840
irq374: t5nex1:3a7 380410814 7744
Total 20060964625 408359
Thanks for any tips,
Rudy
More information about the freebsd-net
mailing list