Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
Rudy
crapsh at monkeybrains.net
Thu Feb 13 07:05:21 UTC 2020
Load seems to be increasing even though outbound traffic is blocked.
I'm seeing 'Discards In' rise up to about 4k/second when doing 4Gbps
inbound traffic.
I see the tx_pause counter incrementing...
dev.cc.1.stats.tx_pause: 2911586 (~ 20/second increase)
Any tips would be great, I'm stumped.
20 cores seems like plenty to do much higher PPS based on what I see on
the freebsd-net list.
Thanks for any pointers,
Rudy
On 2/12/20 7:21 PM, Rudy wrote:
>
>
> I'm having issues with a box that is acting as a BGP router for my
> network. 3 Chelsio cards, two T5 and one T6. It was working great
> until I turned up our first port on the T6. It seems like traffic
> passing in from a T5 card and out the T6 causes a really high load
> (and high interrupts).
>
> Attached graphs (will that work on a mailing list?) shows the load
> going up.
>
> The work around at this point is that I block inbound routes (out
> bound traffic) on cc1.
>
> I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't
> seem to lower Interrupts.
>
>
> # netstat -rn | wc -l
> 891149
>
> CPU: 0.2% user, 0.0% nice, 0.6% system, 64.4% interrupt, 34.8% idle
>
> Traffic (not that much, right?)
>
> Dev RX bps TX bps RX PPS TX PPS Error
> cc0 0 0 0 0 0
> cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink,
> filtering inbound routes to keep TX low)
> cxl0 287 k 2015 M 353 244 k 0 (our network)
> cxl1 940 M 3115 M 176 k 360 k 0 (our network)
> cxl2 634 M 1014 M 103 k 128 k 0 (our network)
> cxl3 1 k 16 M 1 4 k 0
> cxl4 0 0 0 0 0
> cxl5 0 0 0 0 0
> cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0)
> cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0)
> ixl0 913 k 18 M 0 0 0
> ixl1 1 M 30 M 0 0 0
> lagg0 4019 M 1554 M 491 k 271 k 0
> lagg1 1 M 48 M 0 0 0
> FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg
> 1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827
>
>
> # /boot/loader.conf
> # ZFS!
> kern.geom.label.disk_ident.enable="0"
> kern.geom.label.gptid.enable="0"
> zfs_load="YES"
>
> # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card,
> 40/100Gbps prots) devices
> # built into kernel...
> # if_cxgbe_load="YES"
> # t5fw_cfg_load="YES"
> # t6fw_cfg_load="YES"
>
> # Before FreeBSD 13, threading bad on router:
> https://calomel.org/freebsd_network_tuning.html
> machdep.hyperthreading_allowed="0"
>
> # https://wiki.freebsd.org/10gFreeBSD/Router
> hw.cxgbe.toecaps_allowed="0"
> hw.cxgbe.rdmacaps_allowed="0"
> hw.cxgbe.iscsicaps_allowed="0"
> hw.cxgbe.fcoecaps_allowed="0"
>
>
>
>
> # /etc/sysctl.conf
>
> net.inet6.ip6.redirect=0
> net.inet.ip.redirect=0
>
> # FRR needs big buffers for OSPF
> kern.ipc.maxsockbuf=16777216
>
> # Turn FEC off (no link to Cogent otherwise)
> dev.cc.0.fec=0
> dev.cc.1.fec=0
>
> # Entropy not from LAN ports... slows them down.
> kern.random.harvest.mask=65551
>
> # TODO: try this,
> https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html
> # (Default was 2)
> hw.cxgbe.nm_holdoff_tmr_idx=3
>
> net.inet.icmp.icmplim=400
> net.inet.icmp.maskrepl=0
> net.inet.icmp.log_redirect=0
> net.inet.icmp.drop_redirect=1
> net.inet.tcp.drop_synfin=1
> net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports
> net.inet.tcp.msl=7500 # close lost tcp connections in 7.5
> seconds (default 30)
> net.inet.udp.blackhole=1 # drop any UDP packets to closed ports
>
>
>
>
>
> # vmstat -i
> interrupt total rate
> cpu0:timer 54113958 1102
> cpu1:timer 55286089 1125
> cpu2:timer 53479994 1089
> cpu3:timer 55210439 1124
> cpu4:timer 25989916 529
> cpu5:timer 25908493 527
> cpu6:timer 55187512 1123
> cpu7:timer 54097494 1101
> cpu8:timer 54111676 1101
> cpu9:timer 54541125 1110
> cpu10:timer 55283621 1125
> cpu11:timer 55285466 1125
> cpu12:timer 55283322 1125
> cpu13:timer 55287200 1125
> cpu14:timer 55293560 1126
> cpu15:timer 55286002 1125
> cpu16:timer 55285700 1125
> cpu17:timer 55288330 1125
> cpu18:timer 55292101 1126
> cpu19:timer 55287826 1125
> irq265: xhci0 85784 2
> irq266: ahci1 720421 15
> irq267: ixl0:aq 1 0
> irq268: ixl0:rxq0 8568015 174
> irq269: ixl0:rxq1 10726322 218
> irq270: ixl0:rxq2 7915180 161
> irq271: ixl0:rxq3 8527106 174
> irq272: ixl0:rxq4 13080418 266
> irq273: ixl0:rxq5 7456434 152
> irq274: ixl0:rxq6 8012988 163
> irq275: ixl0:rxq7 8807168 179
> irq276: ixl0:rxq8 9247761 188
> irq277: ixl0:rxq9 8439429 172
> irq278: ixl1:aq 1 0
> irq279: ixl1:rxq0 7750089 158
> irq280: ixl1:rxq1 6604876 134
> irq281: ixl1:rxq2 7180431 146
> irq282: ixl1:rxq3 5990725 122
> irq283: ixl1:rxq4 7385876 150
> irq284: ixl1:rxq5 10252020 209
> irq285: ixl1:rxq6 9155331 186
> irq286: ixl1:rxq7 8384309 171
> irq287: ixl1:rxq8 6642749 135
> irq288: ixl1:rxq9 10019391 204
> irq290: t6nex0:evt 4 0
> irq299: t6nex0:1a0 488060047 9935
> irq300: t6nex0:1a1 379683827 7729
> irq301: t6nex0:1a2 369471291 7521
> irq302: t6nex0:1a3 393993838 8020
> irq303: t6nex0:1a4 354430416 7215
> irq304: t6nex0:1a5 599261919 12199
> irq305: t6nex0:1a6 337249778 6865
> irq306: t6nex0:1a7 511196035 10406
> irq308: t5nex0:evt 8 0
> irq309: t5nex0:0a0 3117499 63
> irq310: t5nex0:0a1 3123964 64
> irq311: t5nex0:0a2 3163023 64
> irq312: t5nex0:0a3 3163675 64
> irq313: t5nex0:0a4 3036876 62
> irq314: t5nex0:0a5 3207249 65
> irq315: t5nex0:0a6 2997717 61
> irq316: t5nex0:0a7 2861439 58
> irq317: t5nex0:1a0 670962265 13658
> irq318: t5nex0:1a1 666875099 13575
> irq319: t5nex0:1a2 679409070 13830
> irq320: t5nex0:1a3 700097218 14251
> irq321: t5nex0:1a4 672477227 13689
> irq322: t5nex0:1a5 674052457 13721
> irq323: t5nex0:1a6 660159886 13438
> irq324: t5nex0:1a7 797020409 16224
> irq325: t5nex0:2a0 412131923 8389
> irq326: t5nex0:2a1 561770787 11435
> irq327: t5nex0:2a2 461510991 9394
> irq328: t5nex0:2a3 420200582 8554
> irq329: t5nex0:2a4 392331601 7986
> irq330: t5nex0:2a5 410021075 8346
> irq331: t5nex0:2a6 467853129 9524
> irq332: t5nex0:2a7 423194899 8615
> irq333: t5nex0:3a0 719985 15
> irq334: t5nex0:3a1 613535 12
> irq335: t5nex0:3a2 635461 13
> irq336: t5nex0:3a3 613540 12
> irq337: t5nex0:3a4 614360 13
> irq338: t5nex0:3a5 613547 12
> irq339: t5nex0:3a6 613539 12
> irq340: t5nex0:3a7 613556 12
> irq342: t5nex1:evt 5 0
> irq359: t5nex1:2a0 378696408 7709
> irq360: t5nex1:2a1 383630463 7809
> irq361: t5nex1:2a2 473624262 9641
> irq362: t5nex1:2a3 391170574 7963
> irq363: t5nex1:2a4 393111484 8002
> irq364: t5nex1:2a5 428581285 8724
> irq365: t5nex1:2a6 380367906 7743
> irq366: t5nex1:2a7 382910534 7794
> irq367: t5nex1:3a0 379035068 7716
> irq368: t5nex1:3a1 381198007 7760
> irq369: t5nex1:3a2 379586223 7727
> irq370: t5nex1:3a3 391500901 7969
> irq371: t5nex1:3a4 411892536 8384
> irq372: t5nex1:3a5 395213950 8045
> irq373: t5nex1:3a6 385156810 7840
> irq374: t5nex1:3a7 380410814 7744
> Total 20060964625 408359
>
> Thanks for any tips,
> Rudy
>
>
>
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list