Strange Chelsio performance (T6225-CR)
Date: Mon, 01 Dec 2025 18:01:28 UTC
TL;DR. I can get 10G coming in and out on port 0, Port 1 can only take
traffic at ~ 2.5 Gb/s incoming, but can send out a full 10G.
I have a t6 2 port NIC on a FreeBSD 13 stable server that gets a lot of
backups coming into it via ssh and zfs streams (zrepl). At this point,
I swapped out the NIC, and motherboard and even tried a second NIC but
the same strange 2.5Gb/s incoming results using iperf3 as a test. If I
run iperf3 -c -P5 I get a bit better performance, but still not 10G. On
port 0, its solid 10G in both directions, no dropped packets etc.
port 0 is on a quanta switch
port 1 is a cisco 3850 switch with 4 10G ports. I setup 2 other test
boxes and on the test boxes I can get a full 10G on both ports in a test
environ through the cisco using the same nics and Supermicro MB (H12SSL
and H11SSL). When the H12SSL was my test board, all was good in my
tests. But when I swapped it for the H11SSL that was showing problems, I
had the same issue when I made it live. I have tried different GBICs as
well. Again, the GBICs in the test environment work great in the same
class T6 card
in /boot/loader.conf I have
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.pause_settings="0"
hw.cxgbe.attack_filter="1"
hw.cxgbe.drop_pkts_with_l3_errors="1"
hw.cxgbe.largest_rx_cluster="65536" # 64KB max buffers
hw.cxgbe.safest_rx_cluster="16384" # 16KB default buffers
hw.cxgbe.qsize_rxq="2048" # Double RX queue depth
hw.cxgbe.buffer_packing="1" # Enable buffer packing
hw.cxgbe.fl_pack="1" # Pack multiple frames per buffer
but with all the defaults its the same performance.
sysctl -A dev.cc | egrep "flo|tru"
dev.cc.1.stats.rx_trunc3: 0
dev.cc.1.stats.rx_trunc2: 2
dev.cc.1.stats.rx_trunc1: 0
dev.cc.1.stats.rx_trunc0: 0
dev.cc.1.stats.rx_ovflow3: 0
dev.cc.1.stats.rx_ovflow2: 69
dev.cc.1.stats.rx_ovflow1: 0
dev.cc.1.stats.rx_ovflow0: 0
dev.cc.1.rsrv_noflowq: 0
dev.cc.0.stats.rx_trunc3: 0
dev.cc.0.stats.rx_trunc2: 0
dev.cc.0.stats.rx_trunc1: 0
dev.cc.0.stats.rx_trunc0: 0
dev.cc.0.stats.rx_ovflow3: 0
dev.cc.0.stats.rx_ovflow2: 0
dev.cc.0.stats.rx_ovflow1: 0
dev.cc.0.stats.rx_ovflow0: 0
dev.cc.0.rsrv_noflowq: 0
pciconf -lvcb t6nex0
t6nex0@pci0:193:0:4: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425
device=0x6401 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T6225-CR Unified Wire Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xb7300000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0xb6000000, size 16777216,
enabled
bar [20] = type Memory, range 64, base 0xb7984000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 512(2048) FLR RO NS
max read 4096
link x8(x8) speed 8.0(8.0)
cap 11[b0] = MSI-X supports 256 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[168] = Serial 1 0000000000000000
ecap 000e[188] = ARI 1
ecap 0010[1c8] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 0 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x6801
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[208] = TPH Requester 1
doing an iperf -c -P4 to the problem NIC, I can get a bit more throughput
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.84 GBytes 1.58 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 1.79 GBytes 1.54 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.72 GBytes 1.47 Gbits/sec 0 sender
[ 9] 0.00-10.01 sec 1.72 GBytes 1.47 Gbits/sec receiver
[ 11] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec 0 sender
[ 11] 0.00-10.01 sec 1.79 GBytes 1.54 Gbits/sec receiver
[SUM] 0.00-10.00 sec 7.14 GBytes 6.13 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 7.14 GBytes 6.12 Gbits/sec receiver
otherwise its
0{coldstorage2}# iperf3 -c 192.168.13.254
Connecting to host 192.168.13.254, port 5201
[ 5] local 192.168.13.230 port 38539 connected to 192.168.13.254 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 284 MBytes 2.38 Gbits/sec 0 915 KBytes
[ 5] 1.00-2.00 sec 339 MBytes 2.83 Gbits/sec 0 1.07 MBytes
[ 5] 2.00-3.01 sec 300 MBytes 2.49 Gbits/sec 0 1.29 MBytes
[ 5] 3.01-4.00 sec 303 MBytes 2.58 Gbits/sec 0 1.59 MBytes
[ 5] 4.00-5.00 sec 289 MBytes 2.43 Gbits/sec 0 1.59 MBytes
[ 5] 5.00-6.00 sec 309 MBytes 2.59 Gbits/sec 0 1.59 MBytes
[ 5] 6.00-7.00 sec 288 MBytes 2.41 Gbits/sec 0 1.59 MBytes
[ 5] 7.00-8.00 sec 294 MBytes 2.46 Gbits/sec 0 1.59 MBytes
[ 5] 8.00-9.00 sec 288 MBytes 2.42 Gbits/sec 0 1.59 MBytes
[ 5] 9.00-10.00 sec 287 MBytes 2.41 Gbits/sec 0 1.59 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.91 GBytes 2.50 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.91 GBytes 2.50 Gbits/sec receiver
iperf Done.
0{coldstorage2}#
Asking the LLMs they think resources are not allocated to the ports the
same way which I am a bit suspicious of. This is RELENG_13
hw.cxgbe.tx_coalesce_gap: 5
hw.cxgbe.tx_coalesce_pkts: 32
hw.cxgbe.tx_coalesce: 1
hw.cxgbe.defrags: 0
hw.cxgbe.pullups: 5
hw.cxgbe.lro_mbufs: 0
hw.cxgbe.lro_entries: 8
hw.cxgbe.tscale: 1
hw.cxgbe.safest_rx_cluster: 16384
hw.cxgbe.largest_rx_cluster: 16384
hw.cxgbe.fl_pack: 1
hw.cxgbe.buffer_packing: 1
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.spg_len: 64
hw.cxgbe.fl_pad: -1
hw.cxgbe.fl_pktshift: 0
hw.cxgbe.nm_txcsum: 0
hw.cxgbe.nm_split_rss: 0
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0
hw.cxgbe.tls.combo_wrs: 0
hw.cxgbe.tls.inline_keys: 0
hw.cxgbe.kern_tls: 0
hw.cxgbe.cop_managed_offloading: 0
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 1
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
hw.cxgbe.attack_filter: 1
hw.cxgbe.tx_vm_wr: 0
hw.cxgbe.clock_gate_on_suspend: 0
hw.cxgbe.reset_on_fatal_err: 0
hw.cxgbe.panic_on_fatal_err: 0
hw.cxgbe.pcie_relaxed_ordering: 1
hw.cxgbe.num_vis: 1
hw.cxgbe.doorbells_allowed: 15
hw.cxgbe.fcoecaps_allowed: 0
hw.cxgbe.iscsicaps_allowed: 0
hw.cxgbe.cryptocaps_allowed: -1
hw.cxgbe.rdmacaps_allowed: 0
hw.cxgbe.toecaps_allowed: 1
hw.cxgbe.niccaps_allowed: 33
hw.cxgbe.switchcaps_allowed: 3
hw.cxgbe.linkcaps_allowed: 0
hw.cxgbe.nbmcaps_allowed: 0
hw.cxgbe.fw_install: 1
hw.cxgbe.autoneg: -1
hw.cxgbe.force_fec: -1
hw.cxgbe.fec: -1
hw.cxgbe.pause_settings: 0
hw.cxgbe.config_file: default
hw.cxgbe.interrupt_types: 7
hw.cxgbe.qsize_rxq: 2048
hw.cxgbe.qsize_txq: 1024
hw.cxgbe.holdoff_pktc_idx: -1
hw.cxgbe.holdoff_timer_idx: 1
hw.cxgbe.nnmrxq_vi: 2
hw.cxgbe.nnmtxq_vi: 2
hw.cxgbe.nnmrxq: 8
hw.cxgbe.nnmtxq: 8
hw.cxgbe.native_netmap: 2
hw.cxgbe.holdoff_pktc_idx_ofld: -1
hw.cxgbe.holdoff_timer_idx_ofld: 1
hw.cxgbe.nofldrxq_vi: 1
hw.cxgbe.nofldtxq_vi: 1
hw.cxgbe.nofldrxq: 2
hw.cxgbe.nofldtxq: 8
hw.cxgbe.rsrv_noflowq: 0
hw.cxgbe.nrxq_vi: 1
hw.cxgbe.ntxq_vi: 1
hw.cxgbe.nrxq: 8
hw.cxgbe.ntxq: 16
hw.cxgbe.toe.tls_rx_timeout: 5
hw.cxgbe.toe.rexmt_backoff.15: -1
hw.cxgbe.toe.rexmt_backoff.14: -1
hw.cxgbe.toe.rexmt_backoff.13: -1
hw.cxgbe.toe.rexmt_backoff.12: -1
hw.cxgbe.toe.rexmt_backoff.11: -1
hw.cxgbe.toe.rexmt_backoff.10: -1
hw.cxgbe.toe.rexmt_backoff.9: -1
hw.cxgbe.toe.rexmt_backoff.8: -1
hw.cxgbe.toe.rexmt_backoff.7: -1
hw.cxgbe.toe.rexmt_backoff.6: -1
hw.cxgbe.toe.rexmt_backoff.5: -1
hw.cxgbe.toe.rexmt_backoff.4: -1
hw.cxgbe.toe.rexmt_backoff.3: -1
hw.cxgbe.toe.rexmt_backoff.2: -1
hw.cxgbe.toe.rexmt_backoff.1: -1
hw.cxgbe.toe.rexmt_backoff.0: -1
hw.cxgbe.toe.rexmt_count: 0
hw.cxgbe.toe.rexmt_max: 0
hw.cxgbe.toe.rexmt_min: 0
hw.cxgbe.toe.keepalive_count: 0
hw.cxgbe.toe.keepalive_interval: 0
hw.cxgbe.toe.keepalive_idle: 0
hw.cxgbe.clip_db_auto: 1