From nobody Tue Dec 02 17:09:12 2025 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dLS1C06y0z6Hnjt for ; Tue, 02 Dec 2025 17:09:19 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dLS1B5DYLz3Tt2 for ; Tue, 02 Dec 2025 17:09:18 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.18.1/8.18.1) with ESMTPS id 5B2H9DcR078932 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Tue, 2 Dec 2025 12:09:13 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:902f:fc48:8e57:8ed9] ([IPv6:2607:f3e0:0:4:902f:fc48:8e57:8ed9]) by pyroxene2a.sentex.ca (8.18.1/8.15.2) with ESMTPS id 5B2H9CB0087302 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Tue, 2 Dec 2025 12:09:12 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: Date: Tue, 2 Dec 2025 12:09:12 -0500 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-questions@freebsd.org Sender: owner-freebsd-questions@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Strange Chelsio performance (T6225-CR) To: Navdeep Parhar , FreeBSD Questions References: <4503cca3-dd1d-4bd0-a82c-f9c2866f468a@sentex.net> Content-Language: en-US From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dLS1B5DYLz3Tt2 On 12/2/2025 11:49 AM, Navdeep Parhar wrote: > On 12/1/25 10:01 AM, mike tancsa wrote: >> TL;DR. I can get 10G coming in and out on port 0, Port 1 can only >> take traffic at ~ 2.5 Gb/s incoming, but can send out a full 10G. >> > Have you tried your tests with these two left to their default values? > We don't have a 64KB allocator in the kernel (16KB is the max) so the > first setting isn't even valid. Hi Navdeep,     Thank you for looking! Yes, I tried with no settings in /boot/loader.conf and same behaviour.  I was running with hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.pause_settings="0" hw.cxgbe.attack_filter="1" hw.cxgbe.drop_pkts_with_l3_errors="1" before as this is just an nfs server, zrepl (receive zfs streams over tcp) and inbound sftp connections.  cc0 is the busiest interface and regularly bursts close to full speed. cc0 has an mtu of 9000. cc1 has the default MTU of 1500 on 4 different vlan interfaces. I also tried to blast some udp packets across both links as a test and cc1 also underperforms with some dropped packets (rx_ovflow2,rx_trunc2) where as the same rate is 100% error free on cc0 > > Please also provide the output of > > # sysctl dev.cc | egrep 'link|caps|pause'  sysctl dev.cc | egrep 'link|caps|pause' dev.cc.1.stats.rx_pause: 0 dev.cc.1.stats.tx_pause: 0 dev.cc.1.lpacaps: 458752 dev.cc.1.acaps: 268435460 dev.cc.1.pcaps: 269418502 dev.cc.1.rcaps: 270532612 dev.cc.1.link_fec: 4 dev.cc.1.pause_settings: 0 dev.cc.1.linkdnrc: n/a dev.cc.0.stats.rx_pause: 0 dev.cc.0.stats.tx_pause: 0 dev.cc.0.lpacaps: 458752 dev.cc.0.acaps: 268435460 dev.cc.0.pcaps: 269418502 dev.cc.0.rcaps: 270532612 dev.cc.0.link_fec: 4 dev.cc.0.pause_settings: 0 dev.cc.0.linkdnrc: n/a > > It is possible that the link on the good port has pause based flow > control enabled and the other one doesn't. On the quanta where cc0 is connected (the one without the slowdowns) the pause counters dont seem to be incrementing. I just did a test and it was 802.3x Pause Frames Received................... 7939772 before and after. Same with the problem port on the cisco that cc1 is on cisco-core4#show int te1/1/1 | inc pau      0 watchdog, 31 multicast, 424569 pause input      0 lost carrier, 0 no carrier, 0 pause output cisco-core4#show int te1/1/1 | inc pau      0 watchdog, 31 multicast, 424569 pause input      0 lost carrier, 0 no carrier, 0 pause output cisco-core4# cisco-core4#show int te1/1/4 | inc pau      0 watchdog, 88 multicast, 0 pause input      0 lost carrier, 0 no carrier, 0 pause output cisco-core4# Nothing special about the config interface TenGigabitEthernet1/1/1  description backup4-cc1  switchport trunk allowed vlan 2-4,223  switchport mode trunk  spanning-tree portfast trunk (cc0 interface) (RackA-10G) #show interface ethernet 0/23 Total Packets Received (Octets)................ 164104553985987 Packets Received 64 Octets..................... 2785340519 Packets Received 65-127 Octets................. 3551885604 Packets Received 128-255 Octets................ 2720325376 Packets Received 256-511 Octets................ 2916456887 Packets Received 512-1023 Octets............... 3987305743 Packets Received 1024-1518 Octets.............. 516438495 Packets Received > 1518 Octets................. 0 Packets RX and TX 64 Octets.................... 4127650421 Packets RX and TX 65-127 Octets................ 2635703067 Packets RX and TX 128-255 Octets............... 2696715779 Packets RX and TX 256-511 Octets............... 1789991928 Packets RX and TX 512-1023 Octets.............. 1073841812 Packets RX and TX 1024-1518 Octets............. 3415095303 Packets RX and TX 1519-2047 Octets............. 1866698571 Packets RX and TX 2048-4095 Octets............. 2166348155 Packets RX and TX 4096-9216 Octets............. 2969664207 Total Packets Received Without Errors.......... 1204638026215 Unicast Packets Received....................... 1204629504771 Multicast Packets Received..................... 7939833 Broadcast Packets Received..................... 581611 --More-- or (q)uit Total Packets Received with MAC Errors......... 0 Jabbers Received............................... 0 Fragments Received............................. 0 Undersize Received............................. 0 Alignment Errors............................... 0 FCS Errors..................................... 0 Overruns....................................... 0 Total Received Packets Not Forwarded........... 7939796 802.3x Pause Frames Received................... 7939772 Unacceptable Frame Type........................ 3 Total Packets Transmitted (Octets)............. 253100014214248 Packets Transmitted 64 Octets.................. 1342309902 Packets Transmitted 65-127 Octets.............. 3378784759 Packets Transmitted 128-255 Octets............. 4271357699 Packets Transmitted 256-511 Octets............. 3168502337 Packets Transmitted 512-1023 Octets............ 1381503365 Packets Transmitted 1024-1518 Octets........... 2898656808 Packets Transmitted > 1518 Octets.............. 0 Max Frame Size................................. 9216 Total Packets Transmitted Successfully......... 828148377556 Unicast Packets Transmitted.................... 828055172598 Multicast Packets Transmitted.................. 92093622 Broadcast Packets Transmitted.................. 1111336 Total Transmit Errors.......................... 0 Total Transmit Packets Discarded............... 0 Single Collision Frames........................ 0 Multiple Collision Frames...................... 0 Excessive Collision Frames..................... 0 802.3x Pause Frames Transmitted................ 0 Time Since Counters Last Cleared............... 1768 day 23 hr 31 min 49 sec > > Regards, > Navdeep >