Re: Strange Chelsio performance (T6225-CR)
- In reply to: mike tancsa : "Re: Strange Chelsio performance (T6225-CR)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 04 Dec 2025 17:45:01 UTC
On 12/2/2025 12:09 PM, mike tancsa wrote: > On 12/2/2025 11:49 AM, Navdeep Parhar wrote: >> On 12/1/25 10:01 AM, mike tancsa wrote: >>> TL;DR. I can get 10G coming in and out on port 0, Port 1 can only >>> take traffic at ~ 2.5 Gb/s incoming, but can send out a full 10G. >>> >> Have you tried your tests with these two left to their default >> values? We don't have a 64KB allocator in the kernel (16KB is the >> max) so the first setting isn't even valid. > > Hi Navdeep, > > Thank you for looking! > > Swapped ports and the problem follows the config, not the hardware. Before the "slow" port was cc1 and the fast cc0. I swapped them around so that cc0 was doing what cc1 was and vice versa. The "slow" port is now cc0 and cc1 which had a hard time keeping up was dropping packets :( This is all with pf enabled. Whats odd is if I disable pf, it doubles the performance on the problem port to 5Gb. With or without pf the "fast" port is always fast. Next step. I installed a Connex4 mce NIC and what tested out a full 10G in both directions, is also not performing at full speed?!?!?! Testing this card with the same CPU and motherboard on a test box gives a full 10G on both ports?! Other than kern.ipc.maxsockbuf=85097152 /etc/sysctl.conf doesnt have any network perf valuessecurity.bsd.see_other_uids=0 security.bsd.see_other_gids=0 security.bsd.unprivileged_read_msgbuf=0 security.bsd.unprivileged_proc_debug=0 vfs.zfs.arc.max=89334498304 #vfs.zfs.arc.meta_limit=19333624576 #vfs.zfs.abd_scatter_enabled=0 vfs.nfs.enable_uidtostring=1 vfs.nfsd.enable_stringtouid=1 and loader.conf is hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.pause_settings="0" hw.cxgbe.attack_filter="1" hw.cxgbe.drop_pkts_with_l3_errors="1" dev.mce.1.conf.channels_rsss: 1 dev.mce.0.conf.channels_rsss: 1 dev.cc.1.rss_size: 128 dev.cc.1.rss_base: 128 dev.cc.0.rss_size: 128 dev.cc.0.rss_base: 0 irq157: mpr0 4882056 1390 irq158: nvme0:admin 206 0 irq159: nvme0:io0 20 0 irq160: nvme0:io1 21 0 irq163: nvme0:io4 78 0 irq164: nvme0:io5 81 0 irq175: xhci0 55014 16 irq177: t6nex0:evt 4 0 irq178: t6nex0:0a0 695005 198 irq179: t6nex0:0a1 1221879 348 irq180: t6nex0:0a2 518121 148 irq181: t6nex0:0a3 1134062 323 irq182: t6nex0:0a4 1254544 357 irq183: t6nex0:0a5 501704 143 irq184: t6nex0:0a6 776666 221 irq185: t6nex0:0a7 517170 147 irq188: t6nex0:1a0 61742 18 irq189: t6nex0:1a1 926650 264 irq190: t6nex0:1a2 1104128 314 irq191: t6nex0:1a3 4927 1 irq192: t6nex0:1a4 724290 206 irq193: t6nex0:1a5 77998 22 irq194: t6nex0:1a6 1636306 466 irq195: t6nex0:1a7 2955293 842 irq200: xhci3 12555 4 irq217: ahci3:ch0 137849 39 irq218: ahci3:ch1 139450 40 irq233: mlx5_core0 1 0 irq234: mlx5_core0 143358 41 irq235: mlx5_core0 5 0 irq236: mlx5_core0 1 0 irq238: mlx5_core0 33 0 irq245: mlx5_core0 2 0 irq246: mlx5_core0 1 0 irq257: mlx5_core0 133 0 irq267: mlx5_core0 1 0 irq268: mlx5_core1 1 0 irq269: mlx5_core1 219866 63 irq270: mlx5_core1 8 0 irq271: mlx5_core1 1 0 irq274: mlx5_core1 19 0 irq275: mlx5_core1 4085 1 irq278: mlx5_core1 19 0 irq280: mlx5_core1 2 0 irq281: mlx5_core1 2 0 irq289: mlx5_core1 55 0 irq292: mlx5_core1 19 0 irq301: mlx5_core1 22 0 irq302: mlx5_core1 4 0