From nobody Mon Jun 13 19:17:09 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B7AF3851027 for ; Mon, 13 Jun 2022 19:17:14 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LMLt16J5tz3lq9 for ; Mon, 13 Jun 2022 19:17:13 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1655147830; cv=none; d=zohomail.com; s=zohoarc; b=RntFaSHZXDybgPQA80Fh7cOn02PN6uafJCj6NBKg4GeSk5l8LLoMWcecU0z+cEluDGrPSokGCC5tX4m0NH/fQBhwPXZFgCXOdoJJ23peDLSHwHCZnkYfENubpH/oeakfXFqtpMV8qZ+7QuRBvh60x15SSDOGtqjmjCdQuckpsqE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655147830; h=Content-Type:Cc:Date:From:MIME-Version:Message-ID:Reply-To:Subject:To; bh=agMXMQUhdaMmlyyGwUz19KyqWgbcaaHtO4VJJoq1hYw=; b=TSwR5hIImVG+4bA8obCd8n64EQ2ZmC+SS2FUxu4Of9b1r1iKQhL/ZDrh/LhfneXqq4MA0kJgG527PzoK4bn02rKPNfOC+2Xh+FWAoro83/xjamuu4n0tNEq5m7Ejh5ZEwiByORXRecejB2e591u+JwmQIqJ9TtOSovuMXfGZ7Qw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:cc:message-id:subject:mime-version:content-type:user-agent; b=BdhiM5Y0iZMOXRry4RUbnWXCOUnIbvVVXjnOsutjZFv+RN9DFkNk9EyNPqXhw5UZHzgJl2nMFcCx B9xdKYUYldLzhx4fGcbafQBH3t/tbdTfRzs9ikHsKkYrxYtdBcnQ DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1655147830; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Cc:Cc:Message-Id:Message-Id:Subject:Subject:MIME-Version:Content-Type; bh=agMXMQUhdaMmlyyGwUz19KyqWgbcaaHtO4VJJoq1hYw=; b=o25dwTRyTYwnbmJOWftIJdgslSb2WBcNEiFu2vGy3P14cg3Tca/HwR5txOjqx7TF ub+kr1JEtWWTtz8ei2i3isjQ6Jugc88NKkwo+G9uivGawtI49ubCJV4a3BPefz1QMXB 1PCgf/uaVCiAJEwuaRlChmpgRIQjb8UJFyKTCMSE= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1655147829242142.06154364466965; Mon, 13 Jun 2022 12:17:09 -0700 (PDT) Date: Mon, 13 Jun 2022 15:17:09 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "freebsd-net" Cc: "Santiago Martinez" Message-Id: <1815e7f97d1.121d2386d1208869.8658272274076023822@swiftsmsgateway.com> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3809605_1243039924.1655147829202" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LMLt16J5tz3lq9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=o25dwTRy; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-2.84 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.65)[-0.654]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[zoho.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; FROM_HAS_DN(0.00)[]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_3809605_1243039924.1655147829202 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, No, I do not see any retransmission in Linux (see the forum URL for screens= hots) so I do not think this is a hardware issue. I don't think these cards= have flow control on them. I also do not see any errors, drops, or collisi= ons in netstat -i. It's like the network stack doesnt know what do do initi= ally, it seems to sometimes even out after a few seconds, see below. In Lin= ux I get instant 14.6Gb and it stays that way, with zero retries. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 42022 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 623 MBytes=C2= =A0 5.23 Gbits/sec=C2=A0 171=C2=A0=C2=A0=C2=A0 640 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 613 MBytes=C2= =A0 5.14 Gbits/sec=C2=A0 135=C2=A0=C2=A0=C2=A0 543 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 662 MBytes=C2= =A0 5.55 Gbits/sec=C2=A0 107=C2=A0=C2=A0=C2=A0 471 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 718 MBytes=C2= =A0 6.02 Gbits/sec=C2=A0=C2=A0 32=C2=A0=C2=A0=C2=A0 350 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 709 MBytes=C2= =A0 5.95 Gbits/sec=C2=A0=C2=A0 28=C2=A0=C2=A0=C2=A0 685 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 713 MBytes=C2= =A0 5.98 Gbits/sec=C2=A0=C2=A0 39=C2=A0=C2=A0=C2=A0 603 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 704 MBytes=C2= =A0 5.91 Gbits/sec=C2=A0=C2=A0 95=C2=A0=C2=A0=C2=A0 540 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 716 MBytes=C2= =A0 6.01 Gbits/sec=C2=A0=C2=A0 49=C2=A0=C2=A0=C2=A0 466 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 722 MBytes=C2= =A0 6.06 Gbits/sec=C2=A0 132=C2=A0=C2=A0=C2=A0 752 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 720 MBytes=C2=A0 6.= 04 Gbits/sec=C2=A0=C2=A0 19=C2=A0=C2=A0=C2=A0 649 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 720 MBytes=C2=A0 6.04 Gb= its/sec=C2=A0 267=C2=A0=C2=A0=C2=A0 474 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0=C2=A0 675 MBytes=C2=A0 5.65 Gb= its/sec=C2=A0 138=C2=A0=C2=A0 1.16 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 1.04 GBytes=C2=A0 8.96 Gbits/s= ec=C2=A0 118=C2=A0=C2=A0 1.22 MBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.29 MBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.31 MBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.34 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.34 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.36 MBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.36 MBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.37 MBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.39 MBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.40 MBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.41 MBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.41 MBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.42 MBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.44 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.44 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.44 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.45 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.46 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 31.1 GBytes=C2=A0 8.91 Gb= its/sec=C2=A0 1330=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 31.1 GBytes=C2=A0 8.91 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver Thanks. ---- On Mon, 13 Jun 2022 14:41:05 -0400 Santiago Martinez wrote ---- Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. Hi there, there are a lot of re-transmission there... do you see the same with Linux?=20 Are you seeing any drops or error counters increasing on the switch side?=20 Have you check the sysctl for the card, I never used mellanox, but im pretty sure people here can help you. You can also give it a try disabling control flow. Hope it helps. Santi On 6/13/22 20:25, Mike Jakubik wrote: Hello, I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, however, I am unable to get much more than 6Gb/s when testing with iperf3. The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter) They are connected to a Dell=C2=A0N3224PX-ON switch. Both servers are idle and not in use, with a fresh install of=C2=A0stable/13-ebea872f8, nothing running on them except ssh, sendmail, etc. When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I have tried just about every knob listed in=C2=A0https://calomel.org/freebsd_network_tuning.htm= l=C2=A0with little impact on the performance. The network cards have HW LRO enabled as per the driver documentation (though this only seems to lower IRQ usage with no impact on actual throughput). The same exact servers tested on Linux (fedora 34) produced nearly 3x the performance (see attached screenshots), i was able to get a steady 14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to avg at around 6Gbs but it is very sporadic during the iperf run. I have run out of ideas, any suggestions are welcome. Considering Netflix uses very similar HW and they push 400 Gb/s tells me there is something really wrong here or Netflix isnt sharing all their secret sauce. # ifconfig mce0 mce0: flags=3D8863 metric 0 mtu 1500=20 options=3Dffed07bb =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ether b8:ce:f6:81:df:6a =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet 192.168.10.31 netmask 0xfff= fff00 broadcast 192.168.10.255 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 media: Ethernet 25GBase-CR =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: active =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 nd6 options=3D29 [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 64695 connected to 192.168.10.30 port 5201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 930 MBytes=C2= =A0 7.80 Gbits/sec=C2=A0=C2=A0 62=C2=A0=C2=A0=C2=A0 789 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 942 MBytes=C2= =A0 7.90 Gbits/sec=C2=A0 164=C2=A0=C2=A0=C2=A0 824 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.00 GBytes=C2=A0 8.= 61 Gbits/sec=C2=A0 402=C2=A0=C2=A0=C2=A0 879 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 761 MBytes=C2= =A0 6.39 Gbits/sec=C2=A0=C2=A0 61=C2=A0=C2=A0=C2=A0 588 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2= =A0 6.07 Gbits/sec=C2=A0 220=C2=A0=C2=A0=C2=A0 497 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 723 MBytes=C2= =A0 6.07 Gbits/sec=C2=A0=C2=A0 54=C2=A0=C2=A0=C2=A0 364 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 716 MBytes=C2= =A0 6.01 Gbits/sec=C2=A0 187=C2=A0=C2=A0=C2=A0 682 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2= =A0 6.11 Gbits/sec=C2=A0=C2=A0 86=C2=A0=C2=A0=C2=A0 568 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 761 MBytes=C2= =A0 6.39 Gbits/sec=C2=A0=C2=A0 37=C2=A0=C2=A0=C2=A0 418 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.= 15 Gbits/sec=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0 617 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 734 MBytes=C2=A0 6.16 Gb= its/sec=C2=A0 238=C2=A0=C2=A0=C2=A0 474 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0=C2=A0 736 MBytes=C2=A0 6.17 Gb= its/sec=C2=A0 164=C2=A0=C2=A0=C2=A0 757 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 610 MBytes=C2=A0 5.12 Gb= its/sec=C2=A0 118=C2=A0=C2=A0=C2=A0 579 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.02 GBytes=C2=A0 8.75 Gbits/s= ec=C2=A0 447=C2=A0=C2=A0=C2=A0 449 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2=A0 6.11 Gb= its/sec=C2=A0 132=C2=A0=C2=A0=C2=A0 719 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2=A0 6.07 Gb= its/sec=C2=A0 185=C2=A0=C2=A0=C2=A0 649 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 597 MBytes=C2=A0 5.01 Gb= its/sec=C2=A0 142=C2=A0=C2=A0=C2=A0 570 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 102=C2=A0=C2=A0=C2=A0 484 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0=C2=A0 726 MBytes=C2=A0 6.09 Gb= its/sec=C2=A0=C2=A0 15=C2=A0=C2=A0=C2=A0 569 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 181=C2=A0=C2=A0=C2=A0 527 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 729 MBytes=C2=A0 6.12 Gb= its/sec=C2=A0 118=C2=A0=C2=A0=C2=A0 430 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 116=C2=A0=C2=A0=C2=A0 641 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2=A0 6.10 Gb= its/sec=C2=A0 182=C2=A0=C2=A0=C2=A0 598 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 743 MBytes=C2=A0 6.24 Gb= its/sec=C2=A0 209=C2=A0=C2=A0=C2=A0 614 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0=C2=A0 746 MBytes=C2=A0 6.26 Gb= its/sec=C2=A0=C2=A0 72=C2=A0=C2=A0=C2=A0 758 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0=C2=A0 742 MBytes=C2=A0 6.23 Gb= its/sec=C2=A0 199=C2=A0=C2=A0=C2=A0 675 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0=C2=A0 799 MBytes=C2=A0 6.70 Gb= its/sec=C2=A0 183=C2=A0=C2=A0=C2=A0 542 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 908 MBytes=C2=A0 7.61 Gb= its/sec=C2=A0=C2=A0=C2=A0 7=C2=A0=C2=A0 1.19 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.37 GBytes=C2=A0 11.7 Gbits/s= ec=C2=A0 606=C2=A0=C2=A0 1013 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 11.3 Gbits/s= ec=C2=A0=C2=A0 74=C2=A0=C2=A0 1.02 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 23.7 GBytes=C2=A0 6.79 Gb= its/sec=C2=A0 4771=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 23.7 GBytes=C2=A0 6.79 Gbits/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver I have even tried changing to the RACK TCP stack, only to get slightly better results, however with RACK the amount of retries is nearly 0. [root@db-02 ~]# sysctl net.inet.tcp.functions_default=3Drack=20 net.inet.tcp.functions_default: rack -> rack [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 51894 connected to 192.168.10.30 port 5201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 761 MBytes=C2= =A0 6.38 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 737 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 859 MBytes=C2= =A0 7.21 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 761 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 880 MBytes=C2= =A0 7.38 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 785 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 734 MBytes=C2= =A0 6.16 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 804 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 777 MBytes=C2= =A0 6.52 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 824 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 719 MBytes=C2= =A0 6.03 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 841 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 865 MBytes=C2= =A0 7.26 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 862 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 880 MBytes=C2= =A0 7.38 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 882 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 906 MBytes=C2= =A0 7.60 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 904 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 749 MBytes=C2=A0 6.= 29 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 921 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 798 MBytes=C2=A0 6.69 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0=C2=A0 746 MBytes=C2=A0 6.26 Gb= its/sec=C2=A0 209=C2=A0=C2=A0=C2=A0 772 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 768 MBytes=C2=A0 6.44 Gb= its/sec=C2=A0=C2=A0 35=C2=A0=C2=A0=C2=A0 644 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 948 MBytes=C2=A0 7.95 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 673 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.23 GBytes=C2=A0 10.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 711 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.32 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 748 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 785 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 819 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.30 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 852 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.34 GBytes=C2=A0 11.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 883 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 914 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.36 GBytes=C2=A0 11.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 944 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.33 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 974 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1003 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.30 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.00 MBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.34 GBytes=C2=A0 11.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.32 GBytes=C2=A0 11.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.06 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 957 MBytes=C2=A0 8.03 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 837 MBytes=C2=A0 7.02 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.09 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 729 MBytes=C2=A0 6.11 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 30.6 GBytes=C2=A0 8.77 Gb= its/sec=C2=A0 244=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 30.6 GBytes=C2=A0 8.77 Gbits/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver More data can be found @=C2=A0https://forums.freebsd.org/threads/poor-perfo= rmance-with-stable-13-and-mellanox-connectx-6-mlx5.85460/ Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message. ------=_Part_3809605_1243039924.1655147829202 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Hi,

No= , I do not see any retransmission in Linux (see the forum URL for screensho= ts) so I do not think this is a hardware issue. I don't think these cards h= ave flow control on them. I also do not see any errors, drops, or collision= s in netstat -i. It's like the network stack doesnt know what do do initial= ly, it seems to sometimes even out after a few seconds, see below. In Linux= I get instant 14.6Gb and it stays that way, with zero retries.

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local 192.168= .10.31 port 42022 connected to 192.168.10.30 port 5201
[ ID] = Interval           Transf= er     Bitrate       = ;  Retr  Cwnd
[  5]   0.00-1.00 = ;  sec   623 MBytes  5.23 Gbits/sec  171 &nbs= p;  640 KBytes
[  5]   1.00-2.00 &nb= sp; sec   613 MBytes  5.14 Gbits/sec  135  &n= bsp; 543 KBytes
[  5]   2.00-3.00   = sec   662 MBytes  5.55 Gbits/sec  107   = 471 KBytes
[  5]   3.00-4.00   sec&= nbsp;  718 MBytes  6.02 Gbits/sec   32   = ; 350 KBytes
[  5]   4.00-5.00   sec=    709 MBytes  5.95 Gbits/sec   28  &nbs= p; 685 KBytes
[  5]   5.00-6.00   se= c   713 MBytes  5.98 Gbits/sec   39  &nb= sp; 603 KBytes
[  5]   6.00-7.00   s= ec   704 MBytes  5.91 Gbits/sec   95  &n= bsp; 540 KBytes
[  5]   7.00-8.00   = sec   716 MBytes  6.01 Gbits/sec   49  &= nbsp; 466 KBytes
[  5]   8.00-9.00  = sec   722 MBytes  6.06 Gbits/sec  132   = ; 752 KBytes
[  5]   9.00-10.00  sec = ;  720 MBytes  6.04 Gbits/sec   19    64= 9 KBytes
[  5]  10.00-11.00  sec   7= 20 MBytes  6.04 Gbits/sec  267    474 KBytes
[  5]  11.00-12.00  sec   675 MBytes = ; 5.65 Gbits/sec  138   1.16 MBytes
[  5]=   12.00-13.00  sec  1.04 GBytes  8.96 Gbits/sec  1= 18   1.22 MBytes
[  5]  13.00-14.00 = sec  1.29 GBytes  11.1 Gbits/sec    0  = 1.29 MBytes
[  5]  14.00-15.00  sec  1.2= 9 GBytes  11.1 Gbits/sec    0   1.31 MBytes
[  5]  15.00-16.00  sec  1.29 GBytes = 11.1 Gbits/sec    0   1.34 MBytes
[=   5]  16.00-17.00  sec  1.29 GBytes  11.1 Gbits/se= c    0   1.34 MBytes
[  5] = ; 17.00-18.00  sec  1.29 GBytes  11.1 Gbits/sec  &= nbsp; 0   1.36 MBytes
[  5]  18.00-19.00&= nbsp; sec  1.29 GBytes  11.1 Gbits/sec    0 &= nbsp; 1.36 MBytes
[  5]  19.00-20.00  sec = ; 1.29 GBytes  11.1 Gbits/sec    0   1.37 MBy= tes
[  5]  20.00-21.00  sec  1.29 GBytes&= nbsp; 11.1 Gbits/sec    0   1.39 MBytes
<= div>[  5]  21.00-22.00  sec  1.29 GBytes  11.1 Gbi= ts/sec    0   1.40 MBytes
[  5]=   22.00-23.00  sec  1.29 GBytes  11.1 Gbits/sec &n= bsp;  0   1.41 MBytes
[  5]  23.00-2= 4.00  sec  1.29 GBytes  11.1 Gbits/sec    0&n= bsp;  1.41 MBytes
[  5]  24.00-25.00  sec=   1.29 GBytes  11.1 Gbits/sec    0   1.4= 2 MBytes
[  5]  25.00-26.00  sec  1.29 GB= ytes  11.1 Gbits/sec    0   1.44 MBytes
[  5]  26.00-27.00  sec  1.29 GBytes  11.= 1 Gbits/sec    0   1.44 MBytes
[&nbs= p; 5]  27.00-28.00  sec  1.29 GBytes  11.1 Gbits/sec&nb= sp;   0   1.44 MBytes
[  5]  28= .00-29.00  sec  1.29 GBytes  11.1 Gbits/sec   = ; 0   1.45 MBytes
[  5]  29.00-30.00 = ; sec  1.29 GBytes  11.1 Gbits/sec    0  = ; 1.46 MBytes
- - - - - - - - - - - - - - - - - - - - - - - -= -
[ ID] Interval       &n= bsp;   Transfer     Bitrate   =       Retr
[  5]   0.= 00-30.00  sec  31.1 GBytes  8.91 Gbits/sec  1330 &= nbsp;           sender
[  5]   0.00-30.00  sec  31.1 GBytes&n= bsp; 8.91 Gbits/sec         &n= bsp;        receiver

<= /div>

Thanks.



---- On Mon, 13 Jun 2022 14:41:05 -0400 Santiago Mart= inez <sm@codene= tworks.net> wrote ----



Mike Jakubik

Disclaimer: This e-mail and any attachments are intende= d only for the use of the addressee(s) and may contain information that is = privileged or confidential. If you are not the intended recipient, or respo= nsible for delivering the information to the intended recipient, you are he= reby notified that any dissemination, distribution, printing or copying of = this e-mail and any attachments is strictly prohibited. If this e-mail and = any attachments were received in error, please notify the sender by reply e= -mail and delete the original message.

Hi ther= e, there are a lot of re-transmission there... do you see the same with Linux?

Are you seeing any drops or error cou= nters increasing on the switch side?

Have you check the sysctl for the card, I nev= er used mellanox, but im pretty sure people here can help you.

You can also g= ive it a try disabling control flow.

Hope it helps.

San= ti


On 6/13/22= 20:25, Mike Jakubik wrote:
Hello,

I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, however, I am unable to get much more than 6Gb/s when testing with iperf3.

=
The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)

They are connected to a Dell N3224PX-ON switch. Both servers are idle and not in use, with a fresh install of stable/13-ebea872f8, nothing running on them except ssh, sendmail, etc.

When i test = with iperf3 I am unable to get a higher avg than about 6Gb/s. I have tried just about every knob listed in https://calomel.org/freebsd_network_tuning.html with little impact on the performance. The network cards have HW LRO enabled as per the driver documentation (though this only seems to lower IRQ usage with no impact on actual throughput).

The same exa= ct servers tested on Linux (fedora 34) produced nearly 3x the performance (see attached screenshots), i was able to get a steady 14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to avg at around 6Gbs but it is very sporadic during the iperf run.

I= have run out of ideas, any suggestions are welcome. Considering Netflix uses very similar HW and they push 400 Gb/s tells me there is something really wrong here or Netflix isnt sharing all their secret sauce.

# ifconfig mce0
mce0: flags=3D8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3Dffed07bb<RXCSUM= ,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HW= FILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,TXRTLMT,HWRXTSTMP,NOMAP= ,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT>
 = ;       ether b8:ce:f6:81:df:6a
        inet 192.168.10.31 netmask 0xff= ffff00 broadcast 192.168.10.255
     &n= bsp;  media: Ethernet 25GBase-CR <full-duplex,rxpause,txpause>
  = ;      status: active
  &n= bsp;     nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<= /div>


[root@db-02 ~]# iperf3 -i 1 -t 30 -= c db-01
Connecting to host db-01, port 5201
[&= nbsp; 5] local 192.168.10.31 port 64695 connected to 192.168.10.30 port 5201
[ ID] Interval &nb= sp;         Transfer  &nb= sp;  Bitrate         Retr  Cwnd
[  5]   0.00-1.0= 0   sec   930 MBytes  7.80 Gbits/sec   62    789 KBytes
[  5] = ;  1.00-2.00   sec   942 MBytes  7.90 Gbits/s= ec  164    824 KBytes
[  5]&nbs= p;  2.00-3.00   sec  1.00 GBytes  8.61 Gbits/sec&n= bsp; 402    879 KBytes
[  5]&nbs= p;  3.00-4.00   sec   761 MBytes  6.39 Gbits/= sec   61    588 KBytes
[  5] = ;  4.00-5.00   sec   724 MBytes  6.07 Gbits/s= ec  220    497 KBytes
[  5]&nbs= p;  5.00-6.00   sec   723 MBytes  6.07 Gbits/= sec   54    364 KBytes
[  5] = ;  6.00-7.00   sec   716 MBytes  6.01 Gbits/s= ec  187    682 KBytes
[  5]&nbs= p;  7.00-8.00   sec   728 MBytes  6.11 Gbits/= sec   86    568 KBytes
[  5] = ;  8.00-9.00   sec   761 MBytes  6.39 Gbits/s= ec   37    418 KBytes
[  5] = ;  9.00-10.00  sec   733 MBytes  6.15 Gbits/sec&nb= sp;   8    617 KBytes
[  5] = 10.00-11.00  sec   734 MBytes  6.16 Gbits/sec  238    474 KBytes
[  5]&nbs= p; 11.00-12.00  sec   736 MBytes  6.17 Gbits/sec  164    757 KBytes
[  5]&nbs= p; 12.00-13.00  sec   610 MBytes  5.12 Gbits/sec  118    579 KBytes
[  5]&nbs= p; 13.00-14.00  sec  1.02 GBytes  8.75 Gbits/sec  447    449 KBytes
[  5]&nbs= p; 14.00-15.00  sec   728 MBytes  6.11 Gbits/sec  132    719 KBytes
[  5]&nbs= p; 15.00-16.00  sec   724 MBytes  6.07 Gbits/sec  185    649 KBytes
[  5]&nbs= p; 16.00-17.00  sec   597 MBytes  5.01 Gbits/sec  142    570 KBytes
[  5]&nbs= p; 17.00-18.00  sec   733 MBytes  6.15 Gbits/sec  102    484 KBytes
[  5]&nbs= p; 18.00-19.00  sec   726 MBytes  6.09 Gbits/sec &= nbsp; 15    569 KBytes
[  5] = ; 19.00-20.00  sec   733 MBytes  6.15 Gbits/sec  181    527 KBytes
[  5]&nbs= p; 20.00-21.00  sec   729 MBytes  6.12 Gbits/sec  118    430 KBytes
[  5]&nbs= p; 21.00-22.00  sec   733 MBytes  6.15 Gbits/sec  116    641 KBytes
[  5]&nbs= p; 22.00-23.00  sec   728 MBytes  6.10 Gbits/sec  182    598 KBytes
[  5]&nbs= p; 23.00-24.00  sec   743 MBytes  6.24 Gbits/sec  209    614 KBytes
[  5]&nbs= p; 24.00-25.00  sec   746 MBytes  6.26 Gbits/sec &= nbsp; 72    758 KBytes
[  5] = ; 25.00-26.00  sec   742 MBytes  6.23 Gbits/sec  199    675 KBytes
[  5]&nbs= p; 26.00-27.00  sec   799 MBytes  6.70 Gbits/sec  183    542 KBytes
[  5]&nbs= p; 27.00-28.00  sec   908 MBytes  7.61 Gbits/sec &= nbsp;  7   1.19 MBytes
[  5]  28.0= 0-29.00  sec  1.37 GBytes  11.7 Gbits/sec  606   1013 KBytes
[  5]  29= .00-30.00  sec  1.31 GBytes  11.3 Gbits/sec   74   1.02 MBytes
- - - - - - - - - - = - - - - - - - - - - - - - - -
[ ID] Interval   = ;        Transfer    = ; Bitrate         Retr
[  5]   0.00-30.00  se= c  23.7 GBytes  6.79 Gbits/sec  4771         &nb= sp;   sender
[  5]   0.00-30.00 = ; sec  23.7 GBytes  6.79 Gbits/sec        &nbs= p;         receiver
<= br>

I have even tried changing to the RACK TCP sta= ck, only to get slightly better results, however with RACK the amount of retries is nearly 0.

[= root@db-02 ~]# sysctl net.inet.tcp.functions_default=3Drack
net.inet= .tcp.functions_default: rack -> rack
[root@db-02 ~]# iperf= 3 -i 1 -t 30 -c db-01
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db= -01
Connecting to host db-01, port 5201
[ = 5] local 192.168.10.31 port 51894 connected to 192.168.10.30 port 5201
[ ID] Interval &nb= sp;         Transfer  &nb= sp;  Bitrate         Retr  Cwnd
[  5]   0.00-1.0= 0   sec   761 MBytes  6.38 Gbits/sec  &n= bsp; 0    737 KBytes
[  5] =   1.00-2.00   sec   859 MBytes  7.21 Gbits/se= c    0    761 KBytes
[  5] =   2.00-3.00   sec   880 MBytes  7.38 Gbits/se= c    0    785 KBytes
[  5] =   3.00-4.00   sec   734 MBytes  6.16 Gbits/se= c    0    804 KBytes
[  5] =   4.00-5.00   sec   777 MBytes  6.52 Gbits/se= c    0    824 KBytes
[  5] =   5.00-6.00   sec   719 MBytes  6.03 Gbits/se= c    0    841 KBytes
[  5] =   6.00-7.00   sec   865 MBytes  7.26 Gbits/se= c    0    862 KBytes
[  5] =   7.00-8.00   sec   880 MBytes  7.38 Gbits/se= c    0    882 KBytes
[  5] =   8.00-9.00   sec   906 MBytes  7.60 Gbits/se= c    0    904 KBytes
[  5] =   9.00-10.00  sec   749 MBytes  6.29 Gbits/sec&nbs= p;   0    921 KBytes
[  5] = 10.00-11.00  sec   798 MBytes  6.69 Gbits/sec &nb= sp;  0    938 KBytes
[  5] = 11.00-12.00  sec   746 MBytes  6.26 Gbits/sec  209    772 KBytes
[  5]&nbs= p; 12.00-13.00  sec   768 MBytes  6.44 Gbits/sec &= nbsp; 35    644 KBytes
[  5] = ; 13.00-14.00  sec   948 MBytes  7.95 Gbits/sec &n= bsp;  0    673 KBytes
[  5] = 14.00-15.00  sec  1.23 GBytes  10.5 Gbits/sec  &n= bsp; 0    711 KBytes
[  5] = 15.00-16.00  sec  1.32 GBytes  11.4 Gbits/sec  &n= bsp; 0    748 KBytes
[  5] = 16.00-17.00  sec  1.31 GBytes  11.2 Gbits/sec  &n= bsp; 0    785 KBytes
[  5] = 17.00-18.00  sec  1.29 GBytes  11.1 Gbits/sec  &n= bsp; 0    819 KBytes
[  5] = 18.00-19.00  sec  1.30 GBytes  11.2 Gbits/sec  &n= bsp; 0    852 KBytes
[  5] = 19.00-20.00  sec  1.34 GBytes  11.5 Gbits/sec  &n= bsp; 0    883 KBytes
[  5] = 20.00-21.00  sec  1.29 GBytes  11.1 Gbits/sec  &n= bsp; 0    914 KBytes
[  5] = 21.00-22.00  sec  1.36 GBytes  11.7 Gbits/sec  &n= bsp; 0    944 KBytes
[  5] = 22.00-23.00  sec  1.33 GBytes  11.4 Gbits/sec  &n= bsp; 0    974 KBytes
[  5] = 23.00-24.00  sec  1.31 GBytes  11.2 Gbits/sec  &n= bsp; 0   1003 KBytes
[  5]  24.0= 0-25.00  sec  1.30 GBytes  11.2 Gbits/sec    0   1.00 MBytes
[  5]  25.0= 0-26.00  sec  1.34 GBytes  11.5 Gbits/sec    0   1.03 MBytes
[  5]  26.0= 0-27.00  sec  1.32 GBytes  11.3 Gbits/sec    0   1.06 MBytes
[  5]  27.0= 0-28.00  sec   957 MBytes  8.03 Gbits/sec  &n= bsp; 0   1.07 MBytes
[  5]  28.0= 0-29.00  sec   837 MBytes  7.02 Gbits/sec  &n= bsp; 0   1.09 MBytes
[  5]  29.0= 0-30.00  sec   729 MBytes  6.11 Gbits/sec  &n= bsp; 0   1.10 MBytes
- - - - - - - - - - -= - - - - - - - - - - - - - -
[ ID] Interval   =         Transfer    = Bitrate         Retr
[  5]   0.00-30.00  se= c  30.6 GBytes  8.77 Gbits/sec  244         &nbs= p;   sender
[  5]   0.00-30.00 = sec  30.6 GBytes  8.77 Gbits/sec        &nbs= p;         receiver
<= br>


More data can be found @ <= a target=3D"_blank" href=3D"https://forums.freebsd.org/threads/poor-perform= ance-with-stable-13-and-mellanox-connectx-6-mlx5.85460/" class=3D"x_-177277= 7228moz-txt-link-freetext">https://forums.freebsd.org/threads/poor-performa= nce-with-stable-13-and-mellanox-connectx-6-mlx5.85460/


Mike Jakubik

Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message.





------=_Part_3809605_1243039924.1655147829202--