From nobody Fri Jun 17 00:38:55 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 495C7857D90 for ; Fri, 17 Jun 2022 00:39:01 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LPKsw1m9kz4YRJ for ; Fri, 17 Jun 2022 00:39:00 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1655426336; cv=none; d=zohomail.com; s=zohoarc; b=jrmfKJM0dTKJHEOWUkpDaechuMec2IdyBGwlV78S37jRImXekw5iw2zHPTcpoh6aKs9JumnJT/30YQ7Wjz1bX8WyDGrW4cm6DZNeGvwOEplzxG3ed0iX7jIXCRmRHI7AcGMmKOyueP+Glxw4ggGw7RFlw5yVqJRvjnAhQ7dHus4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655426336; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:References:Subject:To; bh=KI3MPq1t42Xq1NANDkbUeNy5PsBZpYsIZJNh5ipYjRc=; b=J0xqvnhjzVXRU6Gv9vc96i3p2tkKSIHcD4giT0TAw+dtt6yYR7L594O6xpoWPAFDNFwGl9IgcwPmmMMDmFroe5ahBYRgDxBdYkzyw1yr5CS2lgKsit5Nd/OiqE+ak0S03UmlXWqBMlEB+0LCnYb/J7sVjBmZLCDx3mvkGGS6N4c= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:cc:message-id:in-reply-to:references:subject:mime-version:content-type:user-agent; b=uCI6JMUS6FwK4qXlxHrfjsALnEWCCDE1dyvFKh4nnZ3Lps1e9Fuc4Qp/IrqtZyNSepLKt99SBrKq 4iBlqX/V7hLR9T2pV7IM36pSprvw1IFjRLpLJAVqoG5d8p3J/cJP DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1655426336; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type; bh=KI3MPq1t42Xq1NANDkbUeNy5PsBZpYsIZJNh5ipYjRc=; b=Lf73nI9mdyZcKKuaepajfyuoGYUXNp7gJ8+Hf/le8jQgMdnZGpvsXmDAMEwnHNFJ H/xPdsbtNT0QXBts/cNn3v8zv9K4g9J0t3U1/3A8Lon95EXW/mfwhezt2za1Bg2a+tb UKfLtaQbcsPkQutNvAvQ9kVXReUtqmWJVjqOZ/SU= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1655426335114597.1011974352571; Thu, 16 Jun 2022 17:38:55 -0700 (PDT) Date: Thu, 16 Jun 2022 20:38:55 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "freebsd-net" Cc: "Alexander V. Chernikov" Message-Id: <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> In-Reply-To: References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_6963461_1051991644.1655426335083" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LPKsw1m9kz4YRJ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=Lf73nI9m; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-4.00 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.83)[-0.830]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[zoho.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; NEURAL_HAM_MEDIUM(-0.98)[-0.977]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; FROM_HAS_DN(0.00)[]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_6963461_1051991644.1655426335083 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I believe you hit the nail on the head! I am now getting consistent high sp= eeds, even higher than on Linux! Is this a problem with the scheduler? Shou= ld someone in that area of expertise be made aware of this? More importantl= y i guess, would this affect real world performance, these servers will be = running RabbitMQ (it uses quite a bit of bandwidth) and PostgreSQL w/ repli= cation. [root@db-01 ~]# cpuset -l 1 iperf3 -s=20 ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 22445 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 2.17 GBytes=C2=A0 18= .7 Gbits/sec=C2=A0 263=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 2.11 GBytes=C2=A0 18= .1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.14 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.46 GBytes=C2=A0 21= .2 Gbits/sec=C2=A0 696=C2=A0=C2=A0=C2=A0 893 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.49 GBytes=C2=A0 21= .4 Gbits/sec=C2=A0 857=C2=A0=C2=A0=C2=A0 447 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.36 GBytes=C2=A0 20= .3 Gbits/sec=C2=A0 281=C2=A0=C2=A0=C2=A0 566 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.36 GBytes=C2=A0 20= .3 Gbits/sec=C2=A0 348=C2=A0=C2=A0=C2=A0 593 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 2.28 GBytes=C2=A0 19= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 694 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 2.20 GBytes=C2=A0 18= .9 Gbits/sec=C2=A0 165=C2=A0=C2=A0=C2=A0 611 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19= .5 Gbits/sec=C2=A0 183=C2=A0=C2=A0=C2=A0 541 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19.4 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 625 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 2.27 GBytes=C2=A0 19.5 Gbits/s= ec=C2=A0 165=C2=A0=C2=A0=C2=A0 608 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 2.22 GBytes=C2=A0 19.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 679 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.22 GBytes=C2=A0 19.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 726 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0 174=C2=A0=C2=A0=C2=A0 610 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 2.19 GBytes=C2=A0 18.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 644 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 2.20 GBytes=C2=A0 18.9 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 704 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.18 GBytes=C2=A0 18.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 784 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.18 GBytes=C2=A0 18.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 820 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.19 GBytes=C2=A0 18.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 874 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.19 GBytes=C2=A0 18.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 902 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 2.23 GBytes=C2=A0 19.1 Gbits/s= ec=C2=A0 202=C2=A0=C2=A0=C2=A0 661 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 171=C2=A0=C2=A0=C2=A0 894 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 2.33 GBytes=C2=A0 20.0 Gbits/s= ec=C2=A0 341=C2=A0=C2=A0=C2=A0 409 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 167=C2=A0=C2=A0=C2=A0 594 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 2.28 GBytes=C2=A0 19.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 714 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 166=C2=A0=C2=A0=C2=A0 604 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 705 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.86 GBytes=C2=A0 16.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 763 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 816 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 2.24 GBytes=C2=A0 19.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 869 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0 4179=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver iperf Done. [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 51998 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.99 GBytes=C2=A0 17= .1 Gbits/sec=C2=A0 276=C2=A0=C2=A0=C2=A0 456 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 2.00 GBytes=C2=A0 17= .2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 593 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.27 GBytes=C2=A0 19= .5 Gbits/sec=C2=A0 166=C2=A0=C2=A0=C2=A0 556 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19= .8 Gbits/sec=C2=A0 186=C2=A0=C2=A0=C2=A0 558 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19= .8 Gbits/sec=C2=A0 168=C2=A0=C2=A0=C2=A0 558 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19= .6 Gbits/sec=C2=A0 182=C2=A0=C2=A0=C2=A0 545 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19= .7 Gbits/sec=C2=A0 171=C2=A0=C2=A0=C2=A0 507 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19= .7 Gbits/sec=C2=A0 180=C2=A0=C2=A0=C2=A0 517 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19= .7 Gbits/sec=C2=A0 174=C2=A0=C2=A0=C2=A0 568 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 2.28 GBytes=C2=A0 19.6 Gb= its/sec=C2=A0 178=C2=A0=C2=A0=C2=A0 434 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 583 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 180=C2=A0=C2=A0=C2=A0 443 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.32 GBytes=C2=A0 19.9 Gbits/s= ec=C2=A0 166=C2=A0=C2=A0=C2=A0 564 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.35 GBytes=C2=A0 20.2 Gbits/s= ec=C2=A0 350=C2=A0=C2=A0=C2=A0 424 KBytes [=C2=A0 5]=C2=A0 14.00-15.02=C2=A0 sec=C2=A0 1.49 GBytes=C2=A0 12.5 Gbits/s= ec=C2=A0 175=C2=A0=C2=A0 1.41 KBytes [=C2=A0 5]=C2=A0 15.02-16.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.22 GBytes=C2=A0 19.1 Gbits/s= ec=C2=A0 171=C2=A0=C2=A0=C2=A0 784 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.34 GBytes=C2=A0 20.1 Gbits/s= ec=C2=A0 348=C2=A0=C2=A0=C2=A0 436 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.34 GBytes=C2=A0 20.1 Gbits/s= ec=C2=A0 350=C2=A0=C2=A0=C2=A0 547 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19.9 Gbits/s= ec=C2=A0 173=C2=A0=C2=A0=C2=A0 595 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 2.35 GBytes=C2=A0 20.2 Gbits/s= ec=C2=A0 360=C2=A0=C2=A0=C2=A0 439 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.41 GBytes=C2=A0 20.7 Gbits/s= ec=C2=A0 525=C2=A0=C2=A0=C2=A0 580 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 2.43 GBytes=C2=A0 20.9 Gbits/s= ec=C2=A0 510=C2=A0=C2=A0=C2=A0 397 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 2.38 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 532=C2=A0=C2=A0=C2=A0 533 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 2.37 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 344=C2=A0=C2=A0=C2=A0 547 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 2.36 GBytes=C2=A0 20.2 Gbits/s= ec=C2=A0 354=C2=A0=C2=A0=C2=A0 389 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 165=C2=A0=C2=A0=C2=A0 592 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 173=C2=A0=C2=A0=C2=A0 584 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 2.27 GBytes=C2=A0 19.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 701 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 790 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0 6557=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver iperf Done. Thank You! ---- On Thu, 16 Jun 2022 17:00:25 -0400 Alexander V. Chernikov wrote ---- > On 16 Jun 2022, at 21:48, Mike Jakubik wrote:=20 >=20 > After multiple tests and tweaks i believe the issue is not with the HW or= Numa related (Infinity fabric should do around 32GB) but rather with FreeB= SD TCP/IP stack. It's like it cant figure itself out properly for the speed= that the HW can do, i keep getting widely varying results when testing. Be= low is an example of two tests, with about a 15 second break in between the= two.=20 Does pinning iperf to a specific CPU core (or range) address the variety pa= rt? e.g. cpuset -l 1 perf =E2=80=A6=20 The output you shared above shows CPU#83 as the core iperf is running on. J= ust wondering whether the scheduler migrates iperf too often, thrashing the= caches. ------=_Part_6963461_1051991644.1655426335083 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Hi,

I believe you hit= the nail on the head! I am now getting consistent high speeds, even higher= than on Linux! Is this a problem with the scheduler? Should someone in tha= t area of expertise be made aware of this? More importantly i guess, would = this affect real world performance, these servers will be running RabbitMQ = (it uses quite a bit of bandwidth) and PostgreSQL w/ replication.
=

[root@db-01 ~]# cpuset -l 1 iperf3 -s
--= ---------------------------------------------------------
Ser= ver listening on 5201 (test #1)
-----------------------------= ------------------------------

[root@db-02 ~]#= cpuset -l 1 iperf3 -i 1 -t 30 -c db-01
Connecting to host d= b-01, port 5201
[  5] local 192.168.10.31 port 22445 con= nected to 192.168.10.30 port 5201
[ ID] Interval  &= nbsp;        Transfer   &= nbsp; Bitrate         Retr  Cw= nd
[  5]   0.00-1.00   sec  2.1= 7 GBytes  18.7 Gbits/sec  263   1.13 MBytes
[  5]   1.00-2.00   sec  2.11 GBytes = 18.1 Gbits/sec    0   1.14 MBytes
[=   5]   2.00-3.00   sec  2.46 GBytes  21.= 2 Gbits/sec  696    893 KBytes
[  5]=    3.00-4.00   sec  2.49 GBytes  21.4 Gbits/s= ec  857    447 KBytes
[  5] &nb= sp; 4.00-5.00   sec  2.36 GBytes  20.3 Gbits/sec  = 281    566 KBytes
[  5]   5.00-= 6.00   sec  2.36 GBytes  20.3 Gbits/sec  348 =    593 KBytes
[  5]   6.00-7.00 = ;  sec  2.28 GBytes  19.6 Gbits/sec    0 = ;   694 KBytes
[  5]   7.00-8.00&nbs= p;  sec  2.20 GBytes  18.9 Gbits/sec  165  &n= bsp; 611 KBytes
[  5]   8.00-9.00   = sec  2.26 GBytes  19.5 Gbits/sec  183    541 = KBytes
[  5]   9.00-10.00  sec  2.26= GBytes  19.4 Gbits/sec    0    625 KByt= es
[  5]  10.00-11.00  sec  2.27 GBytes&n= bsp; 19.5 Gbits/sec  165    608 KBytes
[&= nbsp; 5]  11.00-12.00  sec  2.22 GBytes  19.0 Gbits/sec=     0    679 KBytes
[  5]&= nbsp; 12.00-13.00  sec  2.22 GBytes  19.0 Gbits/sec &nb= sp;  0    726 KBytes
[  5]  13.= 00-14.00  sec  2.25 GBytes  19.3 Gbits/sec  174 &n= bsp;  610 KBytes
[  5]  14.00-15.00  sec&= nbsp; 2.19 GBytes  18.8 Gbits/sec    0   = ; 644 KBytes
[  5]  15.00-16.00  sec  2.2= 0 GBytes  18.9 Gbits/sec    0    704 KBy= tes
[  5]  16.00-17.00  sec  2.18 GBytes&= nbsp; 18.7 Gbits/sec    0    784 KBytes
[  5]  17.00-18.00  sec  2.18 GBytes  18.= 7 Gbits/sec    0    820 KBytes
= [  5]  18.00-19.00  sec  2.19 GBytes  18.8 Gbits/s= ec    0    874 KBytes
[  5= ]  19.00-20.00  sec  2.19 GBytes  18.8 Gbits/sec &= nbsp;  0    902 KBytes
[  5]  2= 0.00-21.00  sec  2.23 GBytes  19.1 Gbits/sec  202 =    661 KBytes
[  5]  21.00-22.00  se= c  2.30 GBytes  19.8 Gbits/sec  171    894 KB= ytes
[  5]  22.00-23.00  sec  2.33 GBytes=   20.0 Gbits/sec  341    409 KBytes
= [  5]  23.00-24.00  sec  2.29 GBytes  19.7 Gbits/s= ec  167    594 KBytes
[  5]  24= .00-25.00  sec  2.28 GBytes  19.6 Gbits/sec   = ; 0    714 KBytes
[  5]  25.00-26.00=   sec  2.29 GBytes  19.7 Gbits/sec  166  &nbs= p; 604 KBytes
[  5]  26.00-27.00  sec  2.= 26 GBytes  19.4 Gbits/sec    0    705 KB= ytes
[  5]  27.00-28.00  sec  1.86 GBytes=   16.0 Gbits/sec    0    763 KBytes
<= /div>
[  5]  28.00-29.00  sec  2.25 GBytes  19= .3 Gbits/sec    0    816 KBytes
[  5]  29.00-30.00  sec  2.24 GBytes  19.2 Gbits/= sec    0    869 KBytes
- - - - = - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval = ;          Transfer  = ;   Bitrate         Retr<= br>
[  5]   0.00-30.00  sec  67.4 GBytes=   19.3 Gbits/sec  4179       &= nbsp;     sender
[  5]   0= .00-30.00  sec  67.4 GBytes  19.3 Gbits/sec   = ;            &n= bsp;  receiver

iperf Done.
= [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01
Connec= ting to host db-01, port 5201
[  5] local 192.168.10.31 = port 51998 connected to 192.168.10.30 port 5201
[ ID] Interva= l           Transfer = ;    Bitrate        = Retr  Cwnd
[  5]   0.00-1.00  = sec  1.99 GBytes  17.1 Gbits/sec  276    456= KBytes
[  5]   1.00-2.00   sec = ; 2.00 GBytes  17.2 Gbits/sec    0    59= 3 KBytes
[  5]   2.00-3.00   sec&nbs= p; 2.27 GBytes  19.5 Gbits/sec  166    556 KBytes<= br>
[  5]   3.00-4.00   sec  2.31 G= Bytes  19.8 Gbits/sec  186    558 KBytes
=
[  5]   4.00-5.00   sec  2.31 GBytes&nbs= p; 19.8 Gbits/sec  168    558 KBytes
[&nb= sp; 5]   5.00-6.00   sec  2.29 GBytes  19.6 G= bits/sec  182    545 KBytes
[  5]&nb= sp;  6.00-7.00   sec  2.29 GBytes  19.7 Gbits/sec&= nbsp; 171    507 KBytes
[  5]  = 7.00-8.00   sec  2.29 GBytes  19.7 Gbits/sec  180=     517 KBytes
[  5]   8.00-9.0= 0   sec  2.30 GBytes  19.7 Gbits/sec  174 &nb= sp;  568 KBytes
[  5]   9.00-10.00  = sec  2.28 GBytes  19.6 Gbits/sec  178    434 = KBytes
[  5]  10.00-11.00  sec  2.26 GByt= es  19.4 Gbits/sec    0    583 KBytes
[  5]  11.00-12.00  sec  2.31 GBytes  = 19.8 Gbits/sec  180    443 KBytes
[ = 5]  12.00-13.00  sec  2.32 GBytes  19.9 Gbits/sec = ; 166    564 KBytes
[  5]  13.00-14.= 00  sec  2.35 GBytes  20.2 Gbits/sec  350  &n= bsp; 424 KBytes
[  5]  14.00-15.02  sec  = 1.49 GBytes  12.5 Gbits/sec  175   1.41 KBytes
[  5]  15.02-16.00  sec  1.73 GBytes  15.2 G= bits/sec    0   1.12 MBytes
[  = 5]  16.00-17.00  sec  2.22 GBytes  19.1 Gbits/sec = 171    784 KBytes
[  5]  17.00-18.0= 0  sec  2.34 GBytes  20.1 Gbits/sec  348  &nb= sp; 436 KBytes
[  5]  18.00-19.00  sec  2= .34 GBytes  20.1 Gbits/sec  350    547 KBytes
<= /div>
[  5]  19.00-20.00  sec  2.31 GBytes  19= .9 Gbits/sec  173    595 KBytes
[  5= ]  20.00-21.00  sec  2.35 GBytes  20.2 Gbits/sec  = 360    439 KBytes
[  5]  21.00-22.00=   sec  2.41 GBytes  20.7 Gbits/sec  525  &nbs= p; 580 KBytes
[  5]  22.00-23.00  sec  2.= 43 GBytes  20.9 Gbits/sec  510    397 KBytes
[  5]  23.00-24.00  sec  2.38 GBytes  20.= 4 Gbits/sec  532    533 KBytes
[  5]=   24.00-25.00  sec  2.37 GBytes  20.4 Gbits/sec  3= 44    547 KBytes
[  5]  25.00-26.00&= nbsp; sec  2.36 GBytes  20.2 Gbits/sec  354   = ; 389 KBytes
[  5]  26.00-27.00  sec  2.3= 0 GBytes  19.8 Gbits/sec  165    592 KBytes
[  5]  27.00-28.00  sec  2.30 GBytes  19.8= Gbits/sec  173    584 KBytes
[  5]&= nbsp; 28.00-29.00  sec  2.27 GBytes  19.5 Gbits/sec &nb= sp;  0    701 KBytes
[  5]  29.= 00-30.00  sec  2.29 GBytes  19.7 Gbits/sec   = 0    790 KBytes
- - - - - - - - - - - - - - -= - - - - - - - - - -
[ ID] Interval    &n= bsp;      Transfer     Bitrate=          Retr
[ = 5]   0.00-30.00  sec  67.4 GBytes  19.3 Gbits/sec=   6557          &nbs= p;  sender
[  5]   0.00-30.00  sec&n= bsp; 67.4 GBytes  19.3 Gbits/sec      &n= bsp;           receiver

iperf Done.


Thank You!



---- On Thu, 16 Jun 2022 17:00:25 -0400 Alexander V. Chernikov <mel= ifaro@ipfw.ru> wrote ----



> On 16 Jun 2022, at 21:48, Mike Jakubik = <m= ike.jakubik@swiftsmsgateway.com> wrote:
>
> After mult= iple tests and tweaks i believe the issue is not with the HW or Numa relate= d (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP st= ack. It's like it cant figure itself out properly for the speed that the HW= can do, i keep getting widely varying results when testing. Below is an ex= ample of two tests, with about a 15 second break in between the two.
Do= es pinning iperf to a specific CPU core (or range) address the variety part= ? e.g. cpuset -l 1 perf =E2=80=A6
The output you shared above shows CPU= #83 as the core iperf is running on. Just wondering whether the scheduler m= igrates iperf too often, thrashing the caches.

=


------=_Part_6963461_1051991644.1655426335083--