From nobody Thu Jun 29 12:52:44 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsJJl1TRlz4k0rt for ; Thu, 29 Jun 2023 12:52:55 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsJJk6g2lz4DtB for ; Thu, 29 Jun 2023 12:52:54 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:35f7:d883:641f:3e25]) (Authenticated sender: micmac) by mail-n.franken.de (Postfix) with ESMTPSA id 1F0B38008A033; Thu, 29 Jun 2023 14:52:45 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux From: tuexen@freebsd.org In-Reply-To: Date: Thu, 29 Jun 2023 14:52:44 +0200 Cc: "freebsd-transport@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: <19F174D3-F0DB-4CD0-B43C-7A3BD8F47AFF@freebsd.org> References: To: Murali Krishnamurthy X-Mailer: Apple Mail (2.3731.600.7) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: 4QsJJk6g2lz4DtB X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On 29. Jun 2023, at 11:51, Murali Krishnamurthy = wrote: >=20 > Hello FreeBSD Transport experts, > We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in = long RTT setup and happened to compare the performance with Linux VM = with same hypervisor. > We see a substantially better performance with Linux getting close to = the BDP limit, whereas BSD 13 not filling up the pipe enough. > We are trying to figure out what could lead to such a huge difference = and feel we could be missing something here. > Could you please help us to know if there is a way to make it perform = better? > Setup details: > We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu = 23.04/Linux kernel 6.2) were launched on each hypervisor. > Then we ran iperf between, > =E2=80=A2 BSD 13 <-> BSD 13 > =E2=80=A2 Ubuntu <-> Ubuntu > Even though the network environment were same in both cases, we see = Ubuntu performing much better. > Below are connection parameters: > Socket buffer: 16MB > TCP CC Algo: Cubic. We used this as this is suitable for Long Fat = Networks. > Ping RTT: 100 ms between the two end points. > We kept all other parameters to default on both Linux and BSD. > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 = bits/ 1024 =3D 1.25 Gbps > Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP = limit. > FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So = it seems to be doing half as good as Linux. > For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same = and able to meet BDP of 300 Mbps consistently. > Larger socket buffer seems to have an issue. > Please let us know if there are ways to fine tune the system = parameters to make BSD perform better. > Or any other suggestions/queries welcome. > Regards > Murali Hi Murali, I'll bring this up on the biweekly transport call. Best regards Michael