From nobody Tue Nov 21 02:02:08 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SZ7141fK9z51WC9 for ; Tue, 21 Nov 2023 02:02:16 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SZ7141CHwz3FYw; Tue, 21 Nov 2023 02:02:16 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700532136; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dW+Zt/yKe5xOYHVdwzMYvl2BYHjfILCYKY+lXhEIocs=; b=wNT5icH/stkLMHK/SLqo97eKOYyBgdHjbYyFmXmyORxUOOaZPZpuE43gz57GWUXnhUhexo ohh/Hiuj3SKCmultGfz4c/oBWk1b2Gz1S+ySg8tEj5xONBZwSSANrSmFwISoX81y5stfAm Dq9TqtyUgwTgQ4ljZOmsowV4ZSmxGfLrkLYP5CYGYmKZ0fDHD1Jo6YjmsLZEYCEl2QfFS6 MUatqtHrdklWbPvmfv2x7fC37jhPVnTCscRGhbk78mrOl/yvORcT5kYSfL8at3XLhsGEOd pngs0RWmD2wxcYslNO+vUdS3X7Q0z1tiYO9fjAsI3P1yiAnFlr2qQ/ZUyMGGqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700532136; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dW+Zt/yKe5xOYHVdwzMYvl2BYHjfILCYKY+lXhEIocs=; b=jvbR/AJZcILtbda+Wxv+qmFxso3VaTn4yXMcZsh1k4wPr+F5oMxm5Dw4DO9UsCi7wjjnzS 9i5XM7LteDDyjl0RzN7BdS78KE2gTofvifOncEHkUXnd4sEhV/oaXZTQ5tfKi/ZpnU/DX5 +6soSKC8y1wPK7Kkng1Q7cvhedJFJOWXew3cd2GdFHA/oOkA0oBXk9KapZ5KG1wL446ycb MNVufTA+wpJpdapUJvjkYIhSD88bmhIKM4cW08r5kU5ChoedQwUiW+1JIuRlR4dRyn9bUb CEhorBZFk7j+eUHBExDvz3XPOtqGyv5qfdA8GGjffJb04SGggb0qvxTZric3aQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700532136; a=rsa-sha256; cv=none; b=VX0NEctMblVDELvIywVED/08JL6IKuHoCwu5pjlqFkNm40oceTwY/O37mRRHVnOzkXJYz4 ZRdFxPrZtCnMNfok7K8PsjPv6HMRJ4QzAvXhi+Ie8F7jrNGEChffR/AQxsP7NYocsV5KOo tsLEnf5c0n0yOLdstYkJ5DSKbj8wLlJtNo1s2patxX5254kuJZ/XqfT35xfe9zwnrR+0rT AJBNWufSl2atJid1hJAaBvwniyYlYUOUoITQTNukyI1tXGOaEzM4dIG9boAvxw0G0BIYrr Q8NTYRY5rSxK1tpucdFQ02/bbSRg3ffmnsUNbG3fXPo+FCFeDExp5wsEGS82UQ== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SZ7131b8wz153L; Tue, 21 Nov 2023 02:02:14 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: Low performance of BBR compared to cubic From: Zhenlei Huang In-Reply-To: Date: Tue, 21 Nov 2023 10:02:08 +0800 Cc: freebsd-transport@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: References: To: Richard Scheffenegger X-Mailer: Apple Mail (2.3696.120.41.1.4) > Zhenlei, > > Maybe you want to characterize the TX vs. RX codepaths independently of > each other, and run tests with different Stacks on either end (could be > tuned via setsockopt, or tcpsso; or maybe by togging the default in > between starting the iperf3 server side, before running the client, if > this is done via loopback). I'll take care that. > > Currently, the expectation is that the TX codepath of RACK is more > optimized vs. the RX codepath - thus a RACK sender to a base stack > receiver should show the highest performance. So the client will benefit by enabling RACK on servers. That's interesting. > > Best reagrds, > Richard > > > Am 20.11.2023 um 13:29 schrieb Scheffenegger, Richard: > > > > > > BBR has not been looked after for quite some while (and there are no > > plans to invest there). > > > > Among other things, the blatant disregard of flow fairness which BBRv1 > > shows, makes it a poor protocol for general use. > > > > Similar issues also show up with BBRv2 (current version), but still it > > is not considered a reasonable and stable enough protocol - thus the > > IETF is working on BBRv3. > > > > Both of which would require effective re-implementation of the BBR stack. > > > > > > > > RACK is expected to perform better across congested paths, and in the > > presence of various pathological network issues. In particular, the > > receive path is certainly not as performant as the Base Stack currently. > > > > In short: If you want to use BBR, please don't use the current code with > > is at best a variation of BBRv1 - and it's generally known that this > > version of BBR is not "good". > > > > > > (I presume your testing was acrosss a green field / zero loss network, > > with ample bandwidth - maybe the loopback interface even). Yes tested with lo0 and one thread. > > > > Richard > > > > > > > > -----Original Message----- > > > > > > Hi, > > > > While test TCP RACK functions, I tested BBR BTW. > > > > This is a quick test with iperf3 on bare metal ( old MBP i5 2 cores 4 > > threads ). > > The kernel current/15 with debug options disabled. Following is the > > performance result: > > > > freebsd: 37.2 Gbits/sec 1.32 MBytes > > RACK: 27.9 Gbits/sec 1.34 MBytes > > BBR: 2.34 Gbits/sec 223 KBytes > > > > For freebsd and RACK functions the CC is cubic. > > > > The last column is Cwnd. BBR's Cwnd looks quite small. > > > > There's also a report on Telegram Taiwan FreeBSD chat group, but without > > performance details. > > > > I believe there is something wrong with BBR. This is not a reasonable > > good performance compared with other tcp congestion control algorithms. > > > > Or am I missing something ? > > > > Best regards, > > Zhenlei > > > > >