From nobody Wed Feb 01 19:33:32 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6XCd6zp0z3bqKw for ; Wed, 1 Feb 2023 19:33:49 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.ipv6.vt.edu [IPv6:2001:468:c80:a103:2:5000:5555:5555]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6XCd0K9xz3rxP for ; Wed, 1 Feb 2023 19:33:49 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of paul@gromit.dlib.vt.edu has no SPF policy when checking 2001:468:c80:a103:2:5000:5555:5555) smtp.mailfrom=paul@gromit.dlib.vt.edu; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=vt.edu (policy=none) Received: from smtpclient.apple (unknown [IPv6:2001:470:e15b:23::23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id D33B031F7B; Wed, 1 Feb 2023 14:33:44 -0500 (EST) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? From: Paul Mather In-Reply-To: <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> Date: Wed, 1 Feb 2023 14:33:32 -0500 Cc: stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu> References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <4ed8b724-041f-f561-ae60-ab966aefbb68@plan-b.pwste.edu.pl> <282AF730-E5E0-4A50-9F47-E7301B36E5C8@gromit.dlib.vt.edu> <2ed582b9-b544-74bb-2047-99d04924b46b@plan-b.pwste.edu.pl> <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> To: Marek Zarychta X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-1.46 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.96)[-0.964]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[vt.edu : No valid SPF, No valid DKIM,none]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[stable@freebsd.org]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:1312, ipnet:2001:468:c80::/48, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FREEFALL_USER(0.00)[paul]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4P6XCd0K9xz3rxP X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Feb 1, 2023, at 11:26 AM, Paul Mather = wrote: > On Jan 31, 2023, at 3:38 PM, Marek Zarychta = wrote: >=20 >> W dniu 31.01.2023 o 19:31, Paul Mather pisze: >>>> While playing with different mod_cc(4) might bring some = improvement, to get a real boost I'd suggest enabling tcp_rack(4) if = feasible. >>>=20 >>> I am interested in trying this out, but believe it is more feasible = in my case for the -STABLE and -CURRENT systems I am using, not so much = for the -RELEASE systems that are kept up to date via binary = freebsd-update updates. My reading of the tcp_rack(4) man page is that = you have to build a custom kernel as, unlike the cc_* congestion control = algorithms, the loadable tcp_rack module is not built by default. Is = that an accurate reading? >>>=20 >> Yes, this gift from Netflix is probably better suited for -STABLE and = -CURRENT as easier to set up there. There is an excellent, up-to-date = article about it by Klara Systems writers[1]. =46rom my experience = tcp_rack(4) is well suited for congested, lossy or redundant network = paths where loses, duplicated packets or races between packets occur. = Not a panacea, but very performant TCP stack based on the _fair_ = algorithm. In some instances, it might help you to saturate the = bandwidth of the link. TCP algo can be loaded/unloaded/changed on the = fly. In FreeBSD 14-CURRENT you can change it on an active socket with = tcpsso(8) utility, in FreeBSD 12 and 13 you have to restart the app = bound to the socket. >> Please feel free to play with TCP stacks and congestion algos with = the help of benchmarks/iperf3 to find out what prevents the link from = being saturated and give us some feedback here. >>=20 >> [1] = https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/ >=20 >=20 > Thank you, Marek, for the link to the Klara article about building and = using RACK. I'm building it now on a FreeBSD-CURRENT system and will = test it out. It looks like we may have a winner, folks. I built and enabled the = extra TCP stacks and for the first time was able to max out my = connection to the remote FreeBSD system. I get consistently higher = throughput over the 15-hop WAN path to the remote FreeBSD system when = using the RACK TCP stack than when using the default "freebsd" stack. Although the speeds are consistently higher when using the setting = "net.inet.tcp.functions_default=3Drack", they are still variable. = However, rather than the 3--4 MB/s I saw that kicked off this thread, I = now average over 10 MB/s. I actually get the best results with = "net.inet.tcp.functions_default=3Dbbr" (having loaded tcp_bbr). That = behaves very much like the Linux hosts in that speeds climb very quickly = until it saturates the WAN connection. I get the same high speeds from = the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I = will stick with tcp_bbr for now as the default on my remote FreeBSD = servers. It appears to put them on a par with Linux for this WAN link. Cheers, Paul.