From nobody Sat Feb 04 18:11:57 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P8LGD54Fpz3nh4x for ; Sat, 4 Feb 2023 18:12:20 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.126.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P8LGC58pzz3C4b for ; Sat, 4 Feb 2023 18:12:19 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of paul@gromit.dlib.vt.edu has no SPF policy when checking 128.173.126.123) smtp.mailfrom=paul@gromit.dlib.vt.edu; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=vt.edu (policy=none) Received: from smtpclient.apple (unknown [IPv6:2001:470:e15b:23::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id 8CB12485E3; Sat, 4 Feb 2023 13:12:08 -0500 (EST) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? From: Paul Mather In-Reply-To: Date: Sat, 4 Feb 2023 13:11:57 -0500 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <4ed8b724-041f-f561-ae60-ab966aefbb68@plan-b.pwste.edu.pl> <282AF730-E5E0-4A50-9F47-E7301B36E5C8@gromit.dlib.vt.edu> <2ed582b9-b544-74bb-2047-99d04924b46b@plan-b.pwste.edu.pl> <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> <83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu> To: David <2yt@gmx.com> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-1.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.997]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[vt.edu : No valid SPF, No valid DKIM,none]; ASN(0.00)[asn:1312, ipnet:128.173.0.0/16, country:US]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FREEMAIL_TO(0.00)[gmx.com]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; FREEFALL_USER(0.00)[paul]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4P8LGC58pzz3C4b X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Feb 1, 2023, at 5:54 PM, David <2yt@gmx.com> wrote: > On 2/1/23 14:07, Paul Mather wrote: >> On Feb 1, 2023, at 3:14 PM, Marek Zarychta = wrote: >>> W dniu 1.02.2023 o 20:33, Paul Mather pisze: >>>> It looks like we may have a winner, folks. I built and enabled the = extra TCP stacks and for the first time was able to max out my = connection to the remote FreeBSD system. I get consistently higher = throughput over the 15-hop WAN path to the remote FreeBSD system when = using the RACK TCP stack than when using the default "freebsd" stack. >>>>=20 >>>> Although the speeds are consistently higher when using the setting = "net.inet.tcp.functions_default=3Drack", they are still variable. = However, rather than the 3--4 MB/s I saw that kicked off this thread, I = now average over 10 MB/s. >>>>=20 >>>> I actually get the best results with = "net.inet.tcp.functions_default=3Dbbr" (having loaded tcp_bbr). That = behaves very much like the Linux hosts in that speeds climb very quickly = until it saturates the WAN connection. I get the same high speeds from = the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I = will stick with tcp_bbr for now as the default on my remote FreeBSD = servers. It appears to put them on a par with Linux for this WAN link. >>>=20 >>> Thanks for the feedback Paul. Please bear in mind that BBR 1 which = is implemented in FreeBSD is not a fair[1] congestion control algorithm. = Maybe in the future, we will have BBR v2 in the stack, but for now, I = don't recommend using BBR, unless you want to act slightly as a hm.... = network leecher. Maybe Linux hosts behave this way, maybe they have = implemented BBR v2, I am not familiar with Linux TCP stack enhancements. = On the other hand, tcp_rack(4) is performant, well-tested in the FreeBSD = stack, considered fair and more acceptable for a fileserver, though not = ideal, ie. probably more computationally expensive and still missing = some features like TCP-MD5. >>>=20 >>> [1] https://www.mdpi.com/1424-8220/21/12/4128 >>>=20 >> That is a fair and astute observation, Marek. I am also not familiar = with Linux TCP stack implementations but it had occurred to me that = maybe Linux was not being an entirely good netizen whereas FreeBSD was = behaving with impeccable net manners when it came to congestion control = and being fair to others, and that is why Linux was getting faster = speeds for me. Then again, perhaps not. :-) >> In the case of the remote FreeBSD hosts I use at $JOB, they have low = numbers of users and so are more akin to endpoints than servers, so I'm = not worried about "leeching" from them. Also, my ISP download bandwidth = is 1/5th of each FreeBSD system, so hopefully there is still plenty to = go around after I max out my bulk downloads. (Plus, I believe $JOB = prefers my downloads to take half [or less] the time.) :-) >> Hopefully we will get BBR v2 (or something even fairer) at some = point. IIRC, the FreeBSD Foundation has been highlighting some of this = network stack work. It would be a pity for it not to be enabled by = default so more people could use it on -RELEASE without building a = custom kernel. I'm just glad right now I'm not stuck with 3--4 MB/s = downloads any more. >> Cheers, >> Paul. >=20 > Word of caution: >=20 > It would appear not all FreeBSD applications like BBR or RACK. I run a = Magento (e-commerce) VM and was getting weird pauses (hang for a bit = then resume) on the website. Running Magneto requires several other TCP = services and something wasn't happy. Not going to debug the problem, = just wanted to give a heads up. Thanks for the heads-up. Since posting the above I have also noticed = that BBR and RACK aren't unalloyed successes for me. In my experience, = I would also notice pauses and lockups/disconnections to FreeBSD systems = with BBR enabled. I only noticed pauses with RACK on some systems in my = testing, so that was much more usable. The problems I experienced were worst on 13.1-RELEASE. A 13.1-RELEASE = system I built and enabled BBR on was almost unusable to me. RACK was = better but also had issues. Much better in my tests is BBR and RACK on = 13-STABLE and -CURRENT. I had no issues with RACK (other with more = speed variability vs BBR), whereas BBR did lock up one of my FreeBSD = clients in testing. (That client uses a RealTek re NIC, so maybe BBR = tickles more bugs in that.) RACK caused no lockups and yielded good = enough speeds for it to be my go-to combo now over BBR. I figure the better results with enabling BBR and RACK on -STABLE and = -CURRENT vs -RELEASE servers reflects potential improvements/bug fixes = in the implementation since -RELEASE landed. (Disclaimer: the -RELEASE = test system uses em NICs whereas the -STABLE and -CURRENT systems I used = in testing use igb NICs, so maybe that is a factor?) Cheers, Paul.