From nobody Wed Feb 01 21:07:15 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6ZHl1ckFz3c2QM for ; Wed, 1 Feb 2023 21:07:31 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.ipv6.vt.edu [IPv6:2001:468:c80:a103:2:5000:5555:5555]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6ZHl1B7sz43VV for ; Wed, 1 Feb 2023 21:07:31 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2001:470:e15b:23::23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id F3F30205EE; Wed, 1 Feb 2023 16:07:25 -0500 (EST) From: Paul Mather Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2" List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? Date: Wed, 1 Feb 2023 16:07:15 -0500 In-Reply-To: Cc: stable@freebsd.org To: Marek Zarychta References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <4ed8b724-041f-f561-ae60-ab966aefbb68@plan-b.pwste.edu.pl> <282AF730-E5E0-4A50-9F47-E7301B36E5C8@gromit.dlib.vt.edu> <2ed582b9-b544-74bb-2047-99d04924b46b@plan-b.pwste.edu.pl> <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> <83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4P6ZHl1B7sz43VV X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1312, ipnet:2001:468:c80::/48, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Feb 1, 2023, at 3:14 PM, Marek Zarychta = wrote: > W dniu 1.02.2023 o 20:33, Paul Mather pisze: >> It looks like we may have a winner, folks. I built and enabled the = extra TCP stacks and for the first time was able to max out my = connection to the remote FreeBSD system. I get consistently higher = throughput over the 15-hop WAN path to the remote FreeBSD system when = using the RACK TCP stack than when using the default "freebsd" stack. >>=20 >> Although the speeds are consistently higher when using the setting = "net.inet.tcp.functions_default=3Drack", they are still variable. = However, rather than the 3--4 MB/s I saw that kicked off this thread, I = now average over 10 MB/s. >>=20 >> I actually get the best results with = "net.inet.tcp.functions_default=3Dbbr" (having loaded tcp_bbr). That = behaves very much like the Linux hosts in that speeds climb very quickly = until it saturates the WAN connection. I get the same high speeds from = the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I = will stick with tcp_bbr for now as the default on my remote FreeBSD = servers. It appears to put them on a par with Linux for this WAN link. > Thanks for the feedback Paul. Please bear in mind that BBR 1 which is = implemented in FreeBSD is not a fair[1] congestion control algorithm. = Maybe in the future, we will have BBR v2 in the stack, but for now, I = don't recommend using BBR, unless you want to act slightly as a hm.... = network leecher. Maybe Linux hosts behave this way, maybe they have = implemented BBR v2, I am not familiar with Linux TCP stack enhancements. > On the other hand, tcp_rack(4) is performant, well-tested in the = FreeBSD stack, considered fair and more acceptable for a fileserver, = though not ideal, ie. probably more computationally expensive and still = missing some features like TCP-MD5. >=20 >=20 > [1] https://www.mdpi.com/1424-8220/21/12/4128 >=20 That is a fair and astute observation, Marek. I am also not familiar = with Linux TCP stack implementations but it had occurred to me that = maybe Linux was not being an entirely good netizen whereas FreeBSD was = behaving with impeccable net manners when it came to congestion control = and being fair to others, and that is why Linux was getting faster = speeds for me. Then again, perhaps not. :-) In the case of the remote FreeBSD hosts I use at $JOB, they have low = numbers of users and so are more akin to endpoints than servers, so I'm = not worried about "leeching" from them. Also, my ISP download bandwidth = is 1/5th of each FreeBSD system, so hopefully there is still plenty to = go around after I max out my bulk downloads. (Plus, I believe $JOB = prefers my downloads to take half [or less] the time.) :-) Hopefully we will get BBR v2 (or something even fairer) at some point. = IIRC, the FreeBSD Foundation has been highlighting some of this network = stack work. It would be a pity for it not to be enabled by default so = more people could use it on -RELEASE without building a custom kernel. = I'm just glad right now I'm not stuck with 3--4 MB/s downloads any more. Cheers, Paul. --Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii On Feb 1, = 2023, at 3:14 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> = wrote:

W dniu 1.02.2023 = o 20:33, Paul Mather pisze:
It looks like we may have a =
winner, folks.  I built and enabled the extra TCP stacks and for the =
first time was able to max out my connection to the remote FreeBSD =
system.  I get consistently higher throughput over the 15-hop WAN path =
to the remote FreeBSD system when using the RACK TCP stack than when =
using the default "freebsd" stack.

Although the speeds are consistently higher when using the setting =
"net.inet.tcp.functions_default=3Drack", they are still variable.  =
However, rather than the 3--4 MB/s I saw that kicked off this thread, I =
now average over 10 MB/s.

I actually get the best results with =
"net.inet.tcp.functions_default=3Dbbr" (having loaded tcp_bbr).  That =
behaves very much like the Linux hosts in that speeds climb very quickly =
until it saturates the WAN connection.  I get the same high speeds from =
the remote FreeBSD system using tcp_bbr as I do to the Linux hosts.  I =
will stick with tcp_bbr for now as the default on my remote FreeBSD =
servers.  It appears to put them on a par with Linux for this WAN link.

Thanks for the feedback Paul. Please bear in mind = that BBR 1 which is implemented in FreeBSD is not a fair[1] congestion = control algorithm. Maybe in the future, we will have BBR v2 in the = stack, but for now, I don't recommend using BBR, unless you want to act = slightly as a hm.... network leecher. Maybe Linux hosts behave this way, = maybe they have implemented BBR v2, I am not familiar with Linux TCP = stack enhancements. On the other hand, tcp_rack(4) is performant, well-tested in the FreeBSD = stack, considered fair and more acceptable for a fileserver, though not = ideal, ie. probably more computationally expensive and still missing = some features like TCP-MD5.

[1] https://www.mdpi.com/14= 24-8220/21/12/4128


That = is a fair and astute observation, Marek.  I am also not familiar = with Linux TCP stack implementations but it had occurred to me that = maybe Linux was not being an entirely good netizen whereas FreeBSD was = behaving with impeccable net manners when it came to congestion control = and being fair to others, and that is why Linux was getting faster = speeds for me.  Then again, perhaps not. = :-)

In the case of the remote FreeBSD hosts I = use at $JOB, they have low numbers of users and so are more akin to = endpoints than servers, so I'm not worried about "leeching" from them. =  Also, my ISP download bandwidth is 1/5th of each FreeBSD system, = so hopefully there is still plenty to go around after I max out my bulk = downloads.  (Plus, I believe $JOB prefers my downloads to take half = [or less] the time.) :-)

Hopefully we will get = BBR v2 (or something even fairer) at some point.  IIRC, the FreeBSD = Foundation has been highlighting some of this network stack work. =  It would be a pity for it not to be enabled by default so more = people could use it on -RELEASE without building a custom kernel. =  I'm just glad right now I'm not stuck with 3--4 MB/s downloads any = more.

Cheers,

Paul.
= --Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2--