From nobody Wed Feb 01 00:27:42 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P62nR5LCpz3cylR for ; Wed, 1 Feb 2023 00:27:55 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-vs1-xe30.google.com (mail-vs1-xe30.google.com [IPv6:2607:f8b0:4864:20::e30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P62nR3H0Vz45NL for ; Wed, 1 Feb 2023 00:27:55 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-vs1-xe30.google.com with SMTP id y8so17980236vsq.0 for ; Tue, 31 Jan 2023 16:27:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ASwz0KCzJl1Xvmaj+Jv8TyvvwiZ5/08hZ8GhY9F5WQU=; b=YI3Mcad4pll/O67Q1ugeKIY/bIi2AV7OsmOqTJYh12wzqjBoWUKFSgMDMmFf0uBTL6 52oY2Zszy5bmfm8zSsi4fM6puyqZVFwC67i+TdBG0JKSk+A6gKv2izuz+GUu3Z0NTmfh jyegV+NLhD2RM642WvaDCo+teJiobgPzN3uBPRtPqzE6Wk9TxVs7/eY2to0CO6x82oZy /jXiCgU2+E+heQ+kzci7b31GYLqUh6WtvPmFd7p5qFYmYAX5TdOpKJbh1hMCL7GaoMps b/OsXGXLTQ141F+DVdgvRxcFhXi1vZC9QK3EF8hNyQgqi2eP+n/iHim15qulOzypYVFK TZjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ASwz0KCzJl1Xvmaj+Jv8TyvvwiZ5/08hZ8GhY9F5WQU=; b=HrqLPFrR9Z26kMKvhjVbIbr7bI+sf6V7nOH8ODhVQ92cFzdFtJbXEt8om4TGvKtyW6 MC1cOjYt0f00E89Tv3sJHb+S1KAW2HA1O1OBG2vKOe6Jxijm+H7uiFcei4u5Y94MmsX1 eGDlMl/trkAwR/QcESoP4600lmMgAD+KokuJmJ2dX3phlv6/T798JyOc5Dt/CjJqzj0U o/bbgqGGBZJWqdTyokDXO84ivy1vuKVw4vOZclaJSXOIzCcHuridAxJMfQ/IkaD3jbWE r0Qmt1dcsg9XHiw679am3MOg19Ek8IGZYSE9rJvORnVh9MvPjnw9owkX/vhfpWmrYU+N ghYQ== X-Gm-Message-State: AO0yUKU9ztDEOLMd8Rkh2Shbha7uU24yZlrXNaWUFaqEJYtaYCoBx4Wd tB22NVqVOVY12Hm/OyVIToBwbLDmtzY+KCCSv35lNg== X-Google-Smtp-Source: AK7set/zekbE8fCAXcTUXjjAORIbLdVkwNZiYFMO3idA1STRJKT1ZKUZb+TrIXHOExO6aYznerE2nntNckAtb1QkXeQ= X-Received: by 2002:a05:6102:5596:b0:3eb:8473:a258 with SMTP id dc22-20020a056102559600b003eb8473a258mr158135vsb.14.1675211273507; Tue, 31 Jan 2023 16:27:53 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> In-Reply-To: From: Mark Saad Date: Tue, 31 Jan 2023 19:27:42 -0500 Message-ID: Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? To: Paul Mather Cc: David <2yt@gmx.com>, freebsd-stable@freebsd.org Content-Type: multipart/alternative; boundary="000000000000e1ae7205f39884bb" X-Rspamd-Queue-Id: 4P62nR3H0Vz45NL X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000e1ae7205f39884bb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jan 31, 2023 at 5:03 PM Paul Mather wrote= : > On Jan 31, 2023, at 10:39 AM, David <2yt@gmx.com> wrote: > > > On 1/30/23 16:30, Matt Garber wrote: > >> > Any help/insight is gratefully appreciated. > >> > > >> > Cheers, > >> > > >> > Paul. > >> > > >> sysctl net.inet.tcp.cc.algorithm=3Dhtcp > >> I would set "htcp" on the server and home computer to improve > >> through in > >> your type of situation. > >> There may be other FreeBSD sysctls that have bad defaults in this > scenario and could be better tuned, but I doubt changing the CC algorithm > at this point is the problem =E2=80=94 at least not so much a problem tha= t=E2=80=99s > causing throughput to be reduced so drastically. Happy to be wrong if tha= t > does help things quickly and easily, though. > >> (Since OP mentioned that FreeBSD CC was set to CUBIC, that would match > what the Linux boxes are using by default, too, unless they=E2=80=99ve be= en changed > to something newer like BBR=E2=80=A6 so seems like CUBIC *should* be perf= orming > fine on this WAN link, and the difference is something else.) > >> =E2=80=94Matt > > > > I love FreeBSD and very much appreciate the efforts of those people muc= h > smarter. But... I don't think the defaults get enough testing in real wor= ld > conditions. > > > > I came across Paul's issue several years ago and spent a few a hours > testing and found the defaults performed very well on a LAN but could > perform terribly on a many hop WAN. HTCP performs marginally worse on a L= AN > or close WAN connection, but much much better on a many hop WAN connectio= n. > > > [[...]] > > > In my opinion HTCP is a better default for the current state of the > internet. > > > It looks like they already changed the default from NewReno to CUBIC in > FreeBSD-CURRENT. > > I agree with your observation about the defaults vs. real world > conditions. As you observed, I also get great performance in a high-spee= d > LAN, but not so much in a many-hop WAN to an asymmetric ISP connection. > > I actually started down this rabbit hole when I noticed I couldn't manage > more than about 3--4 MB in a single stream and thought my ISP was > throttling me. But then I noticed I would actually get fast/maximum > speeds, e.g., when doing "brew upgrade -v" and Homebrew would be > downloading packages, and so then wondered whether they were throttling > non-HTTP traffic. That led me to discover that even HTTP downloads were > slow to the FreeBSD servers I use remotely at $JOB and, furthermore, that > all traffic to Linux systems I use at $JOB didn't seem to be throttled or > incapable of getting maximum single stream bandwidth matching my ISP's > quoted speeds. :-\ > > I accept that this may just be a peculiarity of my local and remote setup= , > and so appreciate the help and suggestions people have offered in trying = to > debug the issue. > > Cheers, > > Paul. > Paul I was reading this and I wanted to just add two bits, one congestion control only really affects tcp sockets, so if you are using a freebsd router, setting the cc algo here does not do all that much. Second thing, the "ergonomics" of the tcp tunings on FreeBSD vs Redhat are very different. . IMHO try aligning the two boxes tcp tunings and also make sure the network cards used in each server are also aligned. IE do you have LRO enabled on one server and not the other. Is one server using an Intel 10G nic vs a Chelsio 10G nic ? --=20 mark saad | nonesuch@longcount.org --000000000000e1ae7205f39884bb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Jan 31, 2023 at 5:03 PM Paul = Mather <paul@gromit.dlib.vt.e= du> wrote:
2yt@gmx.com> wrote:

> On 1/30/23 16:30, Matt Garber wrote:
>>=C2=A0 =C2=A0 =C2=A0> Any help/insight is gratefully appreciated= .
>>=C2=A0 =C2=A0 =C2=A0>
>>=C2=A0 =C2=A0 =C2=A0> Cheers,
>>=C2=A0 =C2=A0 =C2=A0>
>>=C2=A0 =C2=A0 =C2=A0> Paul.
>>=C2=A0 =C2=A0 =C2=A0>
>>=C2=A0 =C2=A0 sysctl net.inet.tcp.cc.algorithm=3Dhtcp
>>=C2=A0 =C2=A0 I would set "htcp" on the server and home c= omputer to improve
>>=C2=A0 =C2=A0 through in
>>=C2=A0 =C2=A0 your type of situation.
>> There may be other FreeBSD sysctls that have bad defaults in this = scenario and could be better tuned, but I doubt changing the CC algorithm a= t this point is the problem =E2=80=94 at least not so much a problem that= =E2=80=99s causing throughput to be reduced so drastically. Happy to be wro= ng if that does help things quickly and easily, though.
>> (Since OP mentioned that FreeBSD CC was set to CUBIC, that would m= atch what the Linux boxes are using by default, too, unless they=E2=80=99ve= been changed to something newer like BBR=E2=80=A6 so seems like CUBIC *sho= uld* be performing fine on this WAN link, and the difference is something e= lse.)
>> =E2=80=94Matt
>
> I love FreeBSD and very much appreciate the efforts of those people mu= ch smarter. But... I don't think the defaults get enough testing in rea= l world conditions.
>
> I came across Paul's issue several years ago and spent a few a hou= rs testing and found the defaults performed very well on a LAN but could pe= rform terribly on a many hop WAN. HTCP performs marginally worse on a LAN o= r close WAN connection, but much much better on a many hop WAN connection.<= br> >
[[...]]

> In my opinion HTCP is a better default for the current state of the in= ternet.


It looks like they already changed the default from NewReno to CUBIC in Fre= eBSD-CURRENT.

I agree with your observation about the defaults vs. real world conditions.= =C2=A0 As you observed, I also get great performance in a high-speed LAN, b= ut not so much in a many-hop WAN to an asymmetric ISP connection.

I actually started down this rabbit hole when I noticed I couldn't mana= ge more than about 3--4 MB in a single stream and thought my ISP was thrott= ling me.=C2=A0 But then I noticed I would actually get fast/maximum speeds,= e.g., when doing "brew upgrade -v" and Homebrew would be downloa= ding packages, and so then wondered whether they were throttling non-HTTP t= raffic.=C2=A0 That led me to discover that even HTTP downloads were slow to= the FreeBSD servers I use remotely at $JOB and, furthermore, that all traf= fic to Linux systems I use at $JOB didn't seem to be throttled or incap= able of getting maximum single stream bandwidth matching my ISP's quote= d speeds. :-\

I accept that this may just be a peculiarity of my local and remote setup, = and so appreciate the help and suggestions people have offered in trying to= debug the issue.

Cheers,

Paul.
Paul
=C2=A0 I was reading this and I want= ed to just add two bits, one congestion control only really affects tcp soc= kets, so if you are using a freebsd router, setting the cc algo here does n= ot do all that much. Second thing, the "ergonomics" of the tcp tu= nings on FreeBSD vs Redhat are very different.=C2=A0 .=C2=A0 IMHO try align= ing the two boxes tcp tunings and also make sure the network cards used in = each server are also aligned. IE do you have LRO enabled on one server and = not the other. Is one=C2=A0 server using an Intel 10G nic vs a Chelsio 10G = nic ?

--
= mark saad | non= esuch@longcount.org
--000000000000e1ae7205f39884bb--