From nobody Wed Feb 01 22:54:09 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6cft2wKmz3cGPC for ; Wed, 1 Feb 2023 22:54:14 +0000 (UTC) (envelope-from 2yt@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6cfs3J2Cz3Bq2 for ; Wed, 1 Feb 2023 22:54:13 +0000 (UTC) (envelope-from 2yt@gmx.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of 2yt@gmx.com designates 212.227.17.22 as permitted sender) smtp.mailfrom=2yt@gmx.com; dmarc=pass (policy=none) header.from=gmx.com Received: from [192.168.160.51] ([47.5.85.140]) by mail.gmx.net (mrgmx105 [212.227.17.174]) with ESMTPSA (Nemesis) id 1N6KUT-1oYnZy0GiH-016i7q for ; Wed, 01 Feb 2023 23:54:11 +0100 Message-ID: Date: Wed, 1 Feb 2023 15:54:09 -0700 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? Content-Language: en-US References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <4ed8b724-041f-f561-ae60-ab966aefbb68@plan-b.pwste.edu.pl> <282AF730-E5E0-4A50-9F47-E7301B36E5C8@gromit.dlib.vt.edu> <2ed582b9-b544-74bb-2047-99d04924b46b@plan-b.pwste.edu.pl> <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> <83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu> To: freebsd-stable@freebsd.org From: David <2yt@gmx.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K1:skPCEXNlj+3YVtCKwjzpFcprpRiUnfllUTxn1XJrOMoD5IzOUUf AGYtvlMoH9Ghvk+JrID8hQ8XLhEIGmeIlGyNDGkDZAO5iitlxbp0Nup3djfJ7a1OFkWSiId 2PiThiY7atgOafAkcua7s2xfQQDcWnFk36FjpEK9HqTvIdLad86awPGSvMIwYD1TXj1xjHM Vd/gqfJbNvtDNQvyRjjWw== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:nUF/5NLIVVo=;P33NylyXltIHXYc/4URA1uTCgyl jCVJg2eCJBbnnGesKhmQiE9455f2GXad+R7PVv3yFbJpY+rnU92ufHbbTqh7BzIScagUiNjWS mgf7H+kxW+m8DXqCD41LTq18tKMZfQGF/MefJbOtNbcrNJ2Adx5mrfzxKSQ1rSNULtp53Y8Lj 3hZkJWV7/vccM0bMsSbES488GXS7FmYBQeG0GvM+T8kyQSkFkpBxWdKyGmezA+v6y+6e2YJjX w9w/i1B0Ne861JRPDbT/ei2qDqwqZ993Mh4XBZBXLWxwXJLwoM0WVJU3fZROqN1NXJdVpcnLS 1zdOThIJL/mn+FJBfYcA3isH6bpvT6QypM55kdRTbky7L1yUKi9vdwlfAnnDFjqzCCVFB8FgG 0PJnxiAmzyfuEtnfOJN1bw7wHKmKO+Svir1Ug5DtZ+1J40UU4huPEFp2MfSHqTE6oFgWmh9d7 gkGWbcx7W6be04smE9rFK5zwi/yCeOlq/6DC33iNJy7rp0ubN/ts/jmDn/ngBf+o9jycDkT3/ tVoCQv8v/c4LInSQEwWuKlBWZr4SXYkq2qDVzcmszK0iPismPmegpzsiwcB2+q/fmUOovT/Qf X8Fu3qkgz1ANlTQ0REjQujrimYaYVKzKtzQpu8noDjkeEK6DjI+C7lZJCMSDy5jKYZ7i9GAYi P936fM4RmB3Wb8KmfUQt9k51YrgQos40uqLkg8ocQ7jP0NRQtx2GvShvykt/2UxxZXY/saS2u VZJi6BAB/5okBWT9ulpeyj+JZ1ucJNMssBE9+5xcJhyzD26jh6hJkdYlMBuiHPVmDoENXSOV6 ZWDVVE0PPUAPaFk8xNXylN9sGEksEw2NSTzDWOYzJedxRPyCrOcALjumFZT4QWK3a+CDyC8qm gi0k6NmRL6XyWFj6v+bMkN3c1aVrQuhmncU0FgFWvCc6PThQ9TgJaEIV5qCKMV9Uf6/I6NvIY fQfx1w== X-Spamd-Result: default: False [-1.17 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.995]; NEURAL_HAM_LONG(-0.99)[-0.986]; NEURAL_SPAM_SHORT(0.71)[0.707]; DMARC_POLICY_ALLOW(-0.50)[gmx.com,none]; R_SPF_ALLOW(-0.20)[+ip4:212.227.17.0/27]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.17.22:from]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.17.22:from]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmx.com]; FREEMAIL_ENVFROM(0.00)[gmx.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4P6cfs3J2Cz3Bq2 X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 2/1/23 14:07, Paul Mather wrote: > On Feb 1, 2023, at 3:14 PM, Marek Zarychta > wrote: > >> W dniu 1.02.2023 o 20:33, Paul Mather pisze: >>> It looks like we may have a winner, folks. I built and enabled the extra TCP stacks and for the first time was able to max out my connection to the remote FreeBSD system. I get consistently higher throughput over the 15-hop WAN path to the remote FreeBSD system when using the RACK TCP stack than when using the default "freebsd" stack. >>> >>> Although the speeds are consistently higher when using the setting "net.inet.tcp.functions_default=rack", they are still variable. However, rather than the 3--4 MB/s I saw that kicked off this thread, I now average over 10 MB/s. >>> >>> I actually get the best results with "net.inet.tcp.functions_default=bbr" (having loaded tcp_bbr). That behaves very much like the Linux hosts in that speeds climb very quickly until it saturates the WAN connection. I get the same high speeds from the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I will stick with tcp_bbr for now as the default on my remote FreeBSD servers. It appears to put them on a par with Linux for this WAN link. >> >> Thanks for the feedback Paul. Please bear in mind that BBR 1 which is >> implemented in FreeBSD is not a fair[1] congestion control algorithm. >> Maybe in the future, we will have BBR v2 in the stack, but for now, I >> don't recommend using BBR, unless you want to act slightly as a hm.... >> network leecher. Maybe Linux hosts behave this way, maybe they have >> implemented BBR v2, I am not familiar with Linux TCP stack >> enhancements. On the other hand, tcp_rack(4) is performant, >> well-tested in the FreeBSD stack, considered fair and more acceptable >> for a fileserver, though not ideal, ie. probably more computationally >> expensive and still missing some features like TCP-MD5. >> >> [1] https://www.mdpi.com/1424-8220/21/12/4128 >> > > That is a fair and astute observation, Marek.  I am also not familiar > with Linux TCP stack implementations but it had occurred to me that > maybe Linux was not being an entirely good netizen whereas FreeBSD was > behaving with impeccable net manners when it came to congestion control > and being fair to others, and that is why Linux was getting faster > speeds for me.  Then again, perhaps not. :-) > > In the case of the remote FreeBSD hosts I use at $JOB, they have low > numbers of users and so are more akin to endpoints than servers, so I'm > not worried about "leeching" from them.  Also, my ISP download bandwidth > is 1/5th of each FreeBSD system, so hopefully there is still plenty to > go around after I max out my bulk downloads.  (Plus, I believe $JOB > prefers my downloads to take half [or less] the time.) :-) > > Hopefully we will get BBR v2 (or something even fairer) at some point. >  IIRC, the FreeBSD Foundation has been highlighting some of this > network stack work.  It would be a pity for it not to be enabled by > default so more people could use it on -RELEASE without building a > custom kernel.  I'm just glad right now I'm not stuck with 3--4 MB/s > downloads any more. > > Cheers, > > Paul. > Word of caution: It would appear not all FreeBSD applications like BBR or RACK. I run a Magento (e-commerce) VM and was getting weird pauses (hang for a bit then resume) on the website. Running Magneto requires several other TCP services and something wasn't happy. Not going to debug the problem, just wanted to give a heads up. -- David