Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix?

On Feb 1, 2023, at 11:26 AM, Paul Mather <paul@gromit.dlib.vt.edu> wrote:

> On Jan 31, 2023, at 3:38 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
> 
>> W dniu 31.01.2023 o 19:31, Paul Mather pisze:
>>>> While playing with different mod_cc(4) might bring some improvement, to get a real boost I'd suggest enabling tcp_rack(4) if feasible.
>>> 
>>> I am interested in trying this out, but believe it is more feasible in my case for the -STABLE and -CURRENT systems I am using, not so much for the -RELEASE systems that are kept up to date via binary freebsd-update updates.  My reading of the tcp_rack(4) man page is that you have to build a custom kernel as, unlike the cc_* congestion control algorithms, the loadable tcp_rack module is not built by default.  Is that an accurate reading?
>>> 
>> Yes, this gift from Netflix is probably better suited for -STABLE and -CURRENT as easier to set up there. There is an excellent, up-to-date article about it by Klara Systems writers[1]. From my experience tcp_rack(4) is well suited for congested, lossy or redundant network paths where loses, duplicated packets or races between packets occur. Not a panacea, but very performant TCP stack based on the _fair_  algorithm. In some instances, it might help you to saturate the bandwidth of the link. TCP algo can be loaded/unloaded/changed on the fly. In FreeBSD 14-CURRENT you can change it on an active socket with tcpsso(8) utility, in FreeBSD 12 and 13 you have to restart the app bound to the socket.
>> Please feel free to play with TCP stacks and congestion algos with the help of benchmarks/iperf3 to find out what prevents the link from being saturated and give us some feedback here.
>> 
>> [1] https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/
> 
> 
> Thank you, Marek, for the link to the Klara article about building and using RACK.  I'm building it now on a FreeBSD-CURRENT system and will test it out.


It looks like we may have a winner, folks.  I built and enabled the extra TCP stacks and for the first time was able to max out my connection to the remote FreeBSD system.  I get consistently higher throughput over the 15-hop WAN path to the remote FreeBSD system when using the RACK TCP stack than when using the default "freebsd" stack.

Although the speeds are consistently higher when using the setting "net.inet.tcp.functions_default=rack", they are still variable.  However, rather than the 3--4 MB/s I saw that kicked off this thread, I now average over 10 MB/s.

I actually get the best results with "net.inet.tcp.functions_default=bbr" (having loaded tcp_bbr).  That behaves very much like the Linux hosts in that speeds climb very quickly until it saturates the WAN connection.  I get the same high speeds from the remote FreeBSD system using tcp_bbr as I do to the Linux hosts.  I will stick with tcp_bbr for now as the default on my remote FreeBSD servers.  It appears to put them on a par with Linux for this WAN link.

Cheers,

Paul.