Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix?

On Jan 30, 2023, at 9:13 PM, Eugene Grosbein <eugen@grosbein.net> wrote:

> 31.01.2023 4:17, Paul Mather wrote:
> 
>> TL;DR: When working from home, I can max out my residential 200 Mbit network connection when downloading from remote Linux hosts at $JOB but only manage about 20% of my max residential connection speed when downloading from remote FreeBSD hosts at $JOB.  When at $JOB, both FreeBSD and Linux hosts have no problem saturating their GbE connections transferring between each other.  Why is this and how can I debug and fix it?
>> 
>> I have a 200 Mbit residential cable connection (Xfinity, 200 Mbit down/~10 Mbit up).  I've noticed recently that I can easily get 10--20 MB/s download speeds when transferring data from Linux hosts at work but when I try to download that same data from the FreeBSD hosts I use the speed usually tops out at 3--4 MB/s.  These are Linux and FreeBSD hosts that are on the same subnet at work.  Transfers from the FreeBSD hosts at work (within-subnet and within-site) are fine and match those of the Linux hosts---often 112 MB/s.  So, it just appears to be the traffic over the WAN to my home that is affected.  The WAN path from home to this subnet is typically 15 hops with a typical average ping latency of about 23 ms.
>> 
>> The FreeBSD hosts are a mixture of -CURRENT, 13-STABLE, and 13.1-RELEASE.  I had done some TCP tuning based upon the calomel.org <http://calomel.org/> tuning document (https://calomel.org/freebsd_network_tuning.html), but removed those tuning settings when I noticed the problem but the problem still persists.  The only remaining customisation is that the 13-STABLE has "net.inet.tcp.cc.algorithm=cubic".  (I notice that -CURRENT now has this as default so wanted to try that on 13-STABLE, too.)  The FreeBSD systems are using either igb or em NICs.  The Linux systems are using similar hardware.  None has a problem maintaining local GbE transfer speeds---it's only the slower/longer WAN connections that have problems for the FreeBSD hosts.
>> 
>> It seems that Linux hosts cope with the WAN path to my home better than the FreeBSD systems.  Has anyone else noticed this?  Does anyone have any idea as to what is obviously going wrong here and how I might debug/fix the FreeBSD hosts to yield faster speeds?  My workaround at the moment is to favour using the remote Linux hosts for bulk data transfers.  (I don't like this workaround.)
>> 
>> Any help/insight is gratefully appreciated.
> 
> I bet speedy traffic does not cross any NAT boxes but perhaps you employ NAT at your own place.
> Both pfnat and ipfw nat are not compatible with TSO, also sometimes RX/TX checksum offload for NIC produce broken checksums,
> and all that creates excessive retransmissions and timeouts greatly reducing traffic speed.


My IPv4 clients at home are behind a NAT router (OPNsense 23.1 with HTCP CC enabled and FQ_CoDel traffic shaper configured).  The remote systems at $JOB are not behind NAT.  On my OPNsense router I have disabled hardware offloading as (IIRC) these are generally not recommended for routers.  As such, I have "Disable hardware checksum offload"; "Disable hardware TCP segmentation offload"; and "Disable hardware large receive offload" all checked in Interfaces -> Settings in OPNsense.  The home router running OPNsense uses NICs that identify as "82583V Gigabit Network Connection" (emX@pci0:X:0:0:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x150c subvendor=0x8086 subdevice=0x0000).

The FreeBSD systems at $JOB default to having TSO and RXCSUM/TXCSUM enabled.  I disabled these but it didn't make any apparent difference in improving speeds.  I looked at a Linux system on similar hardware and "ethtool -k" indicates it also has TSO and RX/TX checksum offloading enabled.


> You may want to inspect traffic with Wireshark, as it shows retransmissions and generally anomalies with colors,
> or just go ahead and disable TSO and rxcsum/txcsum for external interface.


I'm going to pursue this next.  I made Wireshark captures at the client end but need to collect packet traces at the server side.  Also, with the help of your suggestion, I managed to find a colour configuration to highlight retransmissions.  Hopefully that might help differentiate the FreeBSD vs. Linux situation.  I'm not great at grokking Wireshark traces but I guess I'm about to get better in the future. :-)

Thanks for the suggestions.

Cheers,

Paul.