TCP connection stalled

[Ian Smith]

In freebsd-questions Digest, Vol 665, Issue 6, Message: 5
On Fri, 3 Mar 2017 17:05:09 +0100 Nagy L?szl? Zsolt <gandalf at shopzeus.com> wrote:

[ sorry if charset mismatch mangled your name ]

 > Hello, I have this setup:
 > 
 > Remote computer <---> Internet <----->  Gateway <---->  NAT-ed local
 > network <---> Internal computer
 > 
 > The gateway is a  11.0-RELEASE-p8 running natd + ipfw.
 > 
 > The internet connection has download speed 500Mbps and upload speed 30Mbps.

Lucky you :)  Is that via PPPoE or what protocol? (may matter re mtu)

 > Internet is connected via a public IPv4 address as shown below (real ip
 > hidden with 'x' chars)
 > 
 > nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 > options=c219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO,LINKSTATE>
 >         ether 40:61:86:ed:e6:41
 >         inet 37.xxx.xxx.xxx netmask 0xfffffe00 broadcast 255.255.255.255

That seems a strange broadcast address with that IP and netmask?  Never 
mind, perhaps that's different on FreeBSD 11?

 >         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 >         media: Ethernet autoselect (1000baseT <full-duplex>)
 >         status: active
 > 
 > The first 4 ipfw rules are the following:
 > 
 > add 00005 divert natd all from any to any via nfe0
 > add 00102 allow all from any to any out

That's any packets out to the outside, AND any out to the inside.

Likely best to be more specific (with recv $iface) unless ALL filtering 
is only wanted on inbound packets (from outside or inside interfaces)

 > add 00104 allow tcp from any to any  established
 > add 00201 allow icmp from any to any icmptypes 0,3,8,11,12,13,14
 > 
 > As you can see the MTU for nfe0 was setup to 1500, and ICMP type 3 is
 > allowed from anybody.
 > 
 > If I try to copy something to a remote server with this command:
 > 
 > scp local_file user at remote_computer:~
 > 
 > Then the following happens: the first few 100K data goes through
 > quickly, then the connection becomes stalled. I have tried different
 > remote computers on different ports, but the result is always the same.
 > I have also tried passive mode FTP instead of SCP with the same result:
 > stalled.

Likely due to ipfw(8) /BUGS, near the end (applies to natd too)

     Due to the architecture of libalias(3), ipfw nat is not compatible with
     the TCP segmentation offloading (TSO).  Thus, to reliably nat your net-
     work traffic, please disable TSO on your NICs using ifconfig(8).

See ifconfig(8).  You need ifconfig nfe0 -tso4 and possibly? -vlanhwtso, 
with the same applying on your LAN-facing interface.

 > If I do the same from the internal computer behind NAT, then all uploads
 > and downloads are fast and responsive. The connection is stalled only
 > when I connect from the gateway (or to the gateway). (E.g. copy from
 > internal computer -> remote computer is fast and reliable.)
 > 
 > I was told that this might be an MTU problem. MTU discovery is turned on:
 > 
 > net.inet.tcp.path_mtu_discovery: 1
 > net.inet.sctp.pmtu_raise_time: 600
 > 
 > I have also tested the MTU with ping -D, the following way:
 > 
 > root at gw:~ # ping -D -s 1500 my_remote_host.com
 > PING my_remote_host.com (185.27.xxx.xxx): 1500 data bytes
 > ^C
 > --- my_remote_host.com ping statistics ---
 > 4 packets transmitted, 0 packets received, 100.0% packet loss

Well with don't-fragment set, anywhere in between with a lower mtu (eg 
perhaps 1492 on your or other end's PPPoE link?) will drop those.  
tcpdump should show any "icmp need-frag but don't-frag set" (ono)

 > root at gw:~ # ping -D -s 1400 my_remote_host.com
 > PING my_remote_host.com (185.27.60.106): 1400 data bytes
 > 1408 bytes from 185.27.xxx.xxx: icmp_seq=0 ttl=58 time=18.752 ms
 > 1408 bytes from 185.27.xxx.xxx: icmp_seq=1 ttl=58 time=19.502 ms
 > 1408 bytes from 185.27.xxx.xxx: icmp_seq=2 ttl=58 time=17.846 ms
 > 1408 bytes from 185.27.xxx.xxx: icmp_seq=3 ttl=58 time=16.891 ms
 > ^C
 > --- my_remote_host.com ping statistics ---
 > 4 packets transmitted, 4 packets received, 0.0% packet loss
 > round-trip min/avg/max/stddev = 16.891/18.248/19.502/0.978 ms
 > root at gw:~ #
 > 
 > So it seems that for this particular host, MTU=1400 sould work. So I
 > have changed the default MTU for this interface:
 > 
 > ifconfig nfe0 mtu 1400

That will only make it worse.  Try 1492 or 1460, say, for the ping size 
instead, until you see what works.

 > But there was no effect at all. SCP and FTP connections are both
 > stalled. (I can send an example tcpdump if required, but I could find
 > nothing special in that.)
 > 
 > What else can cause this? What should I look for?

Firstly, TSO will definitely cause problems like that with nat or natd.  

Run tcpdump on the gateway, to observe TCP packets originating from both 
the gateway and packets passing through it from/to the LAN, which should 
show you the MTU being used after MTU path discovery, probably less than 
1500 - unless it's ethernet all the way?

cheers, Ian