TCP connection stalled

[Nagy László Zsolt]

Hello, I have this setup:

Remote computer <---> Internet <----->  Gateway <---->  NAT-ed local
network <---> Internal computer

The gateway is a  11.0-RELEASE-p8 running natd + ipfw.

The internet connection has download speed 500Mbps and upload speed 30Mbps.

Internet is connected via a public IPv4 address as shown below (real ip
hidden with 'x' chars)

nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       
options=c219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO,LINKSTATE>
        ether 40:61:86:ed:e6:41
        inet 37.xxx.xxx.xxx netmask 0xfffffe00 broadcast 255.255.255.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

The first 4 ipfw rules are the following:

add 00005 divert natd all from any to any via nfe0
add 00102 allow all from any to any out
add 00104 allow tcp from any to any  established
add 00201 allow icmp from any to any icmptypes 0,3,8,11,12,13,14

As you can see the MTU for nfe0 was setup to 1500, and ICMP type 3 is
allowed from anybody.

If I try to copy something to a remote server with this command:

scp local_file user at remote_computer:~

Then the following happens: the first few 100K data goes through
quickly, then the connection becomes stalled. I have tried different
remote computers on different ports, but the result is always the same.
I have also tried passive mode FTP instead of SCP with the same result:
stalled.

If I do the same from the internal computer behind NAT, then all uploads
and downloads are fast and responsive. The connection is stalled only
when I connect from the gateway (or to the gateway). (E.g. copy from
internal computer -> remote computer is fast and reliable.)

I was told that this might be an MTU problem. MTU discovery is turned on:

net.inet.tcp.path_mtu_discovery: 1
net.inet.sctp.pmtu_raise_time: 600

I have also tested the MTU with ping -D, the following way:

root at gw:~ # ping -D -s 1500 my_remote_host.com
PING my_remote_host.com (185.27.xxx.xxx): 1500 data bytes
^C
--- my_remote_host.com ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root at gw:~ # ping -D -s 1400 my_remote_host.com
PING my_remote_host.com (185.27.60.106): 1400 data bytes
1408 bytes from 185.27.xxx.xxx: icmp_seq=0 ttl=58 time=18.752 ms
1408 bytes from 185.27.xxx.xxx: icmp_seq=1 ttl=58 time=19.502 ms
1408 bytes from 185.27.xxx.xxx: icmp_seq=2 ttl=58 time=17.846 ms
1408 bytes from 185.27.xxx.xxx: icmp_seq=3 ttl=58 time=16.891 ms
^C
--- my_remote_host.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.891/18.248/19.502/0.978 ms
root at gw:~ #

So it seems that for this particular host, MTU=1400 sould work. So I
have changed the default MTU for this interface:

ifconfig nfe0 mtu 1400

But there was no effect at all. SCP and FTP connections are both
stalled. (I can send an example tcpdump if required, but I could find
nothing special in that.)

What else can cause this? What should I look for?

Thanks,

   Laszlo