ssh via NAT slow on _some_ connections only

[Alejandro Imass]

Hi folks,

This is *very* weird but it's consistent.

Most of my servers run with jailed services and I access the jails
directly with NAT to a private network where the jails run.

Jails network are just aliases of lo0 liske so:


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
        inet 192.168.101.1 netmask 0xffffff00
        inet 192.168.101.80 netmask 0xffffff00
        inet 192.168.101.101 netmask 0xffffff00
[etc.]
        inet 192.168.101.123 netmask 0xffffff00

Then in natd.conf I have nats defined like so:

redirect_port tcp 192.168.101.123:22 12322

At first _all_  my NATed ssh connections were slow until I added -tso
to the main nic ifconfig. So this -tco switch is something that I've
had to add to all my nics for NAT to work properly:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>

Nevertheless, _some_ specific networks are still very slow with NATed ssh.

So for example, I access the server from my home network and I can't
even notice any difference between non-NAT and NATed connections. But
we have one specific remote location where the NATed connections are
really slow.
It's not their network because if they first login to the base server
(no NAT) and _then_ ssh to the private IP, then the performance is
perfect. The issue is only when on the natted port.

In other words: if they ssh -p 12322 like the example above it's
painfully slow, but if they first ssh to the base server and then ssh
to the private IP, the performance is great. This is the exact same
performance issue we were getting before the -tco param, so maybe
there are other flags that affect NAT performance? maybe on that
location's router? Wouldn't this affect the normal ssh connections,
why only the NAT ports have problems?

I really want to avoid to replicate the users in the base system, so
there must be something else that can be done to fix this.
Again, -tco helped a lot but for these particular locations there is
still some problem with the NATed connections we haven't been able to
figure out.

Anyone have any ideas on what could be going on here?

Thanks,

--
Alejandro Imass