ssh over WAN: TCP window too small

[Chris Stankevitz]

John-Mark,

Thank you for your reply.

On 8/25/15 6:03 PM, John-Mark Gurney wrote:
> Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700:
>> # cat /dev/urandom | ssh root at host 'cat > /dev/null'
>
> Don't use this for testing... use /dev/zero or some other device
> that can produce data faster than this...

Okay.  As I'm sure you can imagine, I used urandom to avoid compression 
artifacts.  My urandom produces data at ~300 Mbps... but I will use 
/dev/zero from now on.

> So, our SSH does have the HPN patches:
> https://www.psc.edu/index.php/hpn-ssh
>
> and the README says:
> BUFFER SIZES:
> - if HPN is disabled the receive buffer size will be set to the OpenSSH default
>    of 64K.

Yes... I spent some time reading that document and fretting over whether 
or not HPN was really incorporated in my setup.  I "confirmed" that it 
was available and enabled by setting "HPNDisabled no" and restarting 
sshd (on both sides) without complaint.  I'm half-tempted to build from 
ports to be certain.

> Looks like there are undocumented options like TCPRcvBuf that you can
> use to adjust the recv buffer window...

According to the HPN README the default (which I am using) is the 
"system wide TCP receive buffer size".  I don't know what value that is 
or where it comes from (net.inet.tcp.???).  I will experiment with 
TCPRcvBuf.

> We have code that will auto grow
> buffer sizes properly so that slow connections won't use up too much
> buffer space...

That is what I expected, although I believe openssh tries thwart/limit 
this by requesting particular buffer sizes (I'm really unqualified to 
talk about this).  And it is my understanding that HPN undoes these 
limitations although I'm not sure if it opens the door to FreeBSD having 
full control or uses its own voodoo.

> In a quick test of mine, I'm seeing a buffer size of ~520k from my
> MacOSX box, and ~776k from my 9.2-R box...  Server in both cases is
> a June -CURRENT

Thank you for those numbers.  Since my system is basically stock, I 
wonder if my bad behavior is an artifact of something on my network. 
Did you invoke ssh more or less as "cat /dev/zero | ssh root at host 'cat > 
/dev/null'"?   Are you quoting S-BCNT numbers?

> netstat -xAanfinet is helpful on this...

That is brilliant!  I was using pcap and wireshark to deduce some of 
that info.

I include my sender and receiver netstat's below for the ssh-ing 
/dev/zero.  It differs from iperf (which works well), most notably in 
S-BCNT (~1MB for iperf, ~64kB for ssh).  I think in my case the question is:

- who is keeping S-BCNT so low (openssh, HPN, or FreeBSD)?

- Is the limitation introduced by the sending or receiving system?

- what is the mechanism by which S-BCNT grows when using ssh over 
long/fat pipes?

Thank you again,

Chris

SSH Sender
Recv-Q  0
Send-Q  50132
R-MBUF  0
S-MBUF  16
R-CLUS  0
S-CLUS  14
R-HIWA  66052
S-HIWA  82852
R-LOWA  1
S-LOWA  2048
R-BCNT  0
S-BCNT  57344
R-BMAX  528416
S-BMAX  662816
rexmt   0.29
persist 0
keep    7199.98
2msl    0
delack  0
rcvtime 0.01

SSH Receiver
Recv-Q  0
Send-Q  36
R-MBUF  0
S-MBUF  1
R-CLUS  0
S-CLUS  0
R-HIWA  66052
S-HIWA  33700
R-LOWA  1
S-LOWA  2048
R-BCNT  0
S-BCNT  256
R-BMAX  528416
S-BMAX  269600
rexmt   0.24
persist 0
keep    7199.96
2msl    0
delack  0.06
rcvtime 0.03