ssh over WAN: TCP window too small

[Chris Stankevitz]

John-Mark,

I'm going to rearrange the conversation a bit in the quotes below:


On 8/26/15 3:32 PM, John-Mark Gurney wrote:
> So, I looked at what getsockopt SO_RCVBUF returns, and it returns:
>                  case SO_RCVBUF:
>                          optval = so->so_rcv.sb_hiwat;
>
> Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes
> for the buffer...

I believe that explains everything we are seeing.  HPN developers 
thought SO_RCVBUF would return S-BMAX but it instead returns S-HIWA.

> If it's decided to base it's waiting for ack on SO_RCVBUF, then this
> is probably where the issue is...
>
> Try setting HPNBufferSize to something resonably large, like 1MB, or
> above your bandwidth-delay product to see if this will make a difference..
> Per:
>    Conditions: HPNBufferSize SET, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set
>    Result: HPN Buffer Size = grows to HPNBufferSize
>      The buffer will grow up to the maximum size specified here.


Setting HPNBufferSize to 1 MB does not change S-BCNT which remains stuck 
at ~60KB.  Although I might not necessarily expect a change.  From 
README.hpn:

HPNBufferSize: This is the default buffer size the HPN functionality 
uses when interacting with non-HPN SSH installations.

> It would be interesting to know how long from the read of stdin (and is
> it really reading stdin in 4k blocks?  If so, that should be fixed)
> to the write out the socket... Basicly, how long does it take to encrypt
> the data...
>
>> Note in the above the blocking call to select at time 6.592065 that
>> takes ~0.1 second.  This is the reason my connection is slow.  The
>> content appears to be encrypted... I presume it's an application-level
>> ssh ack.

I posted the in-context kdump at:
  https://www.stankevitz.com/ssh-ktrace.txt

stdin is read in 4k blocks... but each "cycle" reads 3 of these blocks. 
  I define as "cycle" as the period beginning and ending at a 0.1 second 
select(2) sleep.

Looks like encryption of a 4KB block takes 10 microseconds, which means: 
for every 0.1 seconds in select(2), .000030 seconds are encrypting.

>> net.inet.tcp.sendspace: 32768
>> net.inet.tcp.recvspace: 65536
>
> Try increasing these and reporting back...  the buf_auto should override
> those, but this may be limiting it...

Okay.  Now we're getting somewhere.

Increase sendspace on "sending client": no change

Increase recvspace on "sending client": no change

Increase sendspace on "receiving server": no change

Increase recvspace on "receiving server": sending client's S-BCNT 
increases proportionally!

I'm going to try Kurts parameters now...

Chris