ssh over WAN: TCP window too small

[John-Mark Gurney]

Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700:
> # cat /dev/urandom | ssh root at host 'cat > /dev/null'

Don't use this for testing... use /dev/zero or some other device
that can produce data faster than this...

> I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). 
>   tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). 
>   iperf with default options gets the window opened to 500 KBytes 
> (yielding 35 Mbps).
> 
> Both sides of the connection: FreeBSD 10.1 w/default sshd options 
> (except I permit root login).  In particular, HPN is not disabled.
> 
> Can anyone explain my abysmally small TCP window?

Looks like ssh is propbably hard setting the send/recv buffers to
values that are too small...

So, our SSH does have the HPN patches:
https://www.psc.edu/index.php/hpn-ssh

and the README says:
BUFFER SIZES:
- if HPN is disabled the receive buffer size will be set to the OpenSSH default
  of 64K.

You can read more at:
https://svnweb.freebsd.org/base/stable/10/crypto/openssh/README.hpn?annotate=256281

Looks like there are undocumented options like TCPRcvBuf that you can
use to adjust the recv buffer window...  It looks like OpenSSH hard
sets the buffer sizes for some reason...

On FreeBSD, these should never be set unless the option is provided and
 you know what you are doing..  We have code that will auto grow
buffer sizes properly so that slow connections won't use up too much
buffer space...

> Can anyone recommend some tools/tricks to figure out what in FreeBSD 
> and/or base SSH is limiting the send/recv buffer and/or TCP window?

Seems like from looking at the code, things should "just work", so
not sure why you are seeing the smaller window size...

In a quick test of mine, I'm seeing a buffer size of ~520k from my
MacOSX box, and ~776k from my 9.2-R box...  Server in both cases is
a June -CURRENT...  netstat -xAanfinet is helpful on this...

Hope this helps!

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."