RTT and TCP Window size doubts, bandwidth issues

[Ian Smith]

On Wed, 8 Apr 2015 00:10:51 +0200, Marek Salwerowicz wrote:
 > Hi list,
 > 
 > I am trying to find correct setup of sysctl's for following machines (VMs
 > under Vmware Workstation 8) to test large TCP window size:
 > 
 > 
 > There are 2 boxes, each of them has following setup:
 > - % uname -a
 > FreeBSD freeA 10.1-RELEASE-p6 FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21
 > UTC 2015 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
 > amd64
 > 
 > - 4GB of RAM
 > - 1 NIC
 > 
 > without any modifications, iperf reports bandwidth speed ~1Gbit/s between
 > hosts:
 > 
 > server-side:
 > # iperf -s
 > ------------------------------------------------------------
 > Server listening on TCP port 5001
 > TCP window size: 64.0 KByte (default)
 > ------------------------------------------------------------
 > 
 > client-side:
 > 
 > # iperf -c 192.168.108.140
 > ------------------------------------------------------------
 > Client connecting to 192.168.108.140, TCP port 5001
 > TCP window size: 32.5 KByte (default)
 > ------------------------------------------------------------
 > [  3] local 192.168.108.141 port 35282 connected with 192.168.108.140 port
 > 5001
 > [ ID] Interval       Transfer     Bandwidth
 > [  3]  0.0-10.0 sec  1.46 GBytes  1.25 Gbits/sec
 > 
 > 
 > 
 > I want to simulate (using dummynet) link between hosts with bandwidth
 > 400mbit/s and latency ~20ms.
 > In order to do that, I create the ipfw pipe on one box:
 > IPFW="ipfw -q"
 > 
 > 
 > $IPFW pipe 1 config bw 400Mbit/s delay 10ms
 > 
 > $IPFW add 1500 pipe 1 ip from any to any
 > 
 > 
 > after running ipfw, bandwidth with default kernel sysctl becomes lower:
 > 
 > client-side:
 > ------------------------------------------------------------
 > Client connecting to 192.168.108.140, TCP port 5001
 > TCP window size: 32.5 KByte (default)
 > ------------------------------------------------------------
 > [  3] local 192.168.108.141 port 35340 connected with 192.168.108.140 port
 > 5001
 > [ ID] Interval       Transfer     Bandwidth
 > [  3]  0.0-10.1 sec  12.5 MBytes  10.4 Mbits/sec
 > 
 > 
 > 
 > I'd like to achieve bandwidth ~400Mbit/s.
 > 
 > I've modified following sysctl's (both on client- and server-side):
 > 
 > kern.ipc.maxsockbuf=33554432  # (default 2097152)
 > 
 > net.inet.tcp.sendbuf_max=33554432  # (default 2097152)
 > net.inet.tcp.recvbuf_max=33554432  # (default 2097152)
 > 
 > net.inet.tcp.cc.algorithm=htcp  # (default newreno) #enabled in
 > /boot/loader.conf also
 > net.inet.tcp.cc.htcp.adaptive_backoff=1 # (default 0 ; disabled)
 > 
 > net.inet.tcp.cc.htcp.rtt_scaling=1 # (default 0 ; disabled)
 > 
 > net.inet.tcp.mssdflt=1460  # (default 536)
 > 
 > net.inet.tcp.minmss=1300   # (default 216)
 > 
 > net.inet.tcp.rfc1323=1  # (default 1)
 > net.inet.tcp.rfc3390=1  # (default 1)
 > 
 > 
 > net.inet.tcp.sendspace=8388608  # (default 32768)
 > net.inet.tcp.recvspace=8388608  # (default 65536)
 > 
 > net.inet.tcp.sendbuf_inc=32768  # (default 8192 )
 > net.inet.tcp.recvbuf_inc=65536  # (default 16384)
 > 
 > 
 > 
 > But the  results are not really good as I expected:
 > 
 > server-side:
 > # iperf -s
 > ------------------------------------------------------------
 > Server listening on TCP port 5001
 > TCP window size: 8.00 MByte (default)
 > ------------------------------------------------------------
 > 
 > client-side:
 > # iperf -c 192.168.108.140
 > ------------------------------------------------------------
 > Client connecting to 192.168.108.140, TCP port 5001
 > TCP window size: 8.00 MByte (default)
 > ------------------------------------------------------------
 > [  3] local 192.168.108.141 port 21894 connected with 192.168.108.140 port
 > 5001
 > [ ID] Interval       Transfer     Bandwidth
 > [  3]  0.0-10.1 sec  24.2 MBytes  20.2 Mbits/sec
 > 
 > 
 > I was trying to follow the articles:
 > - http://www.psc.edu/index.php/networking/641-tcp-tune
 > - https://fasterdata.es.net/host-tuning/freebsd/
 > 
 > But can't really figure out what / how should be tuned in order to achieve
 > good results.
 > 
 > If anyone could find some time and give me some hints, I'd be pleased!

Many of your tunings are over my head, but I can see that you've nicely 
emulated a half-duplex link; you want one pipe for each direction for a 
fast bidirectional link.  See ipfw(8) /TRAFFIC SHAPING

Also be sure of your setting of sysctl net.inet.ip.fw.one_pass, here and 
also re your later post re ipfw / dummynet, which also wants 2 pipes.

cheers, Ian