TCP Receive buffer scaling without Timestamps

[Steven Hartland]

On 18/02/2017 00:33, Lawrence Stewart wrote:
> On 18/02/2017 09:37, Steven Hartland wrote:
> [snip]
>> So the questions:
>>
>> 1. Has anyone looked at this issue before or is working on it?
>> 2. Do people think we could add support for RTT estimates and enable
>>     receive scaling without major tcp stack changes?
> We already have a non-TS-enabled RTT estimator in place c.f. the
> relevant code in tcp_input.c:
>
> if ((to.to_flags & TOF_TS) != 0 &&
>      to.to_tsecr) {
> 	uint32_t t;
>
> 	t = tcp_ts_getticks() - to.to_tsecr;
> 	if (!tp->t_rttlow || tp->t_rttlow > t)
> 		tp->t_rttlow = t;
> 	tcp_xmit_timer(tp,
> 	    TCP_TS_TO_TICKS(t) + 1);
> } else if (tp->t_rtttime &&
>      SEQ_GT(th->th_ack, tp->t_rtseq)) {
> 	if (!tp->t_rttlow ||
> 	    tp->t_rttlow > ticks - tp->t_rtttime)
> 		tp->t_rttlow = ticks - tp->t_rtttime;
> 	tcp_xmit_timer(tp,
> 			ticks - tp->t_rtttime);
> }
>
>
> The autoscaling implementation just needs to have its insistence on
> using timestamps stuffed into a cannon and shot in the direction of the
> sun. This is the relevant block of rcvbuf code from tcp_input.c:
>
>
> if (V_tcp_do_autorcvbuf &&
>      (to.to_flags & TOF_TS) &&
>      to.to_tsecr &&
>      (so->so_rcv.sb_flags & SB_AUTOSIZE)) {
> 	if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) &&
> 	    to.to_tsecr - tp->rfbuf_ts < hz) {
> 		if (tp->rfbuf_cnt >
> 		    (so->so_rcv.sb_hiwat / 8 * 7) &&
> 		    so->so_rcv.sb_hiwat <
> 		    V_tcp_autorcvbuf_max) {
> 			newsize =
> 			    min(so->so_rcv.sb_hiwat +
> 			    V_tcp_autorcvbuf_inc,
> 			    V_tcp_autorcvbuf_max);
> 		}
> 		/* Start over with next RTT. */
> 		tp->rfbuf_ts = 0;
> 		tp->rfbuf_cnt = 0;
> 	} else
> 		tp->rfbuf_cnt += tlen;  /* add up */
> }
>
> It's pretty trivial to fix by anyone so inclined.
Thanks for the pointers Lawrence, I'm not very familiar with network 
stack but I've had a stab at this, after reading through the current 
code in the blocks you pointed out, along side some experimentation ;-)

The results seem pretty encouraging, with S3 downloads with a ~17ms 
latency on a 1Gbps jumping from ~3MB/s to ~80MB/s.

I'd appreciate feedback on the review which can be found here:
https://reviews.freebsd.org/D9668

Its a bit rough ATM in order to facilitate monitoring the behaviour with 
dtrace.

My initial attempt did try to replicate the tests done to facilitate the 
calls to tcp_xmit_timer, but they couldn't be reused due to the changes 
the tcp_xmit_timer calls themselves trigger, so I based it off the 
already calculated and stored smoothed RTT instead.

     Regards
     Steve