TCP Receive buffer scaling without Timestamps

Lawrence Stewart lstewart at freebsd.org
Mon Feb 20 08:37:05 UTC 2017


On 19/02/2017 12:37, Steven Hartland wrote:
> On 18/02/2017 00:33, Lawrence Stewart wrote:
>> On 18/02/2017 09:37, Steven Hartland wrote:
>> [snip]
>>> So the questions:
>>>
>>> 1. Has anyone looked at this issue before or is working on it?
>>> 2. Do people think we could add support for RTT estimates and enable
>>>    receive scaling without major tcp stack changes?
>> We already have a non-TS-enabled RTT estimator in place c.f. the
>> relevant code in tcp_input.c:
>>
>> if ((to.to_flags & TOF_TS) != 0 &&
>>     to.to_tsecr) {
>> 	uint32_t t;
>>
>> 	t = tcp_ts_getticks() - to.to_tsecr;
>> 	if (!tp->t_rttlow || tp->t_rttlow > t)
>> 		tp->t_rttlow = t;
>> 	tcp_xmit_timer(tp,
>> 	    TCP_TS_TO_TICKS(t) + 1);
>> } else if (tp->t_rtttime &&
>>     SEQ_GT(th->th_ack, tp->t_rtseq)) {
>> 	if (!tp->t_rttlow ||
>> 	    tp->t_rttlow > ticks - tp->t_rtttime)
>> 		tp->t_rttlow = ticks - tp->t_rtttime;
>> 	tcp_xmit_timer(tp,
>> 			ticks - tp->t_rtttime);
>> }
>>
>>
>> The autoscaling implementation just needs to have its insistence on
>> using timestamps stuffed into a cannon and shot in the direction of the
>> sun. This is the relevant block of rcvbuf code from tcp_input.c:
>>
>>
>> if (V_tcp_do_autorcvbuf &&
>>     (to.to_flags & TOF_TS) &&
>>     to.to_tsecr &&
>>     (so->so_rcv.sb_flags & SB_AUTOSIZE)) {
>> 	if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) &&
>> 	    to.to_tsecr - tp->rfbuf_ts < hz) {
>> 		if (tp->rfbuf_cnt >
>> 		    (so->so_rcv.sb_hiwat / 8 * 7) &&
>> 		    so->so_rcv.sb_hiwat <
>> 		    V_tcp_autorcvbuf_max) {
>> 			newsize =
>> 			    min(so->so_rcv.sb_hiwat +
>> 			    V_tcp_autorcvbuf_inc,
>> 			    V_tcp_autorcvbuf_max);
>> 		}
>> 		/* Start over with next RTT. */
>> 		tp->rfbuf_ts = 0;
>> 		tp->rfbuf_cnt = 0;
>> 	} else
>> 		tp->rfbuf_cnt += tlen;  /* add up */
>> }
>>
>> It's pretty trivial to fix by anyone so inclined.
> Thanks for the pointers Lawrence, I'm not very familiar with network
> stack but I've had a stab at this, after reading through the current
> code in the blocks you pointed out, along side some experimentation ;-)

Nice, thanks for running with it.

> The results seem pretty encouraging, with S3 downloads with a ~17ms
> latency on a 1Gbps jumping from ~3MB/s to ~80MB/s.

Amazing what a bit of tuning can do :)

> I'd appreciate feedback on the review which can be found here:
> https://reviews.freebsd.org/D9668

ACK, will follow up there.

Cheers,
Lawrence


More information about the freebsd-transport mailing list