Syncookies break with Windows 8

Fri Feb 1 22:25:26 UTC 2013

On Feb 1, 2013, at 4:05 PM, Ed Maste <emaste at freebsd.org> wrote:

> On 1 February 2013 16:21, Kevin Day <kevin at your.org> wrote:
>> We've got a large cluster of HTTP servers, each server handling >10,000req/sec. Occasionally, and during periods of heavy load, we'd get complaints from some users that downloads were working but going EXTREMELY slowly. After a whole lot of debugging, we narrowed it down to being only Windows 8 clients experiencing this problem. It turns out that FreeBSD's implementation of syncookies is likely violating RFC1323.
> 
> Kevin,
> 
> Thanks for the thorough analysis and report, although I didn't see
> mention of which FreeBSD version you're running.  It looks like andre@
> added storage of the window scale option in the timestamp many years
> ago in r162277[1], so I'm curious if you have an old version or
> there's an issue with this implementation.

This is in 9.1. I saw that change, but based it's not kicking in here (i think) because Windows wasn't setting a TSopt option in the initial SYN. RFC1323 says:

         A TCP may send the Timestamps option (TSopt) in an initial
         <SYN> segment (i.e., segment containing a SYN bit and no ACK
         bit), and may send a TSopt in other segments only if it re-
         ceived a TSopt in the initial <SYN> segment for the connection.

The client is not setting TSopt on the SYN, so we can't set it on the SYN/ACK. 

I can't tell if RFC1323 is saying you MUST support timestamps if you have window scaling or not:

     It is vitally important to use the RTTM mechanism with big
      windows; otherwise, the door is opened to some dangerous
      instabilities due to aliasing.  Furthermore, the option is
      probably useful for all TCP's, since it simplifies the sender.

It appears that the code here won't do window scaling stuffing into timestamps if we didn't get a timestamp on the SYN though. On connection:

                /*
                 * A timestamp received in a SYN makes
                 * it ok to send timestamp requests and replies.
                 */
                if (to->to_flags & TOF_TS) {
                        sc->sc_tsreflect = to->to_tsval;
                        sc->sc_ts = tcp_ts_getticks();
                        sc->sc_flags |= SCF_TIMESTAMP;
                }

Then later on the syncookies ACK:

        /* Additional parameters are stored in the timestamp if present. */
        if (sc->sc_flags & SCF_TIMESTAMP) {
                data =  ((sc->sc_flags & SCF_SIGNATURE) ? 1 : 0); /* TCP-MD5, 1 bit */
                data |= ((sc->sc_flags & SCF_SACK) ? 1 : 0) << 1; /* SACK, 1 bit */
                data |= sc->sc_requested_s_scale << 2;  /* SWIN scale, 4 bits */
                data |= sc->sc_requested_r_scale << 6;  /* RWIN scale, 4 bits */
                data |= md5_buffer[2] << 10;            /* more digest bits */
                data ^= md5_buffer[3];
                sc->sc_ts = data;
                sc->sc_tsoff = data - tcp_ts_getticks();        /* after XOR */
        }

If SCF_TIMESTAMP doesn't get set, the server doesn't do this.

Here's an example tcpdump showing what's happening, with syncookies_only=1 set:

14:22:16.696366 IP (tos 0x0, ttl 118, id 18606, offset 0, flags [DF], proto TCP (6), length 52)
    client.49637 > server.80: Flags [S], cksum 0xe8c0 (correct), seq 4056314475, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
14:22:16.696386 IP (tos 0x0, ttl 64, id 60489, offset 0, flags [DF], proto TCP (6), length 52)
    server.80 > client.49637: Flags [S.], cksum 0x400e (incorrect -> 0x5f10), seq 3099521508, ack 4056314476, win 65535, options [mss 1460,nop,wscale 5,sackOK,eol], length 0
14:22:16.708523 IP (tos 0x0, ttl 118, id 18607, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x9ddf (correct), seq 1, ack 1, win 256, length 0
14:22:16.708537 IP (tos 0x0, ttl 119, id 18608, offset 0, flags [DF], proto TCP (6), length 452)
    client.49637 > server.80: Flags [P.], cksum 0x9aec (correct), seq 1:413, ack 1, win 256, length 412
14:22:16.711885 IP (tos 0x0, ttl 64, id 60794, offset 0, flags [DF], proto TCP (6), length 296)
    server.80 > client.49637: Flags [.], cksum 0x4102 (incorrect -> 0xdd41), seq 1:257, ack 413, win 65535, length 256
14:22:16.782746 IP (tos 0x0, ttl 119, id 18609, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x9b44 (correct), seq 413, ack 257, win 255, length 0
14:22:16.782758 IP (tos 0x0, ttl 64, id 62354, offset 0, flags [DF], proto TCP (6), length 295)
    server.80 > client.49637: Flags [.], cksum 0x4101 (incorrect -> 0x8d90), seq 257:512, ack 413, win 65535, length 255
14:22:16.844941 IP (tos 0x0, ttl 119, id 18610, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x9a46 (correct), seq 413, ack 512, win 254, length 0
14:22:16.844953 IP (tos 0x0, ttl 64, id 63448, offset 0, flags [DF], proto TCP (6), length 294)
    server.80 > client.49637: Flags [.], cksum 0x4100 (incorrect -> 0x9a46), seq 512:766, ack 413, win 65535, length 254
14:22:16.906469 IP (tos 0x0, ttl 119, id 18611, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x9949 (correct), seq 413, ack 766, win 253, length 0
14:22:16.906481 IP (tos 0x0, ttl 64, id 64738, offset 0, flags [DF], proto TCP (6), length 293)
    server.80 > client.49637: Flags [.], cksum 0x40ff (incorrect -> 0x9949), seq 766:1019, ack 413, win 65535, length 253
14:22:16.968393 IP (tos 0x0, ttl 119, id 18612, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x984d (correct), seq 413, ack 1019, win 252, length 0
14:22:16.968414 IP (tos 0x0, ttl 64, id 788, offset 0, flags [DF], proto TCP (6), length 292)
    server.80 > client.49637: Flags [.], cksum 0x40fe (incorrect -> 0x984d), seq 1019:1271, ack 413, win 65535, length 252
14:22:17.036097 IP (tos 0x0, ttl 119, id 18613, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x9752 (correct), seq 413, ack 1271, win 251, length 0
14:22:17.036114 IP (tos 0x0, ttl 64, id 1973, offset 0, flags [DF], proto TCP (6), length 291)
    server.80 > client.49637: Flags [.], cksum 0x40fd (incorrect -> 0x9752), seq 1271:1522, ack 413, win 65535, length 251
14:22:17.095046 IP (tos 0x0, ttl 119, id 18614, offset 0, flags [DF], proto TCP (6), length 40)
    client.49637 > server.80: Flags [.], cksum 0x9652 (correct), seq 413, ack 1522, win 256, length 0
14:22:17.095062 IP (tos 0x0, ttl 64, id 3218, offset 0, flags [DF], proto TCP (6), length 296)
    server.80 > client.49637: Flags [.], cksum 0x4102 (incorrect -> 0x9652), seq 1522:1778, ack 413, win 65535, length 256

You can see where it looks like(from an outside observer) scaling was properly negotiated, but the server is treating the lengths as unshifted amounts.

-- Kevin