Syncookies break with Windows 8

[Kevin Day]

On Feb 1, 2013, at 4:39 PM, Andre Oppermann <oppermann at networx.ch> wrote:
> 
> This is not true.  FreeBSD uses bits in the timestamp to encode all
> recognized TCP options including window scaling.
> 

Sorry, you are correct here. Reading through a half dozen TCP implementations in the last day trying to figure out what was happening and it all turns into a blur. :)

>> This doesn't affect Linux because it uses timestamp options to stuff the client's wscale, so it
>> gets re-learned on the ACK. OpenBSD and OS X don't have syncookies. NetBSD seems to have the same
>> problem if it's new syncookie implementation gets turned on.
> 
> This can't be because of the lack of timestamps.  Linux must be
> encoding the scale in the ISS taking away bits from the cookie.
> 
> I haven't looked into how Linux actually does it recently.
> 

In Linux, it appears if timestamps are not enabled, either because they're turned off by sysctl or the server didn't see one from the client, it disables scaling.

http://fxr.watson.org/fxr/source/net/ipv4/syncookies.c?v=linux-2.6#L235

wscale_ok defaults to 0. It's set to '1' only if the server saw_tstamp, sysctl_tcp_timestamps isn't turned off, etc. 

>> Any thoughts? Was there a reason why we're forcing the use of wscale on syncookie connections?
> 
> We can change the behavior of syncookies in a couple of ways to
> deal with this problem:
> 
> 1/ send syncookies only when the syncache overflows and set wscale
>    to 0 in the SYN-ACK when timestamps are not active.
> 

That sounds similar to Linux, if I'm reading the code correctly.

> At the moment we send syncookies on every SYN-ACK and bump the oldest
> entry from the syncache when it is full.  That results in potentially
> every segment degrading to syncookie only.  The default values are
> insufficient for such high loads.
> 
> In general at 10,000 connections per second you should significantly
> increase the size of your syncache to 3 * conn/sec at least.
> 
> In the loader you can set these tunables:
> 
> net.inet.tcp.syncache.hashsize    = 2048
> net.inet.tcp.syncache.bucketlimit = 32
> net.inet.tcp.syncache.cachelimit  = 65536
> 
> These settings are a bit more complicated than they should be.
> 

We're doing similar. During a DDoS we tried changing to syncookies_only and somehow that got left that way, which is how we ran into this problem more reliably. Turning that off, and using higher syncache sizes helped hide this greatly, but it's probably still happening here and elsewhere occasionally.

-- Kevin