misc/172648: pf(4): 'scrub reassemble tcp' breaks IPv6 packet checksum on SYN ACK

[Mark Martinec]

>Number:         172648
>Category:       misc
>Synopsis:       pf(4): 'scrub reassemble tcp' breaks IPv6 packet checksum on SYN ACK
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Oct 12 19:10:00 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Mark Martinec
>Release:        9.1-PRERELEASE (i.e. 9.1-RC2)
>Organization:
Jozef Stefan Institute, Ljubljana
>Environment:
FreeBSD neli.ijs.si 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #0: Fri Oct 12 18:10:04 CEST 2012     mark at neli.ijs.si:/usr/obj/usr/src/sys/NELI  amd64
>Description:
When pf (packet filter) is enabled and configured with 'scrub reassemble tcp',
IPv6 TCP connections take 9 seconds to establish. Packet capture shows
checksum errors on SYN ACK packets but not on other packets.

A TCP connection establishment (SYN) on IPv6 is (re-)tried four times,
with a 3 second delay between each attempt, while the TCP options are
being simplified each time by the kernel (dropping ECN, CWR, window
scaling, and dropping a timestamp options). Only the fourth attempt
is successful, with no other options but SACK, and this TCP session
then proceeds normally.

Disabling 'scrub reassemble tcp' in the pf avoids the problem.
Similarly, turning off net.inet.tcp.rfc1323 on either end
also avoids the problem, even with 'reassemble tcp' enabled.

The problem does not occur on IPv4 sessions, only on IPv6.

The problem is not associated with interface checksum offloading,
it is repeatable on gif, em, and re interfaces. Also a packet capture
(wireshark) shows packet checksum errors on SYN ACK packets (but
not on the SYN packet) in the first couple of failed attempts, and
no checksum errors on other packets (e.g. after a successfully
established session).

My guess is that the TCP timestamp option triggers a pf bug,
which then miscalculates a packet checksum on SYN ACK.
>How-To-Repeat:
Use the following trivial pf config file:

  scrub all reassemble tcp
  pass all

Then try to establish any TCP session to any IPv6 address.
Any client will do (telnet, ssh, curl, web browser).
Try for example:
  curl -6 -L http://tools.ietf.org/rfc/rfc3021.txt | wc -l

The connection will 'hang' for 9 seconds (until a sufficiently
dumbed-down SYN options are tried), then it proceeds normally.
>Fix:
No known fix.

Two workarounds:
- don't use 'scrub reassemble tcp' in PF, or disable PF
- sysctl net.inet.tcp.rfc1323=0

>Release-Note:
>Audit-Trail:
>Unformatted: