(was: Regression with jails/IPv6/pf) 'scrub reassemble tcp' breaks IPv6 packet checksum on SYN ACK

[Mark Martinec]

Bjoern A. Zeeb wrote on 2012-08-01:
> Any of you who are expereincing problems with packets dropped due to
> invalid checksums with IPv6 and pf after the recent merges, can you
> report back if you also see this without "modulate state" in your
> pf.conf (if you have 'modulate' in there, can you try changing it to
> 'keep' and see if that fixes the problem)?

Indeed, invalid checksums with IPv6 and pf after the recent merges.
I've opened a PR (before finding about this thread):

  http://www.freebsd.org/cgi/query-pr.cgi?pr=172648


pf(4): 'scrub reassemble tcp' breaks IPv6 packet checksum on SYN ACK

When pf (packet filter) is enabled and configured with 'scrub reassemble tcp',
IPv6 TCP connections take 9 seconds to establish. Packet capture shows
checksum errors on SYN ACK packets but not on other packets.

A TCP connection establishment (SYN) on IPv6 is (re-)tried four times,
with a 3 second delay between each attempt, while the TCP options are
being simplified each time by the kernel (dropping ECN, CWR, window
scaling, and dropping a timestamp options). Only the fourth attempt
is successful, with no other options but SACK, and this TCP session
then proceeds normally.

Disabling 'scrub reassemble tcp' in the pf avoids the problem.
Similarly, turning off net.inet.tcp.rfc1323 on either end
also avoids the problem, even with 'reassemble tcp' enabled.

The problem does not occur on IPv4 sessions, only on IPv6.

The problem is not associated with interface checksum offloading,
it is repeatable on gif, em, and re interfaces. Also a packet capture
(wireshark) shows packet checksum errors on SYN ACK packets (but
not on the SYN packet) in the first couple of failed attempts, and
no checksum errors on other packets (e.g. after a successfully
established session).

My guess is that the TCP timestamp option triggers a pf bug,
which then miscalculates a packet checksum on SYN ACK.


How-To-Repeat
Use the following trivial pf config file:

 scrub all reassemble tcp
 pass all

Then try to establish any TCP session to any IPv6 address.
Any client will do (telnet, ssh, curl, web browser).
Try for example:
 curl -6 -L http://tools.ietf.org/rfc/rfc3021.txt | wc -l

The connection will 'hang' for 9 seconds (until a sufficiently
dumbed-down SYN options are tried), then it proceeds normally.


Fix
No known fix.

Two workarounds:
- don't use 'scrub reassemble tcp' in PF, or disable PF
- sysctl net.inet.tcp.rfc1323=0



  Mark