Re: Regression with pf or IPv6 on FreeBSD 14 with IPsec gif(4) tunnel

From: Xin Li <delphij_at_delphij.net>
Date: Wed, 20 Sep 2023 09:02:25 UTC 
On 2023-09-15 4:43 AM, mike tancsa wrote:
> On 9/15/2023 1:38 AM, Xin Li wrote:
>> On 2023-09-14 3:28 AM, Kristof Provost wrote:
>>> On 14 Sep 2023, at 4:54, Xin Li wrote:
>>>> Hi!
>>>> And as a shoot to the dark, I tried again with IPsec (racoon) 
>>>> disabled, and the issue is gone.  My IPsec configuration is fairly 
>>>> common: 
>>
>> I'm still comparing the code and reading the history of changes 
>> between stable/13 and stable/14 to see if there are something obvious, 
>> but more insights from others would be appreciated :)
>>
> [/me takes a in the dark]
> 
> maybe something like net.inet.ipsec.filtertunnel=1 is needed now ?

I think this won't change (I've also tried to set 
net.inet6.ipsec6.filtertunnel=1).

I did some additional experiments and it seems that the last thing that 
the offending process was doing was to write 1504 bytes to the TCP 
socket, which appears to be the SSH negotiation where it is sending 
acceptable ciphers.

tcpdump on the gif(4) interface gave me:

IP6 SERVERv6.22 > LOCALv6.53306: Flags [P.], seq 1:39, ack 1, win 1026, 
options [nop,nop,TS val 4056875356 ecr 3286251611], length 38: SSH: 
SSH-2.0-OpenSSH_9.3 FreeBSD-20230719
IP6 LOCALv6.53306 > SERVERv6.22: Flags [P.], seq 1:22, ack 39, win 129, 
options [nop,nop,TS val 3286251669 ecr 4056875356], length 21: SSH: 
SSH-2.0-OpenSSH_9.4
IP6 LOCALv6.53306 > SERVERv6.22: Flags [.], seq 22:1350, ack 39, win 
129, options [nop,nop,TS val 3286251692 ecr 4056875356], length 1328
IP6 LOCALv6.53306 > SERVERv6.22: Flags [.], seq 22:1350, ack 39, win 
129, options [nop,nop,TS val 3286251692 ecr 4056875356], length 1328
IP6 LOCALv6.53306 > SERVERv6.22: Flags [.], seq 22:1350, ack 39, win 
129, options [nop,nop,TS val 3286251692 ecr 4056875356], length 1328
IP6 LOCALv6.53306 > SERVERv6.22: Flags [.], seq 22:1350, ack 39, win 
129, options [nop,nop,TS val 3286251692 ecr 4056875356], length 1328
IP6 LOCALv6.53306 > SERVERv6.22: Flags [.], seq 22:1350, ack 39, win 
129, options [nop,nop,TS val 3286251692 ecr 4056875356], length 1328
[...repeats...]

The mtu on the gif(4) interface was 1400.  It seems that these packets 
only showed up on gif(4) interface, but didn't get actually sent out on 
the physical interface (where the packets are supposed to be 
encapsulated in ESP).

Another discovery is that disabling path MTU discovery appears to 
mitigate the issue (sysctl net.inet.tcp.path_mtu_discovery=0).  FreeBSD 
14 have added support of PMTUD for IPv6 tunnels, so maybe this is 
somehow related?  (unfortunately I'm not yet familiar enough with the 
code to tell).

Cheers,