IPv6 TCP: first two SYN packets to local v6 unicast addresses ignored

[bcc to net@ for wider exposure]

Hi,

there seems to be an issue with local IPv6 TCP connections on main. I 
have been seeing this for a couple of months at least. pkg upgr on my 
webserver hosting the pkg repo is very slow, all other hosts can connect 
to the pkg repo just fine. So IPv6 connections from external hosts are 
not affected.

I thought I must have misconfigured something, as my setup is a bit 
weird. Yesterday I noticed the same issue on a different host, turns out 
all my 14.0 hosts seem to be affected, cognet@ could also reproduce it 
on one of his systems.

The service/software used does not seem to matter, I tried with port 22, 
25, 80 and 443.

ICMP and UDP don't seem to be affected. ping6 gets replies immediately. 
And UDP connections with nc -l -u / nc -u don't have any delay, sent 
data is received immediately.

Testing local TCP connections show this:

flo@rp64:~ $ ifconfig dwc0|grep 2003
	inet6 2003:cf:df49:c97:4c59:ebff:fec1:463d prefixlen 64 autoconf
flo@rp64:~ $ nc -v 2003:cf:df49:c97:4c59:ebff:fec1:463d 22
[3 second delay here]
Connection to 2003:cf:df49:c97:4c59:ebff:fec1:463d 22 port [tcp/ssh] 
succeeded!
SSH-2.0-OpenSSH_8.9 FreeBSD-20220413

tcpdump on lo0 shows that the first two SYN packets are ignored / time 
out, then the connection is successfully established.

19:28:38.685128 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > 
2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [S], seq 2489479594, win 
65535, options [mss 16324,nop,wscale 6,sackOK,TS val 3410505643 ecr 0], 
length 0

19:28:39.696047 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > 
2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [S], seq 2489479594, win 
65535, options [mss 16324,nop,wscale 6,sackOK,TS val 3410506654 ecr 0], 
length 0

19:28:41.897836 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > 
2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [S], seq 2489479594, win 
65535, options [mss 16324,nop,wscale 6,sackOK,TS val 3410508856 ecr 0], 
length 0

19:28:41.897907 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.22 > 
2003:cf:df49:c97:4c59:ebff:fec1:463d.61294: Flags [S.], seq 2857552476, 
ack 2489479595, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 
1858349482 ecr 3410508856], length 0

19:28:41.897962 IP6 2003:cf:df49:c97:4c59:ebff:fec1:463d.61294 > 
2003:cf:df49:c97:4c59:ebff:fec1:463d.22: Flags [.], ack 1, win 1276, 
options [nop,nop,TS val 3410508856 ecr 1858349482], length 0

I need help debugging this, I don't know how to analyze this further. I 
will start bisecting this, but I thought maybe someone has an idea.

Thanks,
Florian