Re: Weirdness with same-host IPv6 packets

From: Dustin Marquess <dmarquess_at_gmail.com>
Date: Tue, 07 Dec 2021 01:14:36 UTC
So tracing the traffic using tcpdump, the UDP packet itself is being
sent, it's just not making it to the process for some reason:

19:10:23.502313 IP6 (flowlabel 0xc0e2c, hlim 64, next-header UDP (17)
payload length: 16) 2001:470:bc52:4::101.56339 >
2001:470:bc52:4::101.5555: [bad udp cksum 0xc3b2 -> 0x9223!] UDP,
length 8

Doing TCP, it looks like it takes 3 tries before it finally makes it through:

19:12:27.284049 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 40) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [S], cksum 0xc3bf (incorrect ->
0x76e0), seq 827082993, win 65535, options [mss 16324,nop,wscale
14,sackOK,TS val 194207998 ecr 0], length 0
19:12:28.284624 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 40) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [S], cksum 0xc3bf (incorrect ->
0x72f7), seq 827082993, win 65535, options [mss 16324,nop,wscale
14,sackOK,TS val 194208999 ecr 0], length 0
19:12:30.486455 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 40) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [S], cksum 0xc3bf (incorrect ->
0x6a5d), seq 827082993, win 65535, options [mss 16324,nop,wscale
14,sackOK,TS val 194211201 ecr 0], length 0
19:12:30.486525 IP6 (flowlabel 0x81949, hlim 64, next-header TCP (6)
payload length: 40) 2001:470:bc52:4::101.5555 >
2001:470:bc52:4::101.55029: Flags [S.], cksum 0xc3bf (incorrect ->
0xde77), seq 3057129801, ack 827082994, win 65535, options [mss
16324,nop,wscale 14,sackOK,TS val 2005091535 ecr 194211201], length 0
19:12:30.486547 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 32) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [.], cksum 0xc3b7 (incorrect ->
0x4756), ack 1, win 5, options [nop,nop,TS val 194211201 ecr
2005091535], length 0
19:12:30.486653 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 40) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [P.], cksum 0xc3bf (incorrect ->
0x8ef3), seq 1:9, ack 1, win 5, options [nop,nop,TS val 194211201 ecr
2005091535], length 8
19:12:30.486667 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 32) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [F.], cksum 0xc3b7 (incorrect ->
0x474d), seq 9, ack 1, win 5, options [nop,nop,TS val 194211201 ecr
2005091535], length 0
19:12:30.486676 IP6 (flowlabel 0x81949, hlim 64, next-header TCP (6)
payload length: 32) 2001:470:bc52:4::101.5555 >
2001:470:bc52:4::101.55029: Flags [.], cksum 0xc3b7 (incorrect ->
0x474d), ack 10, win 5, options [nop,nop,TS val 2005091535 ecr
194211201], length 0
19:12:30.486758 IP6 (flowlabel 0x81949, hlim 64, next-header TCP (6)
payload length: 32) 2001:470:bc52:4::101.5555 >
2001:470:bc52:4::101.55029: Flags [F.], cksum 0xc3b7 (incorrect ->
0x474c), seq 1, ack 10, win 5, options [nop,nop,TS val 2005091535 ecr
194211201], length 0
19:12:30.722071 IP6 (flowlabel 0x81949, hlim 64, next-header TCP (6)
payload length: 32) 2001:470:bc52:4::101.5555 >
2001:470:bc52:4::101.55029: Flags [F.], cksum 0xc3b7 (incorrect ->
0x4661), seq 1, ack 10, win 5, options [nop,nop,TS val 2005091770 ecr
194211201], length 0
19:12:30.722094 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 40) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [FP.], cksum 0xc3bf (incorrect ->
0x8e07), seq 1:9, ack 1, win 5, options [nop,nop,TS val 194211436 ecr
2005091535], length 8
19:12:30.722150 IP6 (flowlabel 0x9b06b, hlim 64, next-header TCP (6)
payload length: 32) 2001:470:bc52:4::101.55029 >
2001:470:bc52:4::101.5555: Flags [.], cksum 0xc3b7 (incorrect ->
0x4577), ack 2, win 4, options [nop,nop,TS val 194211436 ecr
2005091770], length 0

Any ideas on how to further troubleshoot?

-Dustin

On Sat, Dec 4, 2021 at 3:54 PM Dustin Marquess <dmarquess@gmail.com> wrote:
>
> I'm seeing a weird issue with -CURRENT that I don't recall seeing
> before. It started at least a couple of weeks back and a new build
> from yesterday still shows it. UDP packets inside a host using the
> host's non-loopback address seems to get dropped. TCP does work,
> however there's a delay, almost like the first packet or two also got
> dropped. I don't have any firewalling active, and stopping the VNET
> jails didn't have any effect.
>
> I've been using the machine's local IPv6 IP in /etc/resolv.conf for a
> while. I noticed that logins were taking longer than usual and tracked
> it down to unbound not responding. If I change /etc/resolv.conf to use
> ::1 or the host's IPv4 IP, then it works fine. The host's IPv6 IP does
> work from outside the host, however. I thought it was maybe a weird
> unbound bug, so I did some testing with netcat.
>
> Current ifconfg (other interfaces removed for brevity):
>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
>
> bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
> ether 58:9c:fc:10:f4:55
> inet 192.168.4.101 netmask 0xffffff00 broadcast 192.168.4.255
> inet 192.168.4.12 netmask 0xffffffff broadcast 192.168.4.12
> inet6 2001:470:bc52:4::101 prefixlen 64
> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>         ifmaxaddr 0 port 16 priority 128 path cost 2000000
> member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>         ifmaxaddr 0 port 12 priority 128 path cost 2000000
> nd6 options=1<PERFORMNUD>
> groups: bridge
>
>
> Routing table:
>
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            192.168.4.1        UGS     bridge2
> 127.0.0.1          link#11            UH          lo0
> 192.168.4.0/24     link#19            U       bridge2
> 192.168.4.12       link#19            UH          lo0
> 192.168.4.101      link#19            UHS         lo0
>
> Internet6:
> Destination                       Gateway                       Flags
>    Netif Expire
> ::/96                             ::1                           UGRS        lo0
> default                           2001:470:bc52:4::1            UGS     bridge2
> ::1                               link#11                       UHS         lo0
> ::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
> 2001:470:bc52:4::/64              link#19                       U       bridge2
> 2001:470:bc52:4::101              link#19                       UHS         lo0
> fe80::/10                         ::1                           UGRS        lo0
> fe80::%lo0/64                     link#11                       U           lo0
> fe80::1%lo0                       link#11                       UHS         lo0
> ff02::/16                         ::1                           UGRS        lo0
>
> Testing:
>
> I started a listener:
>
> $ nc -6 -u -l 5555
>
> And in another window, did:
>
> $ echo testing | nc -6 -u ::1 5555   <-- Works
> $ echo testing | nc -6 -u 2001:470:bc52:4::101 5555   <-- Never
> receives the packet
> [ Previous command from a different host DOES work, however]
>
> Switching to TCP:
> $ echo testing | nc -6 ::1 5555  <--- Works
> $ echo testing | nc -6 2001:470:bc52:4::101 5555   <-- Works, after a
> delay however
>
> Trying IPv4:
> $ echo testing | nc -u 127.0.0.1 5555 <--- Works
> $ echo testing | nc -u 192.168.4.101 5555 <--- Works, no delay
>
> So IPv4 is working fine, which is strange.
>
> Has anybody else seen this and have any insight?
>
> -Dustin