IPv6 system not responding to Neighbor Solicitation

From: William Carson <freebsd_at_dsllsn.net>
Date: Thu, 07 Apr 2022 21:22:47 UTC
Hello, recently after enabling ipv6_privacy in /etc/rc.conf and rebooting, I've been unable to get IPv6 connectivity to work in a hosted environment. (I don't know if this is a red herring or not.) I've tried disabling it, and even after rebooting, it still doesn't work. (Doesn't work meaning: I'm unable to ping6 hosts on the Internet that are reachable, e.g. ipv6.google.com.) I confirmed ipv6_privacy is actually disabled:

  # sysctl -a | grep tempaddr
  net.inet6.ip6.use_tempaddr: 0
  net.inet6.ip6.prefer_tempaddr: 0

If I boot into a Linux environment (the provider has a Rescue mode), I'm able to reach IPv6 just fine. Furthermore, if I then reboot back into FreeBSD 13.0-RELEASE-p10 it will work for around ~5 minutes and then connections time out.

Given the behavior and based on some tcpdumps, it looks like my system is not responding to the upstream router's Neighbor Solicitation messages. If I boot into Linux, it respond to the NS messages, the router caches the MAC address, and IPv6 works. If I'm fast enough and reboot into FreeBSD, IPv6 works until the the entry expires, and then I just see this:

13:24:58.901780 IP6 2600:3c00::f03c:91ff:feb0:a56f > 2605:6400:10:968:22:da15:28a6:c800: ICMP6, echo request, seq 40, length 16
13:24:59.277713 IP6 2600:3c00::8678:acff:fe1c:ec41 > ff02::1:ffb0:a56f: ICMP6, neighbor solicitation, who has 2600:3c00::f03c:91ff:feb0:a56f, length 32
13:24:59.277799 IP6 2600:3c00::8678:acff:fe1c:ec41 > ff02::1:ffb0:a56f: ICMP6, neighbor solicitation, who has 2600:3c00::f03c:91ff:feb0:a56f, length 32

3 packets, the echo request, then two NS requests, and no response -- and then it just repeats. 

I confirmed b0:a5:6f is the Device ID part of my MAC: 

  # ifconfig em0
  em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
          options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
          ether f2:3c:91:b0:a5:6f <---
          inet6 fe80::f03c:91ff:feb0:a56f%em0 prefixlen 64 scopeid 0x1
          inet6 2600:3c00::f03c:91ff:feb0:a56f prefixlen 64 autoconf
          inet6 2600:3c00:e000:137::1 prefixlen 128
          inet6 2600:3c00:e000:137::1:1 prefixlen 128
          inet6 2600:3c00:e000:137::2:1 prefixlen 128
          inet6 2600:3c00:e000:137::3:1 prefixlen 128
          inet6 2600:3c00:e000:137:cafe:8a2e:370:7334 prefixlen 128
          inet 96.126.127.161 netmask 0xffffff00 broadcast 96.126.127.255
          inet 173.255.203.45 netmask 0xffffffff broadcast 173.255.203.45
          inet 96.126.122.129 netmask 0xffffffff broadcast 96.126.122.129
          inet 50.116.26.213 netmask 0xffffffff broadcast 50.116.26.213
          media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          nd6 options=8023<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,DEFAULTIF>

Therefore the Solicited-node multicast address ff02::1:ffb0:a56f looks to be correct. I've also confirmed the router's address is within the assigned SLAAC network (Router: 2600:3c00::8678:acff:fe1c:ec41, SLAAC address: 2600:3c00::f03c:91ff:feb0:a56f/64).Furthermore, the multicast address does show up in `ifmcstat`:

  # ifmcstat
  em0:
          inet6 fe80::f03c:91ff:feb0:a56f%em0 scopeid 0x1
          mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
                  group ff02::1:ff70:7334%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:ff:70:73:34
                  group ff02::1:ff03:1%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:ff:03:00:01
                  group ff02::1:ff02:1%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:ff:02:00:01
                  group ff02::1:ff01:1%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:ff:01:00:01
                  group ff02::1:ff00:1%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:ff:00:00:01
          inet 96.126.127.161
          igmpv3 rv 2 qi 125 qri 10 uri 3
                  group 224.0.0.1 mode exclude
                          mcast-macaddr 01:00:5e:00:00:01
          inet6 fe80::f03c:91ff:feb0:a56f%em0 scopeid 0x1
          mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
                  group ff01::1%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:00:00:00:01
                  group ff02::2:bdc6:c84d%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:bd:c6:c8:4d
                  group ff02::2:ffbd:c6c8%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:ff:bd:c6:c8
                  group ff02::1%em0 scopeid 0x1 mode exclude
                          mcast-macaddr 33:33:00:00:00:01
                  group ff02::1:ffb0:a56f%em0 scopeid 0x1 mode exclude <---
                          mcast-macaddr 33:33:ff:b0:a5:6f

I can even ping the address and it replies!

  # ping6 ff02::1:ffb0:a56f
  PING6(56=40+8+8 bytes) fe80::f03c:91ff:feb0:a56f%em0 --> ff02::1:ffb0:a56f
  16 bytes from fe80::f03c:91ff:feb0:a56f%em0, icmp_seq=0 hlim=64 time=0.451 ms
  16 bytes from fe80::f03c:91ff:feb0:a56f%em0, icmp_seq=1 hlim=64 time=0.446 ms
  16 bytes from fe80::f03c:91ff:feb0:a56f%em0, icmp_seq=2 hlim=64 time=0.618 ms
  ^C

Does anyone have any thoughts why it's not responding to the Neighbor Solicitation messages? I've been troubleshooting this for a few days now and can't figure it out. I also tried booting kernel.old (which I think is -p8 or -p9), but it made no difference. I've tried with and without pf enabled -- again, no difference.

I don't know if this is useful, but I validated routes are being discovered:

  # ndp -na
  Neighbor                             Linklayer Address  Netif Expire    S Flags
  2600:3c00:e000:137::1:1              f2:3c:91:b0:a5:6f    em0 permanent R
  2600:3c00:e000:137::1                f2:3c:91:b0:a5:6f    em0 permanent R
  fe80::1%em0                          00:05:73:a0:0f:ff    em0 23h56m36s S R <---
  2600:3c00:e000:137::3:1              f2:3c:91:b0:a5:6f    em0 permanent R
  2600:3c00:e000:137::2:1              f2:3c:91:b0:a5:6f    em0 permanent R
  2600:3c00::f03c:91ff:feb0:a56f       f2:3c:91:b0:a5:6f    em0 permanent R
  fe80::f03c:91ff:feb0:a56f%em0        f2:3c:91:b0:a5:6f    em0 permanent R
  fe80::8678:acff:fe1c:ec41%em0        84:78:ac:1c:ec:41    em0 23h49m7s  S R <---
  2600:3c00:e000:137:cafe:8a2e:370:7334 f2:3c:91:b0:a5:6f   em0 permanent R

  # netstat -nr6
  Routing tables

  Internet6:
  Destination                       Gateway                       Flags     Netif Expire
  ::/96                             ::1                           UGRS        lo0
  default                           fe80::1%em0                   UG          em0 <---
  ::1                               link#2                        UHS         lo0
  ::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
  2600:3c00::f03c:91ff:feb0:a56f    link#1                        UHS         lo0
  2600:3c00:e000:137::1             link#1                        UHS         lo0
  2600:3c00:e000:137::1:1           link#1                        UHS         lo0
  2600:3c00:e000:137::2:1           link#1                        UHS         lo0
  2600:3c00:e000:137::3:1           link#1                        UHS         lo0
  2600:3c00:e000:137:cafe:8a2e:370:7334 link#1                    UHS         lo0
  fe80::/10                         ::1                           UGRS        lo0
  fe80::%em0/64                     link#1                        U           em0
  fe80::f03c:91ff:feb0:a56f%em0     link#1                        UHS         lo0
  fe80::%lo0/64                     link#2                        U           lo0
  fe80::1%lo0                       link#2                        UHS         lo0
  ff02::/16                         ::1                           UGRS        lo0

And here's the IPv6 part in my rc.conf:

  # ipv6
  rtsold_enable="YES"
  rtsold_flags="-aF"
  #ipv6_activate_all_interfaces="YES"
  ipv6_network_interfaces="em0"
  ipv6_default_interface="em0"
  ifconfig_em0_ipv6="inet6 accept_rtadv"
  ifconfig_em0_aliases="\
                 inet6 2600:3c00:e000:0137::0:1/128 \
                 inet6 2600:3c00:e000:0137::1:1/128 \
                 inet6 2600:3c00:e000:0137::2:1/128 \
                 inet6 2600:3c00:e000:0137::3:1/128 \
                 inet6 2600:3c00:e000:0137:cafe:8a2e:0370:7334/128"

I'm at a complete loss. Any help troubleshooting this would be greatly appreciated.

Sincerely,
--wcarson