FreeBSD does not reply to IPv6 Neighbor Solicitations

[John-Mark Gurney]

Andrey V. Elsukov wrote this message on Tue, Jan 12, 2021 at 16:33 +0300:
> On 12.01.2021 05:25, John-Mark Gurney wrote:
> >> The device, where the capture was taken does not respond tot he NS packet.
> >> This might be caused by:
> >>  a) the device has a different configured IP address, than requested
> >>  b) the network card does not listen to the multicast group, which is
> >>     used by the request (you see it only due to the promisc mode of the
> >>     capture). But this is unlikely (due to the promisc mode)
> >>  c) your system is broken
> > 
> > I have some test scripts where something similar to this happens.
> > 
> > I tcpdump shows the request coming into the FreeBSD box (in this case,
> > 13-current main-c255640-gc38e59ce1b0), addressed to the IPv6 of the
> > box, and FreeBSD failing to respond w/ an answer for it's own IP...
> > 
> > This is inconsistent and hard to reproduce, but it does happen with
> > somewhat regularity.
> 
> when this will happen again, it would be nice to make sure that NS
> packets hit the IP stack. E.g. with attached dtrace script.

Ok, I ran the dtrace script when I reproduced the problem, and it did
not produce any output.

Here are the steps that I use to setup the interfaces for this test,
which seems to trigger it...  This is on a single machine, usings the
on board bge0 and a USB ure...  Each interface is put in their own
vnet jail...  I've attached the script, just need to init the jails,
then run the test a couple of times:

sh testinterfaces.sh init ue0 bge0
sh testinterfaces.sh csum ue0 bge0
sh testinterfaces.sh csum ue0 bge0

When it fails, you'll see something like:
[...]
ping FAILED on -rxcsum6!!!!                              
27018 27019                                              
test failed, data in /tmp/testiface.moLZMgFH                                                                      
csumtest failed.                                         

the numbers are the pid's of the tcpdumps that were run... There are also
pcaps of the two sides in the directory...

These are effectively what the script does:
1) start w/ both interfaces in down state (they have ipv6 addresses set
   from previous run)
2) configure csum flags on ure (in this case, -rxcsum6)
3) verify that flag is cleared
4) disable all csum flags on bge0 (-txcsum -rxcsum -txcsum6 -rxcsum6)
5) verify that -txcsum and -rxcsum on bge0 (just realized this is a bug
   in my script that I don't verify the ipv6 versions of the flag)
6) sleep for .5 seconds
7) bring up ure
8) bring up bge0
9) configure inet6 addresses on ure and bge (duplicating the addresses
   already configured)
10) wait for both interfaces to have link, AND the inet6 addresses to
    not be in tentative state
11) sleep .5
12) run ping, and see it fail due to the ND problem

I ran the dtrace script in the host system, which iirc, should be fine.

If you enable the removal of the inet6 addresses in cleaniface, this
error does not seem to trigger...

> Also net.inet6.icmp6.nd6_debug variable should be set to see error
> messages from ND code.

I set it in the jail, and did not see any messages... I reset the
interface, and was able to see the messages when it "worked", so
it looks like the packets aren't hitting the ip stack properly...

> If it doesn't show expected info, this means that packets don't hit IP
> stack. Probably some multicast related problem. In this case it could be
> useful to obtain output of ifmcstat(8).

So, the output of ifmcstat when it isn't working:
root at test:~ # jexec checkjail ifmcstat
lo0:
        inet6 fe80::1%lo0 scopeid 0x1
sysctl net.inet6.mld.ifinfo: No such file or directory
                group ff01::1%lo0 scopeid 0x1 mode exclude
                group ff02::1%lo0 scopeid 0x1 mode exclude
                group ff02::1:ff00:1%lo0 scopeid 0x1 mode exclude
bge0:
        inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
        mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
                group ff01::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:ff:xx:xx:xx

I'm not sure why there's an error on net.inet6.mld.ifinfo, as both my
kernel and userland should be in sync, as of Jan 9th...

so, I made things works, and ran ifmcstat again, and this time it has
an additional group in the output:
[...]
bge0:
        inet6 fe80::12e7:c6ff:fexx:xxxx%bge0 scopeid 0x2
        mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
                group ff02::1:ff00:c43c%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:ff:00:c4:3c
                group ff01::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:00:00:00:01
                group ff02::1:ffxx:xxxx%bge0 scopeid 0x2 mode exclude
                        mcast-macaddr 33:33:ff:xx:xx:xx

and the tcpdump output:
21:10:53.938655 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32
21:10:55.001428 IP6 fc00:b5d:41c:7e37::7e37 > ff02::1:ff00:c43c: ICMP6, neighbor solicitation, who has fc00:b5d:41c:7e37::c43c, length 32

In this case, the interface that is having issues is bge0, so it's a
"real" NIC and should be well supported...

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."