[Bug 257499] IPv6 link routes are not removed from routing table when interface goes down

From: <bugzilla-noreply_at_freebsd.org>
Date: Fri, 30 Jul 2021 08:45:01 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257499

            Bug ID: 257499
           Summary: IPv6 link routes are not removed from routing table
                    when interface goes down
           Product: Base System
           Version: 13.0-STABLE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: alexis.savin@efficientip.com

Hello FreeBSD community,

I'm facing some kind of inconsistent behavior with the kernel's IPv6
implementation.

Directly Connected routes (link routes) are consistently created at startup or
when an network interface is created.

However, these routes remain in the routing table when associated network
interface goes down.
If then deleted (manually or by a specific process such as quagga), these
routes are not restored in the routing table when the interface goes back up,
leading to unexpected behavior.

Tested on 11.4-STABLE and 13.0-STABLE.

Step to Reproduce:
1. Configure an additional network interface on a FreeBSD device with IPv6 and
get it up and running (connected + up)
```
ifconfig_em1="up  media autoselect -txcsum"
ifconfig_em1_alias0="inet 10.10.10.24 netmask 255.255.255.0"
ifconfig_em1_ipv6="inet6 2607:5300:203:2e61::2 prefixlen 64"

```

2. Observe that interface is up and that link routes are pushed into the
routing table (both IPv4 and IPv6)
```
# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:57:85:9f
        hwaddr 08:00:27:57:85:9f
        inet6 fe80::a00:27ff:fe57:859f%em1 prefixlen 64 scopeid 0x2
        inet6 2607:5300:203:2e61::2 prefixlen 64
        inet 10.10.10.24 netmask 0xffffff00 broadcast 10.10.10.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.254      UGS         em0
10.10.10.0/24      link#2             U           em1
10.10.10.24        link#2             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.1.0/24     link#1             U           em0
192.168.1.18       link#1             UHS         lo0

# netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif
Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2607:5300:203:2e61::/64           link#2                        U           em1
2607:5300:203:2e61::2             link#2                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em1/64                     link#2                        U           em1
fe80::a00:27ff:fe57:859f%em1      link#2                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

```

3. Take down the interface em1 and observe that IPv4 link routes
(10.10.10.0/24) are removed from the routing table while IPv6 link routes
(2607:5300:203:2e61::/64, 2607:5300:203:2e61::2 and fe80::%em1/64) remain.
```
# ifconfig em1 down
# ifconfig em1 
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:57:85:9f
        hwaddr 08:00:27:57:85:9f
        inet6 fe80::a00:27ff:fe57:859f%em1 prefixlen 64 tentative scopeid 0x2
        inet6 2607:5300:203:2e61::2 prefixlen 64 tentative
        inet 10.10.10.24 netmask 0xffffff00 broadcast 10.10.10.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.254      UGS         em0
10.10.10.24        link#2             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.1.0/24     link#1             U           em0
192.168.1.18       link#1             UHS         lo0
# netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif
Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2607:5300:203:2e61::/64           link#2                        U           em1
2607:5300:203:2e61::2             link#2                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em1/64                     link#2                        U           em1
fe80::a00:27ff:fe57:859f%em1      link#2                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
```

4. Delete remaining routes
```
route delete -inet6 fe80::%em1/64
route delete -inet6 2607:5300:203:2e61::2
route delete -inet6 2607:5300:203:2e61::/64

# netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif
Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::a00:27ff:fe57:859f%em1      link#2                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
```

# Bring the interface back up, observer that link routes are not restored
```
# ifconfig em1 up
# ifconfig em1 
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:57:85:9f
        hwaddr 08:00:27:57:85:9f
        inet6 fe80::a00:27ff:fe57:859f%em1 prefixlen 64 scopeid 0x2
        inet6 2607:5300:203:2e61::2 prefixlen 64
        inet 10.10.10.24 netmask 0xffffff00 broadcast 10.10.10.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
# netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif
Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::a00:27ff:fe57:859f%em1      link#2                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

```

From my understanding, this is definitely not the expected behavior.
It's not consistent regarding FreeBSD IPv4 implementation, nor consistant
regarding other kernel implementation.
On Linux for instance, these link routes are taken down when network interface
goes down and brough back once back up.
So unless I'm missing some specific configuration setting, I suspect an issue
with the IPv6 network stack.

I guess this doesn't affect everybody, we figured it out while using dynamic
routing (Quagga/FRR +BGP).

Thank in advance for your feedback

Kind regards

-- 
You are receiving this mail because:
You are the assignee for the bug.