Re: IPv6 inconsistent local routing

On 20 Oct 2021, at 18:22, Peter wrote:

> Hello all,
>
> with IPv4 the handling of local traffic was simple: all local traffic
> would go in and out of the 'lo0' interface, no matter which ip-address
> was used or on which interface that address would be placed. This gets
> also obvious from the routing table:
>
>
> root@xxxx:~ # ifconfig
> vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 
> mtu 1500
>         inet 192.168.97.15 netmask 0xffffffe0 broadcast 192.168.97.31
>         inet6 fd00::1 prefixlen 64
>
> root@xxxx:~ # netstat -rn
> Routing tables
>
> Internet:
> Destination        Gateway            Flags     Netif Expire
> 127.0.0.1          link#2             UH          lo0
> 192.168.97.0/27    link#1             U        vtnet0
> 192.168.97.15      link#1             UHS         lo0 <<<<<<
>
> While the network /27 appears with 'vtnet0', the local
> address itself is at 'lo0'.
>
> Practically it looks like this:
>
>
> root@xxxx:~ # ping 192.168.97.15
> ipfw: 1 Accept ICMP:8.0 127.0.0.1 192.168.97.15 out via lo0
> ipfw: 1 Accept ICMP:8.0 127.0.0.1 192.168.97.15 in via lo0
> ipfw: 1 Accept ICMP:0.0 192.168.97.15 127.0.0.1 out via lo0
> ipfw: 1 Accept ICMP:0.0 192.168.97.15 127.0.0.1 in via lo0
>
> root@xxxx:~ # telnet 192.168.97.15 7777
> ipfw: 1 Accept TCP 192.168.97.15:52401 192.168.97.15:7777 out via lo0
> ipfw: 1 Accept TCP 192.168.97.15:52401 192.168.97.15:7777 in via lo0
> ipfw: 1 Accept TCP 192.168.97.15:7777 192.168.97.15:52401 out via lo0
> ipfw: 1 Accept TCP 192.168.97.15:7777 192.168.97.15:52401 in via lo0
>
> All traffic goes thru 'lo0' (and there we could filter it if e.g. we
> want to filter between non-vimage jails).
>
> But now look at IPv6:
>
>
> root@xxxx:~ # ping fd00::1
> ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] out via lo0
> ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] in via lo0
> ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] out via lo0
> ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] in via lo0
>
> root@xxxx:~ # telnet fd00::1 7777
> ipfw: 1 Accept TCP [fd00::1]:53821 [fd00::1]:7777 out via lo0
> ipfw: 1 Accept TCP [fd00::1]:53821 [fd00::1]:7777 in via vtnet0  <<<<<
> ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:53821 out via lo0
> ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:53821 in via lo0
>
>
> Upsala! What's the 'vtnet0' doing there??
>
> But that's not yet the full story. This is 13-stable or RELEASE-13.0.
> And now RELEASE 12.2 (or 12.3-PRERELEASE):
>
>
> root@yyyy:~ # ping6 fd00::1                                            
>          ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] out via lo0
> ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] in via vtnet0
> ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] out via lo0
> ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] in via vtnet0
>
> root@yyyy:~ # telnet fd00::1 7777
> ipfw: 1 Accept TCP [fd00::1]:60375 [fd00::1]:7777 out via lo0
> ipfw: 1 Accept TCP [fd00::1]:60375 [fd00::1]:7777 in via vtnet0
> ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:60375 out via lo0
> ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:60375 in via vtnet0
>
>
> Urgs.
>
> Lets conclude: the behaviour is
>  *  inkonsistent between IPv4 and IPv6
>  *  inkonsistent between incoming and outgoing
>  *  inkonsistent between originate and answer flow
>  *  inkonsistent between protocols (ICMP vs. TCP)
>  *  inkonsistent between releases
>
>
> And now I have a problem. I'm trying to enhance my graphical frontend
> to also work with IPv6:
> https://forums.freebsd.org/threads/nice-or-ugly.81298/post-522355
>
> But how should I code that, when the behaviour is different with every
> usecase?
> And whom might I ask how it would look if it finally settles? (The
> ipfw mailinglist seems mostly empty.)
>
>
>
> You might say, it does not usually make sense to route traffic 
> local-to-
> local over an IPadress of an outbound interface. But actually this
> is common practice in e.g. failover scenarios, where you don't know
> in the app configs if your address is currently local or not - you 
> just
> know it carries the service.


There used to be:

$ sysctl -d net.link.ether.inet.useloopback 
net.inet6.icmp6.nd6_useloopback
net.link.ether.inet.useloopback: Use the loopback interface for local 
traffic
net.inet6.icmp6.nd6_useloopback:

which defined this bahviour.

These were removed in/after the FreeBSD 10 times I think?
I’d start looking from there.



> It was mentioned that one should usually ignore local traffic in the
> firewall rules. But that is not the point: as these packets are 
> labelled
> with the external interface, they will go thru the ruleset for the
> external interface anyway. And there they will be dropped, because
> nobody there knows about them.
>
>
> This has also an effect on the tcp dyn keepalive feature.
> (net.inet.ip.fw.dyn_keepalive) These keepalives usually go
> thru lo0 incoming. But, with IPv6 on Rel. 12 they go thru the external
> interface, incoming. Then on Rel. 13 they go thru lo0 again.
> (And I didn't try Rel. 14 )
>
>
> Cheerio,
> PMc