IPv6 inconsistent local routing

From: Peter <pmc_at_citylink.dinoex.sub.org>
Date: Wed, 20 Oct 2021 18:22:37 UTC
                                                                                
Hello all,                                                                      
                                                                                
with IPv4 the handling of local traffic was simple: all local traffic           
would go in and out of the 'lo0' interface, no matter which ip-address          
was used or on which interface that address would be placed. This gets          
also obvious from the routing table:                                            
                                                                                
                                                                                
root@xxxx:~ # ifconfig                                                          
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500    
        inet 192.168.97.15 netmask 0xffffffe0 broadcast 192.168.97.31           
        inet6 fd00::1 prefixlen 64                                              
                                                                                
root@xxxx:~ # netstat -rn                                                       
Routing tables                                                                  
                                                                                
Internet:                                                                       
Destination        Gateway            Flags     Netif Expire                    
127.0.0.1          link#2             UH          lo0                           
192.168.97.0/27    link#1             U        vtnet0                           
192.168.97.15      link#1             UHS         lo0 <<<<<<                    
                                                                                
While the network /27 appears with 'vtnet0', the local                          
address itself is at 'lo0'.                                                     
                                                                                
Practically it looks like this:                                                 
                                                                                
                                                                                
root@xxxx:~ # ping 192.168.97.15                                                
ipfw: 1 Accept ICMP:8.0 127.0.0.1 192.168.97.15 out via lo0                     
ipfw: 1 Accept ICMP:8.0 127.0.0.1 192.168.97.15 in via lo0                      
ipfw: 1 Accept ICMP:0.0 192.168.97.15 127.0.0.1 out via lo0                     
ipfw: 1 Accept ICMP:0.0 192.168.97.15 127.0.0.1 in via lo0                      
                                                                                
root@xxxx:~ # telnet 192.168.97.15 7777                                         
ipfw: 1 Accept TCP 192.168.97.15:52401 192.168.97.15:7777 out via lo0           
ipfw: 1 Accept TCP 192.168.97.15:52401 192.168.97.15:7777 in via lo0            
ipfw: 1 Accept TCP 192.168.97.15:7777 192.168.97.15:52401 out via lo0           
ipfw: 1 Accept TCP 192.168.97.15:7777 192.168.97.15:52401 in via lo0            
                                                                                
All traffic goes thru 'lo0' (and there we could filter it if e.g. we            
want to filter between non-vimage jails).                                       
                                                                                
But now look at IPv6:                                                           
                                                                                
                                                                                
root@xxxx:~ # ping fd00::1                                                      
ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] out via lo0                     
ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] in via lo0                      
ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] out via lo0                     
ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] in via lo0                      
                                                                                
root@xxxx:~ # telnet fd00::1 7777                                               
ipfw: 1 Accept TCP [fd00::1]:53821 [fd00::1]:7777 out via lo0                   
ipfw: 1 Accept TCP [fd00::1]:53821 [fd00::1]:7777 in via vtnet0  <<<<<          
ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:53821 out via lo0                   
ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:53821 in via lo0                    
                                                                                
                                                                                
Upsala! What's the 'vtnet0' doing there??                                       
                                                                                
But that's not yet the full story. This is 13-stable or RELEASE-13.0.           
And now RELEASE 12.2 (or 12.3-PRERELEASE):                                      
                                                                                
                                                                                
root@yyyy:~ # ping6 fd00::1                                                     ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] out via lo0                     
ipfw: 1 Accept ICMPv6:128.0 [fd00::1] [fd00::1] in via vtnet0                   
ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] out via lo0                     
ipfw: 1 Accept ICMPv6:129.0 [fd00::1] [fd00::1] in via vtnet0                   
                                                                                
root@yyyy:~ # telnet fd00::1 7777                                               
ipfw: 1 Accept TCP [fd00::1]:60375 [fd00::1]:7777 out via lo0                   
ipfw: 1 Accept TCP [fd00::1]:60375 [fd00::1]:7777 in via vtnet0                 
ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:60375 out via lo0                   
ipfw: 1 Accept TCP [fd00::1]:7777 [fd00::1]:60375 in via vtnet0                 
                                                                                
                                                                                
Urgs.                                                                           
                                                                                
Lets conclude: the behaviour is                                                 
 *  inkonsistent between IPv4 and IPv6                                          
 *  inkonsistent between incoming and outgoing                                  
 *  inkonsistent between originate and answer flow                              
 *  inkonsistent between protocols (ICMP vs. TCP)                               
 *  inkonsistent between releases                                               
                                                                                
                                                                                
And now I have a problem. I'm trying to enhance my graphical frontend           
to also work with IPv6:
https://forums.freebsd.org/threads/nice-or-ugly.81298/post-522355               
                                                                                
But how should I code that, when the behaviour is different with every          
usecase?                                                                        
And whom might I ask how it would look if it finally settles? (The              
ipfw mailinglist seems mostly empty.)                                           
                                                                                
                                                                                
                                                                                
You might say, it does not usually make sense to route traffic local-to-        
local over an IPadress of an outbound interface. But actually this              
is common practice in e.g. failover scenarios, where you don't know             
in the app configs if your address is currently local or not - you just         
know it carries the service.                                                    
                                                                                
It was mentioned that one should usually ignore local traffic in the            
firewall rules. But that is not the point: as these packets are labelled        
with the external interface, they will go thru the ruleset for the              
external interface anyway. And there they will be dropped, because              
nobody there knows about them.                                                  
                                                                                
                                                                                
This has also an effect on the tcp dyn keepalive feature.                       
(net.inet.ip.fw.dyn_keepalive) These keepalives usually go                      
thru lo0 incoming. But, with IPv6 on Rel. 12 they go thru the external          
interface, incoming. Then on Rel. 13 they go thru lo0 again.                    
(And I didn't try Rel. 14 )                                                     
                                                                                
                                                                                
Cheerio,                                                                        
PMc