Jail, VNET and IPv6

[Jacques Foucry]

Le jeudi 10 déc. 2020 à 20:37:28 (+0100), satanist à écrit:
> Hi Jacques
> 
> [2020-12-10 10:30] Jacques Foucry <jacques+freebsd at foucry.net>
> > I manage on a hosted server many « clasical » jail with ip adresses as alias of
> > em0.
> >
> > I would like to make a new jail, but using VNET and ipv6. All my tries failed
> > :-( IPv4 work great but IPv6 not.
> 
> Would be nice if you share the concept of your network setup. As far as
> I have understand from your mail it looks like this:
> 
>                                        ------------
>                                        | Jail     |
> [em0] <-> [bridge0] <-> [epair10a] <-> |[epair10b]|


That it!

>                                        ------------
> > netstat -rn
> > [v4output]
> >
> > Internet6:
> > Destination                       Gateway                       Flags     Netif Expire
> > [v6routes]
> > 2a01:4f9:4a:1fd8::/64             link#1                        U           em0
> 
> I think here is the problem. You have the route to your jail on the em0
> interface and not on the bridge. Handbook[0] says:
> 
> > If the bridge host needs an IP address, set it on the bridge interface,
> > not on the member interfaces.
> 
> I would asume this is also true for routes. I asume if you _send_ packages
> on em0 they never reache the bridge.


So I need to add an IPv6 to the bridge and use it as default router for mや
jail if I correctly understand.
> 
> > ifconfig
> > em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > 	options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
> > 	ether b4:2e:99:6a:80:9d
> > 	inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
> > 	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
> > 	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::28 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
> > 	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
> > 	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
> > 	media: Ethernet autoselect (1000baseT <full-duplex>)
> > 	status: active
> > 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > [other interfaces]
> > bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > 	description: vnet-jail-bridge
> > 	ether 02:36:b3:c1:8a:00
> > 	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
> > 	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> > 	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> > 	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> > 	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> > 	        ifmaxaddr 0 port 1 priority 128 path cost 20000
> > 	groups: bridge
> > 	nd6 options=1<PERFORMNUD>
> 
> For v6 the adresses are on em0 for v4 they are on bridge0. Therefore
> v4 works but v6 don't.


All the v6 attached to em0 are "classical" jail, wihout VNET.
> 
> > As you can see thereis a bridge (bridg0) with an IPv4 10.0.0.1/24. PF assume
> > the nat fonction for this range to 10.0.010/24 the new jail IPv4.
> 
> Thise seames strange. You bridge your internal network to the external,
> but also NAT the internel Network. This has some odd side effects. Your
> Jails can ackt like a Host on your upstream-network and every host on
> your upstream-network can ackt like it's just an other jail.
> 
> > [jail config]
> >    exec.start       += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
> > [...]
> >
> > epair10a on the host:
> >
> > epair10a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > 	description: vnet-jitsi
> > 	options=8<VLAN_MTU>
> > 	ether 02:dc:c8:b1:ac:0a
> > 	inet6 fe80::dc:c8ff:feb1:ac0a%epair10a prefixlen 64 scopeid 0x6
> > 	groups: epair
> > 	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> > 	status: active
> > 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> 
> Again the problem with addresses on interfaces in a bridge.
> 
> > I must miss something, or misunderstood something…
> >
> > Any advices are welcome. 
> 
> If you want to continue with a bridged setup I would say you need to
> move the ipv6 config from em0 to bridge0. I would recommend to switch
> to a routed setup.
> 
> satanist
> 
> [0] https://www.freebsd.org/doc/handbook/network-bridging.html


Thanks for your advice and pointers, I will check that.

Regards, and take care.
-- 
Jacques Foucry