Jail, VNET and IPv6

[satanist]

Hi Jacques

[2020-12-10 10:30] Jacques Foucry <jacques+freebsd at foucry.net>
> I manage on a hosted server many « clasical » jail with ip adresses as alias of
> em0.
>
> I would like to make a new jail, but using VNET and ipv6. All my tries failed
> :-( IPv4 work great but IPv6 not.

Would be nice if you share the concept of your network setup. As far as
I have understand from your mail it looks like this:

                                       ------------
                                       | Jail     |
[em0] <-> [bridge0] <-> [epair10a] <-> |[epair10b]|
                                       ------------
> netstat -rn
> [v4output]
>
> Internet6:
> Destination                       Gateway                       Flags     Netif Expire
> [v6routes]
> 2a01:4f9:4a:1fd8::/64             link#1                        U           em0

I think here is the problem. You have the route to your jail on the em0
interface and not on the bridge. Handbook[0] says:

> If the bridge host needs an IP address, set it on the bridge interface,
> not on the member interfaces.

I would asume this is also true for routes. I asume if you _send_ packages
on em0 they never reache the bridge.

> ifconfig
> em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
> 	ether b4:2e:99:6a:80:9d
> 	inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
> 	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
> 	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::28 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
> 	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
> 	media: Ethernet autoselect (1000baseT <full-duplex>)
> 	status: active
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> [other interfaces]
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	description: vnet-jail-bridge
> 	ether 02:36:b3:c1:8a:00
> 	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
> 	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> 	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> 	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> 	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> 	        ifmaxaddr 0 port 1 priority 128 path cost 20000
> 	groups: bridge
> 	nd6 options=1<PERFORMNUD>

For v6 the adresses are on em0 for v4 they are on bridge0. Therefore
v4 works but v6 don't.

> As you can see thereis a bridge (bridg0) with an IPv4 10.0.0.1/24. PF assume
> the nat fonction for this range to 10.0.010/24 the new jail IPv4.

Thise seames strange. You bridge your internal network to the external,
but also NAT the internel Network. This has some odd side effects. Your
Jails can ackt like a Host on your upstream-network and every host on
your upstream-network can ackt like it's just an other jail.

> [jail config]
>    exec.start       += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
> [...]
>
> epair10a on the host:
>
> epair10a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	description: vnet-jitsi
> 	options=8<VLAN_MTU>
> 	ether 02:dc:c8:b1:ac:0a
> 	inet6 fe80::dc:c8ff:feb1:ac0a%epair10a prefixlen 64 scopeid 0x6
> 	groups: epair
> 	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> 	status: active
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Again the problem with addresses on interfaces in a bridge.

> I must miss something, or misunderstood something…
>
> Any advices are welcome. 

If you want to continue with a bridged setup I would say you need to
move the ipv6 config from em0 to bridge0. I would recommend to switch
to a routed setup.

satanist

[0] https://www.freebsd.org/doc/handbook/network-bridging.html