Tunnel interfaces and vnet boundary crossing

From: Milan Obuch <freebsd-net_at_dino.sk>
Date: Mon, 15 Aug 2022 06:53:03 UTC
Hi,

some time ago I managed to design and implement multi-tenant OpenVPN
server using vnet jails. This way I am able to use more OpenVPN
instances on single public IP.

This is made possible using tun/tap interface property allowing to
cross vnet boundary - here is part of my initialisation command sequence
for one instance:

jail -c name=ov1 vnet persist
jexec ov1 hostname -s ov1
jexec ov1 ifconfig lo0 127.0.0.1/8
jexec ov1 sysctl net.inet.ip.forwarding=1
ifconfig tun1 create vnet ov1
/usr/local/sbin/openvpn --cd /usr/local/etc/openvpn --daemon ov1 --config ov1.cfg --writepid /var/run/ov1.pid

In ov1.cfg, relevant bits are

port 1001
management localhost 2001
dev tun1

(Actual numbers are different, but important thing is how they relate
together.)

This way, OpenVPN process runs in base vnet, using one side of
pre-created tun/tap interface, while networking uses the other side of
this interface in child vnet, isolated from base vnet (and other OpenVPN
instances as well).

Presently, I am using vlan interfaces on one ethernet interface to
connect individual instances to their respective local network. I'd
like to replace this with some tunnel interface (gif, gre, ideally
ipsec secured). The best way to illustrate is using Cisco config
snippet:

interface Tunnel1
 vrf forwarding vrf1
 ip address 192.168.0.1 255.255.255.252
 tunnel source Loopback0
 tunnel destination 172.16.0.1

This means outer layer uses base route table for tunnel creation, while
inner layer, packets/datagrams transferred over tunnel, use other vrf.

I tried to mimic this in FreeBSD with following commands:

ifconfig gre1 create tunnel 172.16.1.1 172.16.0.1 vnet ov1
jexec ov1 ifconfig gre1 10.1.0.2/30 10.1.0.1

This does not work. I found some older post which made me believing
this is caused by clearing whole tunnel configuration after moving
interface into different vnet. My (failed) tests indicate this is most
probably the cause.

So, my question is, does anybody use tunnel interface similar way? Is
it possible to achieve what I am trying with netgraph? I am able to
create some inter-vnet link using epair interface, but this is
something different. Or ideally, is somebody using IPSEC with VNET
jails, processing encapsulating packets in base and raw content in some
child vnet?

Regards,
Milan