HOWTO - jails - FreeBSD 12 + VNET + ZFS

Ernie Luzar luzar722 at gmail.com
Fri Jan 25 21:30:57 UTC 2019 

BulkMailForRudy wrote:
> I love using jails.  For many years, I used a tool to help out: ezjail, 
> now I am just raw-dogging it by using the config file in /etc/jail.conf
> 
> 
> Here is my config:
> 
> # /etc/jail.conf
> # VNET is used to send an epair to each jail.
> # The epair is renamed jail0 with exec.created in each jail.
> # exec.prestart Script creates bridge0 if needed.
> 
> # Global settings applied to all jails.
> 
> # haven't found a good reason to run a jail as NOT root
> exec.system_user  = "root";
> exec.jail_user    = "root";
> mount.devfs;
> allow.raw_sockets;
> devfs_ruleset     = "5";
> 
> # Networking and the exec cycle
> $uplinkdev        = "ix0";
> vnet;
> vnet.interface    = "jail0";               # default 
> vnet interface
> exec.prestart     = "ifconfig bridge0 > /dev/null 2> /dev/null || ( 
> ifconfig bridge0 create up && ifconfig bridge0 addm $uplinkdev )";
> exec.prestart    += "ifconfig $epair create 
> up                 || echo 'Skipped creating epair 
> (exists?)'";
> exec.prestart    += "ifconfig bridge0 addm 
> ${epair}a           || echo 'Skipped adding bridge member 
> (already member?)''";
> exec.created      = "ifconfig ${epair}b name 
> jail0Â Â Â Â Â Â Â Â Â Â Â Â  || echo 'Skipped renaming ifdev to jail0'";
> exec.clean;
> exec.start        = "/bin/sh /etc/rc";
> exec.stop         = "/bin/sh /etc/rc.shutdown";
> exec.poststop     = "ifconfig bridge0 deletem ${epair}a";
> #exec.poststop    += "ifconfig ${epair}a destroy";
> 
> # Per-jail settings
> ns1 {
>     path          = "/data/ns1.monkeybrains.net/";
> Â Â Â  host.hostname = "ns1.monkeybrains.net";
>     $epair        = "epair0";  # must be unique in every jail
> }
> 
> tac {
>     path          = "/data/tac.monkeybrains.net/";
> Â Â Â  host.hostname = "tac.monkeybrains.net";
>     $epair        = "epair1";
> }
> 
> 
> =====================================
> 
> Here is a look at ifconfig before and after jail creation.
> 
> 
> ============Â  Before jails start up ============
> 
> ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> 
> Â Â Â  ether ac:1f:6b:6a:14:78
> Â Â Â  inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255
> Â Â Â  inet6 fe80::ae1f:aaaa:aaaa:1478%ix0 prefixlen 64 scopeid 0x1
> Â Â Â  inet6 2607:f598::a:a prefixlen 64
> Â Â Â  media: Ethernet autoselect (1000baseT <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> Â Â Â  inet6 ::1 prefixlen 128
> Â Â Â  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
> Â Â Â  inet 127.0.0.1 netmask 0xff000000
> Â Â Â  groups: lo
> 
> ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
> mtu 1500
> options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6> 
> Â Â Â  ether ac:1f:6b:6a:14:78
> Â Â Â  inet 208.69.40.26 netmask 0xffffff00 broadcast 208.69.40.255
> Â Â Â  inet6 fe80::ae1f:6bff:fe6a:1478%ix0 prefixlen 64 scopeid 0x1
> Â Â Â  inet6 2607:f598::d045:281a prefixlen 64
> Â Â Â  media: Ethernet autoselect (1000baseT <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

> ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> 
> Â Â Â  ether ac:1f:6b:6a:14:79
> Â Â Â  media: Ethernet autoselect
> Â Â Â  status: no carrier
> Â Â Â  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> Â Â Â  inet6 ::1 prefixlen 128
> Â Â Â  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
> Â Â Â  inet 127.0.0.1 netmask 0xff000000
> Â Â Â  groups: lo
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
> 1500
> Â Â Â  ether 02:16:09:1c:af:00
> Â Â Â  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> Â Â Â  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> Â Â Â  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> Â Â Â  member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> Â Â Â Â Â Â Â Â Â Â Â  ifmaxaddr 0 port 6 priority 128 path cost 2000
> Â Â Â  member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> Â Â Â Â Â Â Â Â Â Â Â  ifmaxaddr 0 port 5 priority 128 path cost 2000
> Â Â Â  member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> Â Â Â Â Â Â Â Â Â Â Â  ifmaxaddr 0 port 1 priority 128 path cost 2000
> Â Â Â  groups: bridge
> Â Â Â  nd6 options=1<PERFORMNUD>

> epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
> metric 0 mtu 1500
> Â Â Â  options=8<VLAN_MTU>
> Â Â Â  ether 02:8d:76:e8:34:0a
> Â Â Â  inet6 fe80::8d:76ff:fee8:340a%epair0a prefixlen 64 scopeid 0x5
> Â Â Â  groups: epair
> Â Â Â  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

> epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
> metric 0 mtu 1500
> Â Â Â  options=8<VLAN_MTU>
> Â Â Â  ether 02:7a:d1:7c:f8:0a
> Â Â Â  inet6 fe80::7a:d1ff:fe7c:f80a%epair1a prefixlen 64 scopeid 0x6
> Â Â Â  groups: epair
> Â Â Â  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> 
> 
> 
> ============Â  Start up jails ============
> 
> # service jail start
> Starting jails: ns1 tac.
> 
> # ifconfig
> 
> ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
> mtu 1500 
> options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6> 
> Â Â Â  ether ac:1f:6b:6a:14:78
> Â Â Â  inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255
> Â Â Â  inet6 fe80::ae1f:aaaa:aaaa:1478%ix0 prefixlen 64 scopeid 0x1
> Â Â Â  inet6 2607:f598::a:a prefixlen 64
> Â Â Â  media: Ethernet autoselect (1000baseT <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> Â Â Â  inet6 ::1 prefixlen 128
> Â Â Â  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
> Â Â Â  inet 127.0.0.1 netmask 0xff000000
> Â Â Â  groups: lo
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
> 1500
> Â Â Â  ether 02:16:09:1c:af:00
> Â Â Â  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> Â Â Â  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> Â Â Â  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> Â Â Â  member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> Â Â Â Â Â Â Â Â Â Â Â  ifmaxaddr 0 port 6 priority 128 path cost 2000
> Â Â Â  member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> Â Â Â Â Â Â Â Â Â Â Â  ifmaxaddr 0 port 5 priority 128 path cost 2000
> Â Â Â  member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> Â Â Â Â Â Â Â Â Â Â Â  ifmaxaddr 0 port 1 priority 128 path cost 2000
> Â Â Â  groups: bridge
> Â Â Â  nd6 options=1<PERFORMNUD>
> epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
> metric 0 mtu 1500
> Â Â Â  options=8<VLAN_MTU>
> Â Â Â  ether 02:8d:76:e8:34:0a
> Â Â Â  inet6 fe80::8d:76ff:fee8:340a%epair0a prefixlen 64 scopeid 0x5
> Â Â Â  groups: epair
> Â Â Â  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

> epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
> metric 0 mtu 1500
> Â Â Â  options=8<VLAN_MTU>
> Â Â Â  ether 02:7a:d1:7c:f8:0a
> Â Â Â  inet6 fe80::7a:d1ff:fe7c:f80a%epair1a prefixlen 64 scopeid 0x6
> Â Â Â  groups: epair
> Â Â Â  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> 
> 
> # jls
>    JID  IP Address      
> Hostname                      Path
> Â Â Â  19Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  ns1.monkeybrains.net 
> /data/ns1.monkeybrains.net
> 
> Â Â Â  20Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  tac.monkeybrains.net 
> /data/tac.monkeybrains.net
> 
> 
> # jexec ns1 ifconfig

> jail0: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> Â Â Â  options=8<VLAN_MTU>
> Â Â Â  ether 02:8d:76:e8:34:0b
> Â Â Â  groups: epair
> Â Â Â  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

> # jexec tac ifconfig

> jail0: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> Â Â Â  options=8<VLAN_MTU>
> Â Â Â  ether 02:7a:d1:7c:f8:0b
> Â Â Â  groups: epair
> Â Â Â  media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> Â Â Â  status: active
> Â Â Â  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

> vlan91: flags=8003<UP,BROADCAST,MULTICAST> metric 0 mtu 1500
> Â Â Â  ether 00:00:00:00:00:00
> Â Â Â  groups: vlan
> Â Â Â  vlan: 0 vlanpcp: 0 parent interface: <none>
> Â Â Â  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> 
> 


You have to learn to crawl before you can run. Start with a single vnet 
jail in jail.conf until you get something that works. Fix your post by 
getting rid of those   characters. Your post subject says + ZFS and 
you have no ZFS options in your jail.conf. Edit out lo0 on ifconfig 
displays, they add no info to this post.

In the ifconfig before jail start shows ix0 2 time with different ip 
address. Why?

jexec tac ifconfig shows vlan91, but nowhere do you show this being 
created or assigned to this jail. What is going on here?

exec.system_user  = "root";    un-necessary, remove
exec.jail_user    = "root";  un-necessary, remove
allow.raw_sockets;              only valid in non-vnet jails
devfs_ruleset     = "5";    What is custom contents of this rule #5

The vnet.interface statement needs to be per jail.

Each vnet jail must have it's own epair mumber. This is not something 
you can do in global section. Must be per jail.

Do you have any entries in the vnet jails rc.conf file? If so, show.

What is your overall goal? Be able to access the public internet?

Is this host you are trying to create working vnet jails on, on a LAN or 
is it the gateway host?

How do you test your vnet jail?

Keep in mind that just because your vnet jail starts does not mean that 
its working. Just means nothing fatal happened to cause it to dump.

Bye

More information about the freebsd-current mailing list