vnet/jail crashdump

[Ronald Klop]

On Mon, 03 Aug 2020 20:27:07 +0200, Ernie Luzar <luzar722 at gmail.com> wrote:

> Ronald Klop wrote:
>> Hi,
>>  After stopping a jail I get a crashdump.
>> core.txt:  
>> https://www.klop.ws/core_2eef39c581f90f2f0c4921e43f1998c1/core.txt.0
>>  Jail.conf:
>> ----------
>> exec.stop = "/bin/sh /etc/rc.shutdown";
>> exec.clean;
>>  exec.prestart = "ifconfig bridge0 > /dev/null 2> /dev/null || (  
>> ifconfig bridge0 create && ifconfig bridge0 addm vtnet0 && ifconfig  
>> bridge0 up)";
>>  exec.consolelog = "/var/log/jail_${name}_console.log";
>>  mount.devfs;
>> path = "/data/jails/$name";
>> host.hostname = "$name";
>> mount.fstab = "/data/jails/fstab.$name";
>> vnet;
>> allow.mlock;
>> devfs_ruleset="110";
>>  freebsd12 {
>>     osrelease = 12.1-RELEASE-p4;
>>     osreldate = 1201000;
>>     vnet.interface = "epair0b";
>>     # make sure the exec.prestart has a "+=" as we de it in the global  
>> definition
>>     # when checking for the bridge
>>     exec.prestart += "ifconfig epair0 create up";
>>     exec.prestart += "ifconfig bridge0 addm epair0a";
>>     exec.prestart += "ifconfig epair0b link 02:xxxxxx:0c";
>>     exec.start = "dhclient epair0b";
>>     exec.start += "/bin/sh /etc/rc";
>>     exec.poststop  = "ifconfig bridge0 deletem epair0a";
>>     exec.poststop += "ifconfig epair0a destroy";
>>  }
>> freebsd13 {
>>     vnet.interface = "epair1b";
>>     # make sure the exec.prestart has a "+=" as we de it in the global  
>> definition
>>     # when checking for the bridge
>>     exec.prestart += "ifconfig epair1 create up";
>>     exec.prestart += "ifconfig bridge0 addm epair1a";
>>     exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d";
>>     exec.start = "dhclient epair1b";
>>     exec.start += "/bin/sh /etc/rc";
>>     exec.poststop  = "ifconfig bridge0 deletem epair1a";
>>     exec.poststop += "ifconfig epair1a destroy";
>> }
>> ----------
>>  What can I do to help debug?
>>
>
>
> Don't understand why you have these 2 statements
>
>       exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d";
>       exec.start = "dhclient epair1b";


Using dhcp on a fixed MAC is much faster in my network. This might be  
written in a better way. Please enlighten me. After a lot of twiddling  
with settings this worked.


> There is a well known bug with bridge vnet tear down since release 9.0.  
> Their is a rewrite of if_bridge going on right now to fix the problem  
> and increase the performance of if_bridge. As of today this fix is not  
> in 12.2 stable or 13.0 current.


Ah ok, so it is a known issue.


> There also looks like a bug in jail(8) when you have both vnet jails and  
> non-vnet jails being started on the same host at the same time. In most  
> cases the host just loses internet access until all the jails are  
> stopped. Some times you will get a system crash.


Ok. Not my use case, but good to know.


> This jail.conf def seems to work around the bridge tear down problem
>
> #  vnet jail using the bridge/epair method on 12.1
> v0jail1 {
> host.hostname   = "v0jail1";
> path            = "/usr/jails/v0jail1";
> mount.fstab     = "/usr/local/etc/fstab/v0jail1";
> exec.consolelog = "/var/log/v0jail1.console.log";
> mount.devfs;
> devfs_ruleset   = "4";
> vnet            = "new";
> vnet.interface  = "epair55b";
> exec.prestart   = "ifconfig epair55  create up";
> exec.prestart  += "ifconfig bridge0 addm epair55a";
> exec.prestart  += "ifconfig epair55a descr vnet-v0jail1";
> exec.prestart  += "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0  
> alias";
> exec.start      = "/bin/sh /etc/rc";
> exec.start     += "ifconfig epair55b inet 10.0.48.1 netmask  
> 255.255.255.0";
> exec.start     += "route add default 10.0.48.2";
> exec.prestop    = "ifconfig epair55b -vnet v0jail1";
> exec.stop       = "/bin/sh /etc/rc.shutdown";
> exec.poststop   = "ifconfig bridge0 deletem epair55a";
> exec.poststop  += "sleep 2";
> exec.poststop  += "ifconfig epair55a destroy";
> exec.poststop  += "ifconfig bridge0 inet 10.0.48.2 -alias";
> }
>
> Remember that your host firewall processes all traffic in & out of the  
> host including any vnet jail traffic. Yes a vnet jail has its own stack  
> and can have its own firewall, but the host firewall still has the last  
> say. The host must NAT any private ip addresses used by the vnet jails.
>
> jail.conf jail definitions are based on hard codded ip addresses. You  
> can not use the host dhcp to assign local lan private ip addresses to a  
> jail.
>
> You may find this helpful
>
> https://forums.freebsd.org/threads/vnet-jail-with-public-internet-access-using-the-bridge-epair-method.76071/
>

Thanks for all the info.

Ronald.