jail(8) bug with vnet & non-vnet jails running at same time?

[Ernie Luzar]

Dan Langille wrote:
>> On Aug 2, 2020, at 1:48 PM, Ernie Luzar <luzar722 at gmail.com> wrote:
>>
>> Hello list;
>> Please review configuration looking for something I may have missed. Hopping someone can suggest something that will change the behavior eliminating the problem.
>>
>>
>> Equipment. Real hardware, 12.1 release, amd64 dual cpu.
>>
>> Description;
>> non-vnet jails and vnet jails using the bridge/epair method can ping the public internet when only non-vnet jails are started at a time or when only vnet jails are started at a time. But when both non-vnet jails and vnet jails are started together then neither one can ping the public internet. The order of the jails definitions in the jail.conf file has no effect on changing what is happening.
>>
>> Bug description:
>> When non-vnet jails are started their ip addresses are added to the NIC facing the public AFTER the public ip address and the non-vnet jail has access to the public internet. But when both non-vnet jails and vnet jails are started at the same time then the non-vnet jails ip addresses gets added before the public ip address of the NIC facing the public internet causing the host to lose all access to the public internet. This seems to be a jail(8) bug.
>>
>> It makes no difference which command method is used to start and stop the jails.
>> Service jail onestart jailname   or  jail –cv jailname
> 
> This may be related to my twitter rant about vnet problems in my own jails:
> 
>   https://twitter.com/DLangille/status/1289944047763693569
> 
> The symptoms you describe to similar to my own.  I cannot access ports on jails on the same host, but I can access ports on other hosts.
> 

Your twitter posts are all pf firewall related.  From what I can tell 
you are using local only vnet jails and want to talk between them.

Do you have any non-vnet jails running on the host where the 2 vnet 
jails are running?

Do you have any local only vnet jails working on any other systems?

To me knowledge there is only 1 way to have local only vnet jails to 
talk to each other.  Do not assign ip address to epairXa or to the 
bridge. Only assign an ip address to epairXb the interface in the vnet 
jail. All the vnet jails you want to be local only have to be members on 
the same bridge.