When to use Jails with VNET, and when not?!

[Matthew Seaman]

On 28/01/2019 18:13, Parsa Samet via freebsd-questions wrote:
> Would someone please give me a brief explanation of when to use
> jails with VNET and when not to? If VLAN-ing is not my concern, and 
> services I use do not need a separate network stack - let’s say I run
> anything from DNS server to MailServer, Database, Java Application
> Server, VCS, CICD implementations, Streamers, Log Analyzers and etc.,
> but believe they don’t in all scenarios need separate stacks - would
> there be anything else left for me to benefit from VNET?

There is no general clear-cut reason to use VNET jails over traditional
ones -- for the vast majority of cases, either style will serve you
well.  There are some edge cases where the decision is easier:

  * Jails with no networking -- these might not sound very useful, but
    for example, they are used heavily by poudriere for providing clean
    build environments.  In this case, there's no need to add all the
    host-side pieces for a VNET jail, like configuring a bridge0

  * Jails where they need an independent routing table or firewall
    config, typically because management of the jail has been delegated
    to a different group than manages the containing host.  Not always
    though: on a multi-homed system jailed applications can quite easily
    need different routing than the main host.  These are certainly best
    served through VNET jails.

  * Jails where the software needs access to a standard loopback
    interface or where loopback traffic should not be routed via a
    network accessible interface for security reasons.  Again, VNET
    jails are appropriate here.

  * In a mixed environment of bare-metal servers and jails, configuring
    jails to use VNET means that they behave much more like standard
    hosts.  This may simplify management, particularly if you use
    configuration management systems like ansible or puppet.

An example of software that benefits from the third case is unbound
where it rejects packets that appear on a different interface than it
expects.

I suspect that for the services you mention, there is no compelling case
either way between VNET and traditional jails.  You get to choose fairly
arbitrarily.

> All services I run on my servers are in a jail, and only some rare
> services are in an OpenBSD vm on top of bhyve. Also, I’m on FreeBSD
> 12.0-RELEASE-p2 with ZFS.

Yes -- 12.0-RELEASE is most suitable for running VNET enabled jails.
You could do it on older FreeBSD versions, but the code was a lot less
polished and you'ld need a custom kernel.  ZFS works nicely with jails
-- consider investigating the iocage jail management software:
https://github.com/iocage/iocage which stores jail configurations as ZFS
properties.

	Cheers,

	Matthew

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20190129/53c6e323/attachment.sig>