Container Networking for jails

From: Doug Rabson <dfr_at_rabson.org>
Date: Thu, 30 Jun 2022 07:04:24 UTC
I wanted to get a quick sanity check for my current approach to container
networking with buildah and podman. These systems use CNI (
https://www.cni.dev) to set up the network. This uses a sequence of
'plugins' which are executables that perform successive steps in the
process - a very common setup uses a 'bridge' plugin to add one half of an
epair to a bridge and put the other half into the container's vnet. IP
addresses are managed by an 'ipam' plugin and an optional 'portmap' plugin
can be used to advertise container service ports on the host. All of these
plugins run on the host with root privileges.

In kubernetes and podman, it is possible for more than one container to
share a network namespace in a 'pod'. Each container in the pod can
communicate with its peers directly via localhost and they all share a
single IP address.

Mapping this over to jails, I am using one vnet jail to manage the network
namespace and child jails of this to isolate the containers. The vnet jail
uses '/' as its root path and the only things which run inside this jail
are the CNI plugins. Using the host root means that a plugin can safely
call host utilities such as ifconfig and route without having to trust the
container's version of them. An important factor here is that the CNI
plugins will only be run strictly before the container (to set up) or
strictly after (to tear down) - at no point will CNI plugins be executed at
the same time as container executables.

The child jails use ip4/6=inherit to share the vnet and each will use a
root path to the container's contents in the same way as a normal
non-hierarchical jail.

Can anyone see any potential security problems here, particularly around
the use of nested jails? I believe that the only difference between this
setup and a regular non-nested jail is that the vnet outlives the container
briefly before it is torn down.