Poudriere nested jail configuration - want explanation for each setting

[Andreas Sommer]

Hi all (main poudriere authors in CC, hoping this is etiquette-friendly),

I'm currently writing a tutorial that includes setting up Poudriere within a jail (nested). While the appropriate settings are nicely listed at https://github.com/freebsd/poudriere/wiki/poudriere_in_jail, I'd like to get clarity which ones are actually needed and *why*. Of course I did some research in freebsd/poudriere code which resulted in the write-up I have until now (feel free to improve grammar/text as well):

> Jail config:
>     ip4.addr += "lo0|127.0.0.3";
>     exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0";
>     allow.chflags;
>     allow.mount;
>     allow.mount.devfs;
>     allow.mount.nullfs;
>     allow.mount.procfs;
>     allow.mount.tmpfs;
>     allow.mount.zfs; # only needed if you use ZFS
>     allow.raw_sockets; # optional
>     allow.socket_af; # optional
>     allow.sysvipc; # optional
>     children.max=16;
>     enforce_statfs=1;
>     [...]

> - `ip4.addr += "lo0|127.0.0.3"` adds another IPv4 address to the jail. You will later configure Poudriere's `LOIP4` variable in order to assign this loopback address to build jails which are not supposed to talk to the Internet or other machines in your network, such as during the `build` phase. If you ever have a build which requires Internet access during build, Poudriere supports a variable `ALLOW_NETWORKING_PACKAGES` as workaround. However, you should prefer the clean way, i.e. performing downloads and other Internet-facing tasks earlier, in the `fetch` phase which is permitted Internet access.
> - `allow.chflags` allows Poudriere to render certain system files in the build jail immutable, since builds are not supposed to overwrite e.g. `/bin/sh`
> - `allow.mount` and the other `allow.mount.*` options enable Poudriere to mount certain required filesystems into the build jails
> - `allow.raw_sockets` (permit use of raw sockets) and `allow.socket_af` (use of any socket address family) are applied to the Internet-capable build jails. This is mostly only helpful so that you can run tools like `ping` in interactive mode i.e. when entering a build jail to debug problems. You could disable these permissions.
> - `allow.sysvipc` is actually deprecated in favor of three separate settings `sysvmsg`/`sysvsem`/`sysvshm` to restrict jails to only see their own shared memory objects (via "SYS V" IPC primitives). However, Poudriere can only pass on `allow.sysvipc` to build jails because it cannot read the relevant sysctl information for the three separate parameters (as of FreeBSD 11.2). With this deprecated configuration, the jail could read shared memory of processes outside the jail. This is only relevant for certain software that depends on IPC features, like PostgreSQL, so chances are small for this to affect security. You can remove this configuration unless you depend on a port that requires it during build.
> - `children.max=16` allows 16 sub-jails below the worker jail. You can raise this number later if you have a lot of CPUs and Poudriere tries to create more build jails than permitted. Note that each Poudriere build will try to create a reference jail and 2 build jails per "job", and its default is to use the number of CPUs (as output by `sysctl -n hw.ncpu`) as job count.
> - `enforce_statfs=1` is required together with `allow.mount` in order to mount certain filesystems.

Can you confirm the descriptions?

Especially for the `allow.sysvipc` topic, I digged a little more: There's `security.jail.sysvipc_allowed={0,1}` which is passed on to build jails by Poudriere. And for the 3 non-obsolete jail settings `sysvmsg`/`sysvsem`/`sysvshm`, sysctl has to offer `kern.features.sysv_{shm,sem,msg}` and `security.jail.param.sysv{shm,sem,msg}.` (with a dubious dot at the end!). And here's the problem: no matter whether I configure the jail with `{sysvmsg,sysvsem,sysvshm} = "new";` or leave out any SYSVIPC setting (= disallow altogether), the sysctls stay the same at:

> $ sudo jexec myjail sysctl -a | grep .sysv
> kern.features.sysv_shm: 1             <---- these are always 1
> kern.features.sysv_sem: 1
> kern.features.sysv_msg: 1
> security.jail.param.sysvshm.: 0       <---- these are always 0
> security.jail.param.sysvsem.: 0
> security.jail.param.sysvmsg.: 0
> security.jail.param.allow.sysvipc: 0  <---- these are unrelated to the 3 separate settings
> security.jail.sysvipc_allowed: 0

On the other hand, I confirmed the `{sysvmsg,sysvsem,sysvshm} = "new";` settings do what they should (using FreeBSD 11.2 and PostgreSQL) – prohibit the jail from seeing outer shared objects. Therefore I concluded that Poudriere simply has no visibility whether the "poudriere jail" has those features, and hence cannot pass those on to its nested build jails. It does that for `allow.sysvipc`, though, since the sysctl `security.jail.sysvipc_allowed` shows the correct value within the jail. Reference:

poudriere/src/share/poudriere/include/common.sh.freebsd
> if [ ${JAILED} -eq 0 ] || \
>     [ $(sysctl -n security.jail.sysvipc_allowed) -eq 1 ]; then
> 	JAIL_PARAMS="${JAIL_PARAMS:+${JAIL_PARAMS} }allow.sysvipc"
> fi

Thanks in advance,
 Andreas