[Bug 280339] Handbook section 17.5.2 Creating a Thin Jail Using NullFS results in broken jails

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280339

            Bug ID: 280339
           Summary: Handbook section 17.5.2 Creating a Thin Jail Using
                    NullFS results in broken jails
           Product: Documentation
           Version: Latest
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: Books & Articles
          Assignee: doc@FreeBSD.org
          Reporter: tommy@s3cr3t.codes

Section 17.5 "Thin Jails" contains directions to create jails having a large
amount of reuse. However, jails created following these instructions have a
large number of broken symlinks inside the jails.

To see the problem, follow exactly the instructions at
https://docs.freebsd.org/en/books/handbook/jails/#creating-thin-jail-nullfs. I
only tried the ZFS instructions, but I suspect this issue applies equally to
UFS. I also replaced `13.2-RELEASE` with `$(uname -r)` everywhere in those
instructions - my workaround for #275685. After executing the last command in
that section (`service jail start thinjail`) attempt to install a package in
that jail: `pkg -j thinjail bootstrap -f`. You can see that there is an error,
and the `pkg` command isn't bootstrapped there. Running `pkg bootstrap -f` from
a shell "inside" the jail (via `jexec`) likewise fails.

The issue is that the certificate files symlinked inside the jail at
`/etc/ssl/certs` use relative paths to find the target files; e.g.,
`../../../usr/share/certs/trusted/GlobalSign_Root_E46.pem`. However, these
relative paths are effectively broken by the reusable-base/skeleton separation
described in 17.5. From within the jail's view of `/etc/ssl/certs`, the path
`../../../usr` (prefix of these symlinks' targets) is a directory at (absolute
path inside the jail) `/skeleton/usr` which contains only another directory
`local`. To repair the symlinks, you could prefix an extra level of `../`
(i.e., `../../../../usr/share/certs/trusted/GlobalSign_Root_E46.pem` for the
example from above). This gets you the relative path to the correct `usr`
wherein `share/certs/trusted/GlobalSign_Root_E46.pem` exists. After changing
all symlinks in `/etc/ssl/certs` within the jail, it can be seen that `pkg
bootstrap -f` works from inside the jail, as also `pkg -k thinjail bootstrap
-f` works from the host.

It's not clear to me what is the recommended solution; adding documentation and
code blocks in 17.5 to "fix up" all symlinks inside the jail is probably
incorrect, because this section is designed to facilitate reuse, so that all
jails using the same release "base" template could be updated at once from the
host: `freebsd-update -b /usr/local/jails/templates/13.2-RELEASE-base/ fetch
install`. This can conceivable clobber some or all of the fixed-up symlinks.
Basically, I suspect that the nullfs-thin-jails approach is doomed without
quite a bit of workaround. I could write my own suite of wrappers for `pkg`,
`freebsd-update`, and potentially other commands that know how to operate on
jails from the host, but the effort involved with that starts to push me toward
one of the externally-maintained jails management suites that I had hoped to
avoid.

-- 
You are receiving this mail because:
You are the assignee for the bug.