Tar output mode for installworld

[M. Warner Losh]

In message: <20070715133016.GA53853 at psconsult.nl>
            Paul Schenkeveld <fb-hackers at psconsult.nl> writes:
: On Sat, Jul 14, 2007 at 11:28:05PM -0700, Tim Kientzle wrote:
: > >>>This is easy to implement ... just build
: > >>>a description of the final archive in a nice verbose
: > >>>text format such as:
: > >>
: > >>...which is done by NetBSD for the unprivileged release building via
: > >>build.sh. Anyone interested in working on this should possibly have a
: > >>look there.
: > 
: > Here's a rough implementation of my core idea.  Add the
: > attached archive_read_support_format_ntree.c to libarchive
: > and patch archive_read_support_format_all.c, then
: > rebuild libarchive and bsdtar.  You'll then be able
: > to read, extract, etc, a format I'm calling "ntree"
: > for now. This similar to NetBSD's "metalog" format, except:
: > 
: > 1) First line must be "#%ntree".  This is used as a file signature.
: > 
: > 2) Blank lines and lines beginning with '#' are ignored.
: > 
: > 3) All other lines have the following format:
: > 
: > <filename> <key>=<value> <key>=<value> ...
: > 
: > Where key is one of:
: >   time:  decimal seconds since beginning of epoch
: >   gid,uid: decimal group/user ID
: >   gname,uname: textual group/user name
: >   mode: octal
: >   type: as in mtree, defaults to 'file'
: >   content: name of a file on disk
: 
: Great!
: 
: This is exactly the next BIG step in (Free)BSD build/distribute/install
: that I have been waiting for.  In the mean time I've been playing a lot
: with automating and customizing the build/distribute/install process
: of FreeBSD, partly inspired by nanobsd.
: 
: Basically the process is something like:
: 
:   __MAKE_CONF=my_make.conf make buildworld
:   __MAKE_CONF=my_make.conf make installworld DESTDIR=/some/place
:   $CUSTOMIZATIONS /some/place
:   tar czf tarball.tgz -C /some/place
:   # Find a way to get the tarball.tgz to its destination
:   tar xpf tarball.tgz    # at the destination
: 
: For the $CUSTOMIZATIONS to work it would be very nice to extend your
: proposal with the following features:
: 
:   - Allow scattered lines describing attributes of the same file but
:     retain the order in which they appear so that $CUSTOMIZATIONS can
:     override attributes set by make installworld.
: 
:   - Implement something like:
:       <filename> remove
:     to undo the installation of files (embedded systems hardly ever
:     need stuff like documentation, NLS and so on but the make.conf and
:     src.conf knobs are not fine-grained enough for all situations)
: 
:   - Implement something like:
:       <filename> move=<newpath>
:     to allow $CUSTOMIZATIONS to move things around.
:     A special case here should be observed:
:       /bin/foo ...
:       ...
:       /bin/foo move=/bin/foo.orig
:       /bin/foo ...
:     here the $CUSTOMIZATIONS move some file so another place and installs
:     a wrapper.  Probably a sorting algorithm that retains the order in
:     which /bin/foo lines appear does half the trick and libarchive should
:     collect and accumulate lines with the same filename and only emit or
:     extract a file after the last line with the same name OR when an move=
:     key appears and reset file info immediately after this line.
: 
: Having a file describing everything that gets installed would also benefit
: later upgrades to a system.  In my experiments I've implemented a manifest
: that gets shipped with the tarball.tgs (still mean to put it inside but
: got some ENOTIME errors).  My manifest also includes md5 and/or sha256
: checksums for each file so the upgrade tool can easily detect which files
: were altered/moved/removed by the sysadmin an sensibly act accordingly.
: Again ENOTIME prevented me so far from writing the update tool but I can
: certainly help out here.
: 
: If all works out well, this could also benefit ports where a package
: could be created without installing the port on the build system.  Many
: hurdles to take, I know but certainly a goal to consider.

I've done something similar to what you suggest here.  Let me recount
what our build process does at work.  I like the idea of having an
unprivileged environment, but I see some problems with it.  Maybe the
problems I see are due to my experiences at work where we've been
building 13-20MB freebsd systems for the past 8 years.

First, we checkout a FreeBSD source tree into
/some/place/chroot/usr/src.  We do a buildworld, and installworld into
/some/place/chroot.  We then use that chroot to build our software.

We do the chroot process for multiple reasons.  First, we insulate the
build process from the machine we're building on 100% this way.
*EVERYTHING* builds inside the chroot, which means there's no chance
of host contamination.  Doing this 'heavy weight' solution has a large
number of advantages, since one gets to tailor the ports and other
packages installed very precisely.  It also allows us to build
different products at different revisions all on the same machine.  It
is not uncommon for me to have 15 different product versions being
built at the same time, all of them with variant ports versions, ports
configurations, and in some cases FreeBSD configurations.

The other big advantage of doing this is that it requires no special
tools, apart from the build harness.  All the normal FreeBSD install
processes work.  All the ports install processes work.  There's no
need to worry that this one uses tar, and that one uses cpio and this
other one uses bsdinstall.  They all install into
/some/place/chroot/image/... with the very low threshold of 'supports
DESTDIR' or 'can be made to act like they support DESTDIR'.

Since we do this, subsetting turns out to be trivial.  One just needs
to have a list of directories to do the install.

One icky part of what we do is our mtree stuff.  We lay down FreeBSD,
with the subset of directories, and then use mtree files to remove all
files that don't appear in the mtree file.  We do this before we
install the packages we build and the custom software we put onto the
device.  There's a second mtree pass that we use to put things like
dev nodes (<= 4.x) and symlinks into place.

All of these things are nice and easy, but many require root.  We have
to give all the developers on our servers root.  This is OK for us
because if we don't trust you with root, you wouldn't have a job with
us (and if you repeatedly abuse it, you will be out the door too).  We
mitigate it by doing everything through sudo, which logs everything.

While it would be nice to have it be done w/o privileges, chroot
absolutely requires root.  I'm guessing that it would be hard to have
the nice benefits of a chroot in an unprivileged environment.  You can
get the simple 'use these files' stuff, but being able to reproduce
files down to the same md5 even after upgrading the server is a bid
advantage.

Warner