Tar output mode for installworld

[Paul Schenkeveld]

On Sat, Jul 14, 2007 at 11:28:05PM -0700, Tim Kientzle wrote:
> >>>This is easy to implement ... just build
> >>>a description of the final archive in a nice verbose
> >>>text format such as:
> >>
> >>...which is done by NetBSD for the unprivileged release building via
> >>build.sh. Anyone interested in working on this should possibly have a
> >>look there.
> 
> Here's a rough implementation of my core idea.  Add the
> attached archive_read_support_format_ntree.c to libarchive
> and patch archive_read_support_format_all.c, then
> rebuild libarchive and bsdtar.  You'll then be able
> to read, extract, etc, a format I'm calling "ntree"
> for now. This similar to NetBSD's "metalog" format, except:
> 
> 1) First line must be "#%ntree".  This is used as a file signature.
> 
> 2) Blank lines and lines beginning with '#' are ignored.
> 
> 3) All other lines have the following format:
> 
> <filename> <key>=<value> <key>=<value> ...
> 
> Where key is one of:
>   time:  decimal seconds since beginning of epoch
>   gid,uid: decimal group/user ID
>   gname,uname: textual group/user name
>   mode: octal
>   type: as in mtree, defaults to 'file'
>   content: name of a file on disk

Great!

This is exactly the next BIG step in (Free)BSD build/distribute/install
that I have been waiting for.  In the mean time I've been playing a lot
with automating and customizing the build/distribute/install process
of FreeBSD, partly inspired by nanobsd.

Basically the process is something like:

  __MAKE_CONF=my_make.conf make buildworld
  __MAKE_CONF=my_make.conf make installworld DESTDIR=/some/place
  $CUSTOMIZATIONS /some/place
  tar czf tarball.tgz -C /some/place
  # Find a way to get the tarball.tgz to its destination
  tar xpf tarball.tgz    # at the destination

For the $CUSTOMIZATIONS to work it would be very nice to extend your
proposal with the following features:

  - Allow scattered lines describing attributes of the same file but
    retain the order in which they appear so that $CUSTOMIZATIONS can
    override attributes set by make installworld.

  - Implement something like:
      <filename> remove
    to undo the installation of files (embedded systems hardly ever
    need stuff like documentation, NLS and so on but the make.conf and
    src.conf knobs are not fine-grained enough for all situations)

  - Implement something like:
      <filename> move=<newpath>
    to allow $CUSTOMIZATIONS to move things around.
    A special case here should be observed:
      /bin/foo ...
      ...
      /bin/foo move=/bin/foo.orig
      /bin/foo ...
    here the $CUSTOMIZATIONS move some file so another place and installs
    a wrapper.  Probably a sorting algorithm that retains the order in
    which /bin/foo lines appear does half the trick and libarchive should
    collect and accumulate lines with the same filename and only emit or
    extract a file after the last line with the same name OR when an move=
    key appears and reset file info immediately after this line.

Having a file describing everything that gets installed would also benefit
later upgrades to a system.  In my experiments I've implemented a manifest
that gets shipped with the tarball.tgs (still mean to put it inside but
got some ENOTIME errors).  My manifest also includes md5 and/or sha256
checksums for each file so the upgrade tool can easily detect which files
were altered/moved/removed by the sysadmin an sensibly act accordingly.
Again ENOTIME prevented me so far from writing the update tool but I can
certainly help out here.

If all works out well, this could also benefit ports where a package
could be created without installing the port on the build system.  Many
hurdles to take, I know but certainly a goal to consider.

-- Paul Schenkeveld