Importing NetBSD's mtree (and install)

[Warner Losh]

On Sep 17, 2012, at 2:19 PM, Brooks Davis wrote:

> As part of an effort to improve our ability to create full system images
> without root access I would like to import NetBSD's version of mtree.
> By doing so we would gain the -C option which produces mtree files
> compatible with libarchive and makefs (one line per file with full
> path) and the -N option which allows a stand alone set of passwd and
> group files.
> 
> When mated with the -U and -M options to NetBSD's install we will have
> most[0] of the pieces require to allow installworld to run as a user and
> then build images containing proper permissions.  The rest of this post
> will focus on my plans for mtree since it is the logical next step.
> 
> NetBSD's mtree is missing a few features present in our mtree.  Most of
> them looks simple to implement and I plan to do so before any import.
> The only exception is -i which implements indenting of old-style mtree
> files to match the format of the files in /etc/mtree/.  It will either
> need another name or to be dropped.  I honestly don't see the point in
> it so I'm not sure if it's worth keeping if people will need to alter
> their scripts regardless, but I'm willing to be convinced otherwise.
> 
> Importing mtree requires importing or implementing some enhancements to
> libc.  First, the strsvis() function is required which is most
> easily handled by importing NetBSD's vis/unvis implementations with the
> addition of the VIS_GLOB set of characters.  Compatibly wrappers will
> be required for existing vis(3) functions and for unvis() due to ABI
> changes, but they will be minimal.
> 
> Second, pwcache_userdb(), uid_from_user(), pwcache_groupdb(), and
> gid_from_group() provide useful enhancements to the uid_from_user()
> and group_from_gid().  They appear to be entirely compatible with our
> current implementation so simply importing the enhanced version seems
> like the best course.
> 
> Finally, FreeBSD and Mac OS have the functions fflagstostr() and
> strtofflags() in libc.  NetBSD has flags_to_string() and
> string_to_flags() in libutil.  The latter could be a very thin wrapper
> around fflagstostr() and the former is exactly strtofflags().  I think
> the best course is probably to provide compatible wrappers for mtree's
> internal use, but I could be convinced to make them more globally
> available.
> 
> Does anything about this plan seem seriously objectionable?

Looks good to me.  I started doing this a while ago an ran into lots of problems, many of which you've outlined here. They weren't hard problems, just numerous enough for me to lose interest in the project before I completed it.  Glad to see somebody has pushed through.

> I've written some of this up along with a list of missing features in
> the wiki.  I'll keep progress up to date there:
> 
> http://wiki.freebsd.org/NetBSDMtree
> 
> -- Brooks
> 
> [0] We also lack a tool to build disk images including partition tables,
> but Marcel is looking into this.

makefs will build the disk image w/o the MBR/GPT tables.  For most flash devices, this is sufficient.  For more complex situations, like where the boot loader groks FAT but not UFS, that needs some help... Once upon a time, you could use dd + fdisk to create an MBR image, but nothing apart from the kernel understands using it.  gpart is so darn easy to use, it is a shame that you have to make a privileged trip to the kernel to use it. 

Warner