[Proof of Concept] Stacked unionfs based 'tinderbox'

[David Naylor]

Hi,

As some may have noticed on -current I have been working on using
stacked unionfs
to implement a 'tinderbox' type build system.  I have successfully
used the scripts to
build x11/xorg (and have compared the results to using the traditional
approach using
pkg_add).  The build system is stable except for one nasty corner
case: deadlocks.

To setup a compatible test environment requires:
 - recompile the kernel with `options KSTACK_PAGES=32`, otherwise the
kernel will
   panic with a double fault.  WITNESS options results in substantial
performance
   degradation.
 - patch mtree (see below) [optional]
 - create the appropriate chroot environment (and reboot) [see below
for corner case]

A performance bottleneck in mtree was identified.  This resulted in
mtree (as run by
port install) consuming ~20% of the build time.  See bin/143732 for a patch and
further details.

Here are my build times (in the chroot):
# time make -C /usr/ports/x11/xorg install clean
     2397.42 real      2228.35 user      1151.00 sys

# time ./ports-union-builder.sh (old mtree)
     8123.25 real      2280.53 user      6319.77 sys

# time ./ports-union-builder.sh (new mtree)
     6456.11 real      2272.07 user      5778.74 sys

# time ./ports-tinder-builder.sh (new mtree)
     4270.82 real      2961.88 user      1636.27 sys

The "new mtree" is the mtree with the above mentioned patch applied.
ports-tinder-builder is an approximate to how the read tinderbox works.

The normal tinderbox approach takes ~80% more time to install compared to the
quick and dirty approach.  The stacked unionfs approach takes ~170% more time
(an increase of ~50% over the tinderbox approach).  Some performance gains can
be had if one uses memory backed storage (vs HDD in this case).

This approach should have application where wearing of the underlying storage
(such as SDD) is of concern.

Further investigation suggests that stacking unionfs scales
exponentially, not linearly.
This may be an area for further improvements.

The corner case: sometimes the directories involved experience a
deadlock.  Any IO
under those directories stall the program.  I have found that deleting
files in the chroot
prior to running the unionfs script reliably causes the deadlock.
Since WITNESS had
significant performance impact the only data into the deadlocks come
from procstat.

Below are a variety of procstat -kk calls for deadlocked programs:
68217 100119 pkg-config       -                mi_switch+0x16f
sleepq_wait+0x3b __lockmgr_args+0x641 ffs_lock+0x8c VOP_LOCK1_APV+0x46
unionfs_lock+0x28a VOP_LOCK1_APV+0x46 unionfs_lock+0x161
VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46
unionfs_lock+0x161 VOP_LOCK1_APV+0x46 unionfs_lock+0x161
VOP_LOCK1_APV+0x46 unionfs_lock+0x161 VOP_LOCK1_APV+0x46
unionfs_lock+0x161
 6574 100201 ls               -                mi_switch+0x16f
sleepq_wait+0x3b __lockmgr_args+0x654 ffs_lock+0x8c VOP_LOCK1_APV+0x46
_vn_lock+0x44 vget+0x67 cache_lookup+0x4f8 vfs_cache_lookup+0xad
VOP_LOOKUP_APV+0x40 lookup+0x44d namei+0x4ec kern_statat_vnhook+0x82
kern_statat+0x15 lstat+0x22 syscall+0x1e7 Xfast_syscall+0xe1

Is anyone able to fix the deadlocking issue?

Regards,

David.

P.S. This has been tested under 8 and 9.
P.P.S. The scripts used are attached