Building ports without needing to installing dependencies into /usr/local

Theron Tarigo theron at
Wed Aug 21 20:56:18 UTC 2019

Hello all,

I'd like to share with you the following summary of my progress in 
adapting the FreeBSD ports framework to gain the capability to build 
packages (including dependencies) in isolation of the local system 
configuration in /usr/local, whereas the existing behavior is to require 
dependencies installed there.  This work is the result of my 
participation in FreeBSD's Google Summer of Code.

Due to the widespread assumption by ports that dependencies are 
installed at build time into their final deployed locations, this has 
been a non-trivial undertaking.  Whereas Poudriere remedies this by 
running the ports framework entirely within Jails, this project attempts 
to patch all build-time scripts and tooling to access required files 
from an environment-controlled location.  In most cases this is 
accomplished by a userspace library to catch and rewrite file paths in 
POSIX API calls, which has also been developed as part of this project:

The project was motivated by the observation that the ports framework 
as-is (without external tools) fundamentally lacks the capability of 
completing all build work before installation is performed, which is 
found in nearly all single-project build systems including FreeBSD 
base.  This is different from tools such as Poudriere or Synth as it is 
designed as a core dependency-handling mechanism of the ports build 
system rather than as an external management or automation tool.

The need to touch some core parts of the FreeBSD ports framework means 
that readiness of the project for adoption may be a long time from now.  
That said, I feel confident in declaring it a successful experiment, 
after testing a limited number of ports under the new scheme (in which 
${LOCALBASE} i.e. /usr/local is never touched):
Of 5638 ports known (1979 ports selected randomly from ports tree, and 
their dependencies), 75% were built successfully, since 23% were skipped 
due to failed dependencies.
Of 4230 ports that could be tested (all dependencies were satisfied), 
97% succeeded.
I would have liked to have tested the entire ports tree, but haven't had 
access to sufficient machine-time.
I've discussed these success rates with my mentor, Bakul Shah, and we 
agreed that this demonstrates the usefulness of the method.
Of course the project is not ready for adoption as the default way of 
building ports until 100% success here is achieved, but merging of the 
code on an earlier schedule should be feasible since it is implemented 
as an option which can remain turned off by default without changing 
existing behaviors.

 From a user's perspective, the project currently provides a 
lighter-weight alternative to one of the core features of Poudriere: A 
port and all its dependencies are buildable in isolation (like Jail) to 
create one or more packages to be later installed on one or more systems.

It is light-weight because:
- Creation of virtual environments for dependencies is done purely in 
userspace, so Jail and Nullfs are not used.
- The feature is usable through the standard Make targets; no top-down 
separate script or program is used to start the builds.
- No preparation other than checking out the (modified, to be eventually 
merged) ports tree is required.
- Direct changes to existing files in the ports framework are kept to a 

Other points of potential interest:
- Superuser is not required at any part of the process (other than the 
installation of the resulting packages).

It should not be misunderstood as trying to be a potential replacement 
for Poudriere: It does not perform jailed testing, incremental rebuilds, 
utilization of ZFS, or many other advanced features, nor are such 
features planned, nor even appropriate for direct inclusion into 

The scheme currently works only on amd64, due to a small piece of the 
userspace virtual environment implementation involving machine code 
manipulation, but this can be extended to support other architectures.

For those interested in helping to test, the work is available on Github 
as follows.  The ports are synchronized to quarterly 2019Q3 as of Aug 15.
An example of how it can be tested (no need for superuser):
(note that /usr/lib/debug/libexec/ (from base-dbg or 
from installworld) must exist)
$ git clone -b separated 
--depth 1
$ cd freebsd-ports
devel/gmake config-recursive package-recursive
To be extra sure it is not relying on /usr/local: try instead with 
LOCALBASE=/usr/nlocal (of course then packages won't install to the 
default local prefix).
Resulting packages will be in $HOME/ports/packages/All/
This modified ports tree is intended to behave exactly as the official 
one when PORTS_SEPARATED_BUILD=1 is NOT used.  Anywhere that this is 
violated is a bug that I must fix.

Much cleanup and far more thorough testing beyond what I have 
accomplished in the time of this project are needed before this should 
be used in production or considered for merging into official ports tree.

Your feedback would be much appreciated and will help me to prepare a 
report on the project's successes, shortcomings, and future directions 
with respect to the community's needs.


More information about the soc-status mailing list