Building ports without needing to installing dependencies into /usr/local
Theron Tarigo
theron at freebsd.org
Wed Aug 21 20:56:18 UTC 2019
Hello all,
I'd like to share with you the following summary of my progress in
adapting the FreeBSD ports framework to gain the capability to build
packages (including dependencies) in isolation of the local system
configuration in /usr/local, whereas the existing behavior is to require
dependencies installed there. This work is the result of my
participation in FreeBSD's Google Summer of Code.
Due to the widespread assumption by ports that dependencies are
installed at build time into their final deployed locations, this has
been a non-trivial undertaking. Whereas Poudriere remedies this by
running the ports framework entirely within Jails, this project attempts
to patch all build-time scripts and tooling to access required files
from an environment-controlled location. In most cases this is
accomplished by a userspace library to catch and rewrite file paths in
POSIX API calls, which has also been developed as part of this project:
https://github.com/therontarigo/freebsd-user-namespace/
The project was motivated by the observation that the ports framework
as-is (without external tools) fundamentally lacks the capability of
completing all build work before installation is performed, which is
found in nearly all single-project build systems including FreeBSD
base. This is different from tools such as Poudriere or Synth as it is
designed as a core dependency-handling mechanism of the ports build
system rather than as an external management or automation tool.
The need to touch some core parts of the FreeBSD ports framework means
that readiness of the project for adoption may be a long time from now.
That said, I feel confident in declaring it a successful experiment,
after testing a limited number of ports under the new scheme (in which
${LOCALBASE} i.e. /usr/local is never touched):
Of 5638 ports known (1979 ports selected randomly from ports tree, and
their dependencies), 75% were built successfully, since 23% were skipped
due to failed dependencies.
Of 4230 ports that could be tested (all dependencies were satisfied),
97% succeeded.
I would have liked to have tested the entire ports tree, but haven't had
access to sufficient machine-time.
I've discussed these success rates with my mentor, Bakul Shah, and we
agreed that this demonstrates the usefulness of the method.
Of course the project is not ready for adoption as the default way of
building ports until 100% success here is achieved, but merging of the
code on an earlier schedule should be feasible since it is implemented
as an option which can remain turned off by default without changing
existing behaviors.
From a user's perspective, the project currently provides a
lighter-weight alternative to one of the core features of Poudriere: A
port and all its dependencies are buildable in isolation (like Jail) to
create one or more packages to be later installed on one or more systems.
It is light-weight because:
- Creation of virtual environments for dependencies is done purely in
userspace, so Jail and Nullfs are not used.
- The feature is usable through the standard Make targets; no top-down
separate script or program is used to start the builds.
- No preparation other than checking out the (modified, to be eventually
merged) ports tree is required.
- Direct changes to existing files in the ports framework are kept to a
minimum.
Other points of potential interest:
- Superuser is not required at any part of the process (other than the
installation of the resulting packages).
It should not be misunderstood as trying to be a potential replacement
for Poudriere: It does not perform jailed testing, incremental rebuilds,
utilization of ZFS, or many other advanced features, nor are such
features planned, nor even appropriate for direct inclusion into
/usr/ports/Mk.
The scheme currently works only on amd64, due to a small piece of the
userspace virtual environment implementation involving machine code
manipulation, but this can be extended to support other architectures.
For those interested in helping to test, the work is available on Github
as follows. The ports are synchronized to quarterly 2019Q3 as of Aug 15.
An example of how it can be tested (no need for superuser):
(note that /usr/lib/debug/libexec/ld-elf.so.1.debug (from base-dbg or
from installworld) must exist)
$ git clone https://github.com/therontarigo/freebsd-ports -b separated
--depth 1
$ cd freebsd-ports
$ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C
devel/gmake config-recursive package-recursive
To be extra sure it is not relying on /usr/local: try instead with
LOCALBASE=/usr/nlocal (of course then packages won't install to the
default local prefix).
Resulting packages will be in $HOME/ports/packages/All/
This modified ports tree is intended to behave exactly as the official
one when PORTS_SEPARATED_BUILD=1 is NOT used. Anywhere that this is
violated is a bug that I must fix.
Much cleanup and far more thorough testing beyond what I have
accomplished in the time of this project are needed before this should
be used in production or considered for merging into official ports tree.
Your feedback would be much appreciated and will help me to prepare a
report on the project's successes, shortcomings, and future directions
with respect to the community's needs.
Theron
More information about the soc-status
mailing list